[UI Feature] The UI should show that a task is pending due to `cache_serialize=True`

[fg91]

Motivation: Why do you think this is important?

Cache serializing is a convenient mechanism to prevent expensive computation to be performed multiple times in parallel.

However, when a task is pending because of cache_serialize=True, there is no way for users to see that this is the reason and not a potential other problem.

Goal: What should the final outcome look like, ideally?

When a task is pending because it is waiting for another execution with the same cache version and input kwargs, the UI should mention this and provide a link to the execution blocking the one at hand.

Propose: Link/Inline OR Additional context

No response

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes

[dosubot]

Hey @fg91! Nice to see you bringing up another interesting point. Hope you've been well!

Thank you for your feature request. I agree that enhancing the Flyte UI to display a message and provide a link to the blocking execution when a task is pending due to cache_serialize=True would be beneficial for users.

As per the RFC document, a new task phase, WaitingForCache, has been introduced for tasks annotated with cache_serialize that do not currently hold the respective artifact reservation. This phase, along with reservation ownership and expiration metadata, is viewable in the FlyteConsole. Given this infrastructure, it's feasible to extend the UI to include a specific message about the waiting reason and a direct link to the blocking execution based on the reservation ownership information.

The task status is updated in the Flyte codebase within the task_execution_manager.go file, specifically in the updateTaskExecutionModelState and createTaskExecution functions. These functions handle the creation and updating of task execution models, which include the task status. The status is then reflected in the UI through the retrieval of these models and their transformation into the appropriate format for display. You can find more details in the flyteadmin/pkg/manager/impl/task_execution_manager.go file.

The cache_serialize=True attribute in the Flyte codebase is handled as part of the cache serialization mechanism for cacheable tasks. This mechanism is designed to prevent simultaneous evaluations of cacheable tasks over identical inputs, aiming to improve resource utilization and reduce duplication of work. The logic for handling cache_serialize involves several components across the Flyte ecosystem, including user-side annotation, FlyteIDL and gRPC calls, the Datacatalog service, and FlytePropeller integration.

I hope this information helps. If you have any further questions or need more details, feel free to ask.

Sources

• rfc/core language/1461-cache-serialize-api.md
• flyteadmin/pkg/manager/impl/task_execution_manager.go

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[pingsutw]

The change will be similar to this PR, but we can have another icon for cache_serialize.

[kumare3]

I think this is really important too

[Mathis-Z]

Not sure if this is related but the UI does not show pending (the Kubernetes status) tasks in general. They are marked as "Running" instead. I just had a case where the resource requirements of a task where not satisfiable so it stayed "Running" forever.

[github-actions]

"Hello 👋, this feature request has been inactive for over 9 months. To help maintain a clean and focused backlog, we'll be marking this issue as stale and will close the issue if we detect no activity in the next 14 days. Thank you for your contribution and understanding! 🙏"

[kumare3]

Not sure if this is related but the UI does not show pending (the Kubernetes status) tasks in general. They are marked as "Running" instead. I just had a case where the resource requirements of a task where not satisfiable so it stayed "Running" forever.

The ui shows pending at the task level not the node level. This has all been changed in 2.0