Open
Conversation
Implemented new datacatalog functionality required for cache eviction Updated to latest unreleased version of flyteidl and flyteplugins Signed-off-by: Nick Müller <nmueller@blackshark.ai>
Allows for fields to be explicity set/updated to nil Signed-off-by: Nick Müller <nmueller@blackshark.ai>
Allows for re-use by cache manager Signed-off-by: Nick Müller <nmueller@blackshark.ai>
Added endpoint for evicting execution cache Added endpoint for evicting task execution cache Signed-off-by: Nick Müller <nmueller@blackshark.ai>
Signed-off-by: Nick Müller <nmueller@blackshark.ai>
Extended reservation retrieval to allow querying via artifact tag in catalog client interface Signed-off-by: Nick Müller <nmueller@blackshark.ai>
Added method to delete catalog artifact by ID Signed-off-by: Nick Müller <nmueller@blackshark.ai>
Signed-off-by: Nick Müller <nmueller@blackshark.ai>
…epropeller and flytestdlib Signed-off-by: Nick Müller <nmueller@blackshark.ai>
Signed-off-by: Nick Müller <nmueller@blackshark.ai>
Added new CacheEvictionError message representing an error encountered during eviction of stored data Added new UpdateTaskExecution endpoint for updating task executions, currently only supporting cache eviction Signed-off-by: Nick Müller <nmueller@blackshark.ai>
Ran go mod tidy Signed-off-by: Nick Müller <nmueller@blackshark.ai>
Signed-off-by: Nick Müller <nmueller@blackshark.ai>
…oints Signed-off-by: Nick Müller <nmueller@blackshark.ai>
Signed-off-by: Nick Müller <nmueller@blackshark.ai>
Signed-off-by: Nick Müller <nmueller@blackshark.ai>
grpc-gateway parsing of URL params does not work for joined endpoint at the moment - fixed in major version upgrade Added extra CacheEvictionErrorCode enum entries Signed-off-by: Nick Müller <nmueller@blackshark.ai>
…artifacts to datacatalog Signed-off-by: Nick Müller <nmueller@blackshark.ai>
Signed-off-by: Nick Müller <nmueller@blackshark.ai>
Signed-off-by: Nick Müller <nmueller@blackshark.ai>
Signed-off-by: Nick Müller <nmueller@blackshark.ai>
Signed-off-by: Nick Müller <nmueller@blackshark.ai>
Signed-off-by: Nick Müller <nmueller@blackshark.ai>
Implement deleting of artifacts as bulk operation Signed-off-by: Nick Müller <nmueller@blackshark.ai>
Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>
Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #4655 +/- ##
==========================================
+ Coverage 59.00% 59.07% +0.07%
==========================================
Files 645 647 +2
Lines 55578 55972 +394
==========================================
+ Hits 32792 33065 +273
- Misses 20194 20301 +107
- Partials 2592 2606 +14
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
…on-past-executions' into feature/datacatalog-cache-deletion
Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>
Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>
Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>
Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>
Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>
pvditt
commented
Jan 31, 2024
Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>
Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>
…ations Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>
Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>
Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>
hamersaw
reviewed
Feb 13, 2024
Member
hamersaw
left a comment
There was a problem hiding this comment.
lets cleanup imports and resolve merge conflicts and then merge!
Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>
Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>
Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>
Signed-off-by: Paul Dittamo <pvdittamo@gmail.com>
Contributor
Author
|
put up change for console: flyteorg/flyteconsole#851 |
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tracking issue
#2867
Why are the changes needed?
Flyte doesn't support the deletion of cached task executions. Adding cache eviction/cleanup gives Flyte users more control over their data helping with things such as GDPR compliance + reducing their cache size.
Opted to support just single task executions as invalidating an entire workflow's cache can run into timeout issues going down the nesting of nodes.
What changes were proposed in this pull request?
How was this patch tested?
unit tests
Run workflow that caches task executions
Setup process
(NOTE - a follow up PR will be to get that change merged into console since console doesn't support CACHE_EVICT CatalogCacheStatus)
Screenshots
Check all the applicable boxes
Related PRs
Docs link