scripts: west_commands: blobs: support ZEPHYR_BLOBS_CACHE #99681
+35
−3
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
zephyrbot
requested review from
carlescufi,
marc-hb,
mbolivar,
pdgendt and
swinslow
In environment variable `ZEPHYR_BLOBS_CACHE` one or more directories (separated by `;`) can be specified. `west blobs fetch` will search in these folders for according file and copy it if it exists. If only one directory was given, it will automatically fill the cache by fetching the blob to this cache dir (with prefixed sha256 hashsum) and subsequently copy the file to the blobs path. Otherwise it fetches the blob as usual directly to the blobs path. Signed-off-by: Thorsten Klein <[email protected]>
thorsten-klein
force-pushed
the
west-blobs-cache
branch
from
ec18545 to
37f8b62
Compare
|
pdgendt
reviewed
Nov 19, 2025
| cached_filenames = [f'{sha256}-{path.name}', path.name] | ||
|
|
||
| # check if the blob is already cached | ||
| cache_dirs = os.getenv('ZEPHYR_BLOBS_CACHE', '') |
Contributor
There was a problem hiding this comment.
Try to avoid using environment variables where CLI arguments or west config options are perfectly viable.
Contributor
Author
There was a problem hiding this comment.
Yes, you are right. I will update it tomorrow to a west config option.
Any wishes for the option name?
zephyr.blobs-cache-dir?
Contributor
There was a problem hiding this comment.
Maybe blobs.cache-dir is sufficient?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds support for cache directories for blobs via the
ZEPHYR_BLOBS_CACHEenvironment variable.The variable may contain a single directory or multiple directories separated by
;.When running
west blobs fetch:If one or more cache directories are provided, the command searches them in order for the requested blob.
If the blob is found, it is copied to the according path. It searches for a file named as the blob, or the name prefixed with the sha256 hashsum.
If exactly one cache directory is provided, the cache is automatically filled:
when a blob is not found in the cache, it is fetched, stored in that cache directory (with its SHA-256 hash as a prefix), and then copied to the according path.
If multiple cache directories are provided, the command behaves like a lookup-only cache:
if the blob is not found in any of the specified directories, it is fetched directly into the blobs directory as usual (i.e., without populating the caches).
Why?
This feature can help to speed up builds and save network bandwidth in a CI environment (that start with an empty workspace), as blobs are cached and not re-downloaded.
It also helps to become independent from 3rd party servers, as all blobs can be pre-fetched and packaged and uploaded to own server.
Considered Alternative
There could be a remapping of the url from blobs done in a config file. But this is much more complex than this solution.