docs: add SOCI parallel pull best practices guide for faster Spark image pulls#1120
Merged
Conversation
…age pulls Signed-off-by: vara-bonthu <vara.bonthu@gmail.com>
nabuskey
reviewed
Jul 7, 2026
| ```bash | ||
| # EMR on EKS base images are on the public ECR gallery (6.9.0+), so this | ||
| # needs no auth. For any other image, point the command at your own tag. | ||
| docker manifest inspect public.ecr.aws/emr-on-eks/spark/emr-7.12.0:latest |
Collaborator
There was a problem hiding this comment.
This command doesn't return the information it claims to return.
docker manifest inspect public.ecr.aws/emr-on-eks/spark/emr-7.12.0:latest
{
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
"manifests": [
{
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"size": 5566,
"digest": "sha256:0f54eac08a4bee6c719f7b6b4b9087c21c26c43800f4ebc23cdf0f4d578a8ed3",
"platform": {
"architecture": "arm64",
"os": "linux",
"variant": "v8"
}
},
{
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"size": 5566,
"digest": "sha256:a22b0a0060a975eb0ea8cfd8c5e82955e93872f36591126a66385d86fffaf685",
"platform": {
"architecture": "amd64",
"os": "linux"
}
}
]
}
you need to do docker manifest inspect --verbose public.ecr.aws/emr-on-eks/spark/emr-7.12.0:latest
|
|
||
| ## SOCI parallel pull mode | ||
|
|
||
| The [SOCI snapshotter](https://github.com/awslabs/soci-snapshotter) added a parallel pull and unpack mode in **v0.11.0**. It splits large layers into chunks, downloads the chunks over many concurrent HTTP connections, and unpacks multiple layers at once. This is not lazy loading. It is a full, up-front pull that parallelizes the work. There is no image rebuild, no SOCI index artifacts, and no pipeline changes. It works on your existing images, from any registry, for any workload. |
Collaborator
There was a problem hiding this comment.
There are two places where it says This is not lazy loading. but it never defined what it is. I would rewrite this as:
The SOCI snapshotter (https://github.com/awslabs/soci-snapshotter) is best known for lazy loading, but its v0.11.0 parallel pull mode does the reverse. It runs a complete, up-front pull, splitting large layers into chunks, fetching them over many concurrent HTTP connections, and unpacking several layers at once. Because nothing about the image itself changes, there are no index artifacts to build and no pipeline steps to add, and it works against whatever registry you already use.
|
|
||
| The [SOCI snapshotter](https://github.com/awslabs/soci-snapshotter) added a parallel pull and unpack mode in **v0.11.0**. It splits large layers into chunks, downloads the chunks over many concurrent HTTP connections, and unpacks multiple layers at once. This is not lazy loading. It is a full, up-front pull that parallelizes the work. There is no image rebuild, no SOCI index artifacts, and no pipeline changes. It works on your existing images, from any registry, for any workload. | ||
|
|
||
| Recent Amazon EKS optimized AMIs for **Amazon Linux 2023 and Bottlerocket** ship the snapshotter built in, enabled through the `FastImagePull` nodeadm feature gate. `FastImagePull` is currently marked **experimental** by the EKS AMI project, so validate it on a staging node pool before rolling it out broadly. In our testing with a ~3 GB Spark image, enabling parallel pull together with the storage configuration below produced a large reduction in cold-pull time on fresh nodes. |
| kubectl describe pod <pod> | grep -A1 Pulled | ||
| ``` | ||
|
|
||
| The most common failure mode is a PoC run on an instance below the size threshold, where the feature gate no-ops and results look identical to baseline. |
Collaborator
There was a problem hiding this comment.
Is this actually a failure?
|
|
||
| Parallel pull attacks the pull. These attack the problem around it, roughly in order of impact: | ||
|
|
||
| **Slim your custom layers.** Your base image is fixed, but everything you add on top is yours to cut. For Spark, ship application jars and Python dependencies from S3 via `spark.jars` and `--py-files` instead of baking them into the image. Anything removed comes straight off every cold pull. See [Loading Spark dependencies from S3](#loading-spark-dependencies-from-s3) below for concrete examples. |
Collaborator
There was a problem hiding this comment.
Is this actually a good idea? You'd still have to download them so you are downloading multiple jars per executor. Node level caching can't be used. This is especially bad when you have multiple pods needing the same jars.
|
|
||
| **Extend node lifetime.** Review Karpenter consolidation and `expireAfter` settings. Longer lived nodes mean more cache hits and fewer cold pulls. If nodes churn every few minutes, pull time dominates regardless of how fast the pull is. | ||
|
|
||
| **Watch baseline network bandwidth.** At parallel pull speeds, smaller instances become network bound once ENA burst credits deplete. Baseline bandwidth, not burst, is what sustained scale-out sees. |
|
|
||
| ## Loading Spark dependencies from S3 | ||
|
|
||
| The biggest lever on cold-pull time you actually control is what you bake into the image. A connector, an application jar, and a few Python libraries can easily add a gigabyte that every fresh node re-pulls. Spark can fetch all of it from S3 at submit time instead, so those bytes never touch the image. |
Collaborator
There was a problem hiding this comment.
Same as above. I am not sure if this actually helps.
Co-authored-by: Manabu McCloskey <manabu.mccloskey@gmail.com>
Signed-off-by: vara-bonthu <vara.bonthu@gmail.com>
nabuskey
approved these changes
Jul 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
🛑 Please open an issue first to discuss any significant work and flesh out details/direction. When we triage the issues, we will add labels to the issue like "Enhancement", "Bug" which should indicate to you that this issue can be worked on and we are looking forward to your PR. We would hate for your time to be wasted.
Consult the CONTRIBUTING guide for submitting pull-requests.
FastImagePullnodeadm feature gate on AL2023 and Bottlerocket.Motivation
More
website/docsorwebsite/blogsection for this featurepre-commit run -awith this PR. Link for installing pre-commit locallyFor Moderators
Additional Notes