fix(ci): pull the KWOK images once per run instead of once per job (#2497)

Signed-off-by: Mark Chmarny <mark@chmarny.com>

Author: mchmarny

.github/actions/kwok-test/action.yml

@@ -167,6 +167,70 @@ runs:
         make build
         ls -la dist/
 
+    # ── Image cache: load the public images this lane needs from the tarball
+    # the prime-images job saved, instead of pulling them here (#2483).
+    #
+    # preload_image checks the host Docker cache before every pull attempt, so
+    # an image loaded here means the lane never contacts the registry at all —
+    # which is the point: 127 concurrent jobs pulling the same image is what
+    # trips the per-IP throttle.
+    #
+    # BEST EFFORT, unlike the prime job that fills the cache. Every step below
+    # tolerates a miss (fail-on-cache-miss defaults to false, and the load step
+    # swallows a non-zero), because the fallback is intact: preload_image pulls,
+    # and the kubelet pulls behind that. A fork PR whose cache scope differs, or
+    # a first run after a pin bump, must degrade to today's behavior rather than
+    # fail. The gate lives in prime-images, not here.
+    #
+    # Placed after setup-build-tools because resolving the pins needs yq.
+    - name: Resolve image pins and cache keys
+      id: images
+      shell: bash
+      run: |
+        set -euo pipefail
+        bash kwok/scripts/lib/image-cache.sh settings .settings.yaml | tee -a "$GITHUB_OUTPUT"
+
+    - name: Restore registry image
+      uses: actions/cache/restore@cdf6c1fa76f9f475f3d7449005a359c84ca0f306  # v5.0.3
+      with:
+        path: /tmp/kwok-image-cache/registry
+        key: ${{ steps.images.outputs.registry_key }}
+
+    # Gitea is installed only for the *-git deployers (see install-infra.sh), so
+    # the other lanes skip a ~250MB restore they would never use.
+    - name: Restore Gitea image
+      if: endsWith(inputs.deployer, '-git')
+      uses: actions/cache/restore@cdf6c1fa76f9f475f3d7449005a359c84ca0f306  # v5.0.3
+      with:
+        path: /tmp/kwok-image-cache/gitea
+        key: ${{ steps.images.outputs.gitea_key }}
+
+    - name: Load cached images
+      shell: bash
+      env:
+        REGISTRY_IMAGE: ${{ steps.images.outputs.registry_image }}
+        GITEA_IMAGE: ${{ steps.images.outputs.gitea_image }}
+        DEPLOYER: ${{ inputs.deployer }}
+      run: |
+        set -uo pipefail
+
+        # A miss is not fatal — preload_image still pulls — but it does mean
+        # this lane is back to contending for the registry, which is the whole
+        # point of #2483. Annotate it: a key that drifts from the prime job's
+        # would otherwise leave every lane pulling again, green, with the fix
+        # silently inert and nothing to notice it by.
+        load_image() { # <dir> <image>
+          if bash kwok/scripts/lib/image-cache.sh load "$1" "$2"; then
+            return 0
+          fi
+          echo "::warning::Image cache miss for $2 — this lane will pull it from the registry"
+        }
+
+        load_image /tmp/kwok-image-cache/registry "${REGISTRY_IMAGE}"
+        if [[ "${DEPLOYER}" == *-git ]]; then
+          load_image /tmp/kwok-image-cache/gitea "${GITEA_IMAGE}"
+        fi
+
     - name: Run KWOK recipe test
       id: validate
       shell: bash

.github/workflows/kwok-recipes.yaml

@@ -76,7 +76,13 @@ jobs:
     name: Discover Recipes
     if: github.event_name != 'schedule' || github.repository == 'nvidia/aicr'
     runs-on: ubuntu-latest
-    timeout-minutes: 5
+    # Raised from 5→10 when image-cache_test.sh joined the script unit tests
+    # below. Those suites are deliberately wall-clock bound — they assert that
+    # a hanging pull or an exhausted budget stays inside its deadline, which
+    # costs real seconds — and the block now runs ~90s locally before this job
+    # even starts classifying recipes. A discovery job that times out would be
+    # exactly the kind of self-inflicted red this change exists to remove.
+    timeout-minutes: 10
     outputs:
       tier1_pairs: ${{ steps.classify.outputs.tier1_pairs }}
       tier2_pairs: ${{ steps.classify.outputs.tier2_pairs }}
@@ -98,6 +104,7 @@ jobs:
           bash kwok/scripts/lib/sync-budget_test.sh
           bash kwok/scripts/lib/profile-select_test.sh
           bash kwok/scripts/lib/preload-image_test.sh
+          bash kwok/scripts/lib/image-cache_test.sh
           bash kwok/scripts/run-all-recipes_test.sh
 
       - name: Classify recipes into tiers
@@ -375,12 +382,75 @@ jobs:
           if (( dropped_count > 0 )); then
             echo "Filtered from matrix: ${dropped_count} recipe(s) with no matching KWOK profile — $(echo "$dropped" | jq -r 'join(", ")')"
           fi
+  # ── Image cache priming: one pull per run, not one per matrix job (#2483) ──
+  #
+  # The KWOK lanes need two public images — an in-cluster OCI registry, and
+  # Gitea for the *-git deployers. Every matrix job used to pull them itself.
+  # In run 33350102144, 127 jobs pulled registry:3.1.1 at the same moment from
+  # the shared GitHub-runner egress IPs; 125 succeeded and the one that was
+  # shed reddened main. That is a per-IP throttle, not an outage, so retrying
+  # harder cannot fix it — #2496 narrowed the window, it did not close it. The
+  # fix is to stop making 127 requests.
+  #
+  # This job makes one, saves a tarball, and lets actions/cache carry it to the
+  # matrix, which loads it from disk (see .github/actions/kwok-test). On a warm
+  # cache not even this job pulls.
+  #
+  # HARD GATE by design, unlike the best-effort preload it feeds: if an image is
+  # genuinely unreachable, failing one job in a minute beats 127 jobs each
+  # burning a rollout timeout. The summary job fails with it — without that, a
+  # failed prime would skip the tiers and the run would report green.
+  #
+  # One job per image (rather than one job for both) so a Gitea failure cannot
+  # discard a registry pull that already succeeded: actions/cache saves at the
+  # end of a job only when that job succeeds.
+  prime-images:
+    name: 'Prime Image Cache (${{ matrix.image }})'
+    if: github.event_name != 'schedule' || github.repository == 'nvidia/aicr'
+    runs-on: ubuntu-latest
+    timeout-minutes: 15
+    strategy:
+      fail-fast: false
+      matrix:
+        # Must match IMAGE_CACHE_IMAGES in kwok/scripts/lib/image-cache.sh.
+        image: [registry, gitea]
+    steps:
+      - name: Checkout
+        uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1  # v7.0.1
+        with:
+          persist-credentials: false
+
+      # .settings.yaml stays the single source of truth for the pins; the key
+      # derivation is shared with the consuming side through image-cache.sh, so
+      # the two cannot drift into never agreeing on a key.
+      - name: Resolve image pins and cache keys
+        id: images
+        shell: bash
+        run: |
+          set -euo pipefail
+          bash kwok/scripts/lib/image-cache.sh settings .settings.yaml | tee -a "$GITHUB_OUTPUT"
+
+      - name: Cache image tarball
+        uses: actions/cache@cdf6c1fa76f9f475f3d7449005a359c84ca0f306  # v5.0.3
+        with:
+          path: /tmp/kwok-image-cache/${{ matrix.image }}
+          key: ${{ steps.images.outputs[format('{0}_key', matrix.image)] }}
+
+      - name: Prime image
+        shell: bash
+        env:
+          IMAGE: ${{ steps.images.outputs[format('{0}_image', matrix.image)] }}
+          CACHE_DIR: /tmp/kwok-image-cache/${{ matrix.image }}
+        run: |
+          set -euo pipefail
+          bash kwok/scripts/lib/image-cache.sh save "${CACHE_DIR}" "${IMAGE}"
+
   # Thin caller of the shared kwok-test-run.yaml reusable workflow (#1172).
   # tier1_pairs is well under 256 entries, so it's passed in a single call —
   # no batching needed.
   test-tier1:
     name: 'Tier 1'
-    needs: discover
+    needs: [discover, prime-images]
     if: >-
       github.event_name != 'schedule' &&
       needs.discover.outputs.tier1_pairs != '[]' &&
@@ -393,7 +463,7 @@ jobs:
   # Helm-only by deliberate policy — see ADR-003 "Tier 2 deployer coverage".
   test-tier2:
     name: 'Tier 2'
-    needs: discover
+    needs: [discover, prime-images]
     if: >-
       github.event_name == 'pull_request' &&
       needs.discover.outputs.tier2_pairs != '[]' &&
@@ -411,7 +481,7 @@ jobs:
   # single run in its own group so they all run in parallel.
   test-tier3:
     name: 'Tier 3'
-    needs: discover
+    needs: [discover, prime-images]
     concurrency:
       group: kwok-tier3-${{ github.sha }}-${{ matrix.batch.id }}
       cancel-in-progress: false
@@ -430,7 +500,7 @@ jobs:
   # ── Summary: aggregate all tiers ──
   summary:
     name: KWOK Test Summary
-    needs: [test-tier1, test-tier2, test-tier3]
+    needs: [prime-images, test-tier1, test-tier2, test-tier3]
     runs-on: ubuntu-latest
     timeout-minutes: 5
     if: always()
@@ -440,19 +510,31 @@ jobs:
           echo "## KWOK Cluster Validation Summary" >> $GITHUB_STEP_SUMMARY
           echo "" >> $GITHUB_STEP_SUMMARY
 
+          prime="${{ needs.prime-images.result }}"
           tier1="${{ needs.test-tier1.result }}"
           tier2="${{ needs.test-tier2.result }}"
           tier3="${{ needs.test-tier3.result }}"
 
-          echo "| Tier | Result |" >> $GITHUB_STEP_SUMMARY
+          echo "| Stage | Result |" >> $GITHUB_STEP_SUMMARY
           echo "|------|--------|" >> $GITHUB_STEP_SUMMARY
+          echo "| Image cache priming | ${prime} |" >> $GITHUB_STEP_SUMMARY
           echo "| Tier 1 (generic) | ${tier1} |" >> $GITHUB_STEP_SUMMARY
           echo "| Tier 2 (diff-aware) | ${tier2} |" >> $GITHUB_STEP_SUMMARY
           echo "| Tier 3 (full matrix) | ${tier3} |" >> $GITHUB_STEP_SUMMARY
           echo "" >> $GITHUB_STEP_SUMMARY
 
           failed=false
 
+          # Priming gates every tier, so its failure must be checked HERE and
+          # not inferred from the tiers. A failed prime leaves them "skipped",
+          # and the tier checks below deliberately tolerate "skipped" — so
+          # without this the whole run would report green having validated
+          # nothing.
+          if [[ "$prime" == "failure" || "$prime" == "cancelled" ]]; then
+            echo "Image cache priming failed or was cancelled — no tier ran" >> $GITHUB_STEP_SUMMARY
+            failed=true
+          fi
+
           # Tier 1 must pass (unless skipped on schedule-only with no recipes)
           if [[ "$tier1" == "failure" || "$tier1" == "cancelled" ]]; then
             echo "Tier 1 (generic) failed or was cancelled" >> $GITHUB_STEP_SUMMARY

kwok/README.md

@@ -231,6 +231,24 @@ Manual trigger:
 gh workflow run kwok-recipes.yaml -f recipe=your-recipe-name
 ```
 
+### Public image cache
+
+The lanes need two images from public registries — the in-cluster OCI registry and, for the `*-git` deployers, Gitea. Both are pinned in `.settings.yaml` under `testing_tools`.
+
+A full Tier 3 run fans out to well over a hundred concurrent jobs, and a job that pulls these itself competes with every sibling for the same per-IP quota at the registry: 127 jobs pulling at once reliably gets one or two shed, which reddens the run with nothing wrong in the repo (#2483). So CI pulls each image exactly once, in the `prime-images` job, and carries it to the matrix as a tarball in `actions/cache`. Each test job loads from that tarball, and `preload_image` then finds the image already in the host Docker cache and never contacts the registry.
+
+`kwok/scripts/lib/image-cache.sh` owns both ends, so the priming job and the test jobs derive the cache key from the same code:
+
+```bash
+bash kwok/scripts/lib/image-cache.sh settings .settings.yaml   # resolve pins + keys
+bash kwok/scripts/lib/image-cache.sh save <dir> <image>        # pull once, write the tarball
+bash kwok/scripts/lib/image-cache.sh load <dir> <image>        # restore it into Docker
+```
+
+Priming is a hard gate: if an image is genuinely unreachable, the run fails there rather than in every lane. Loading is best effort — a cache miss falls back to `preload_image`'s pull, and the kubelet pull behind that, so a cold cache degrades to the old behavior instead of failing.
+
+Local runs (`make kwok-test-all`) do not use the cache; they pull through `preload_image` as before.
+
 ## Troubleshooting
 
 **Pods stuck Pending** — check tolerations and node selectors: