From f205b8317dc98dc3817832beb9ec1e031037d4c7 Mon Sep 17 00:00:00 2001 From: Nathan Hensley Date: Wed, 1 Jul 2026 11:24:00 -0700 Subject: [PATCH] feat(ci): nightly UAT version matrix + versioned install (#1274) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Turn the nightly UAT batch into a controller that runs a cross-version regression per reservation: main (built from source) plus the previous N stable releases, ordered by `uat-broker schedule` and time-boxed. - uat-nightly-batch.yaml: controller loop — one matrix leg per reservation (parallel), each expanding an ordered, deadline-bounded schedule and dispatching uat-run.yaml per cell sequentially (shared reservation lease), resolving the dispatched run by run-name and watching it; drop-oldest at the deadline; controller-side superseded reconcile. - uat-run.yaml: aicr_version input (dispatch + call), threaded to the per-cloud callers; run-name "UAT @ " so the controller can discover the dispatched run. - uat-aws.yaml / uat-gcp.yaml: aicr_version workflow_call input. main cells build from source; release cells install the released binary and let it self-resolve its version's validator/agent images (no image builds). - .github/actions/install-aicr-release: shared composite action that downloads, checksum-verifies, and provenance-verifies the released binary (cosign verify-blob-attestation against the archived SLSA Build Provenance v1 bundle, pinned to NVIDIA CI's keyless signer) before use — fail-closed. - docs/contributor/uat.md: version-matrix section, tunables, roadmap update. Related #1274, #1264. Signed-off-by: Nathan Hensley --- .../actions/install-aicr-release/action.yml | 111 ++++++++++ .github/workflows/uat-aws.yaml | 47 +++- .github/workflows/uat-gcp.yaml | 47 +++- .github/workflows/uat-nightly-batch.yaml | 204 +++++++++++++++--- .github/workflows/uat-run.yaml | 25 +++ docs/contributor/uat.md | 20 +- 6 files changed, 410 insertions(+), 44 deletions(-) create mode 100644 .github/actions/install-aicr-release/action.yml diff --git a/.github/actions/install-aicr-release/action.yml b/.github/actions/install-aicr-release/action.yml new file mode 100644 index 000000000..6b0b516e6 --- /dev/null +++ b/.github/actions/install-aicr-release/action.yml @@ -0,0 +1,111 @@ +# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +name: Install released aicr +description: >- + Download a released aicr binary, verify its integrity (checksum) and + provenance (SLSA Build Provenance v1 attestation, keyless-signed by NVIDIA's + on-tag.yaml release workflow), and install it to ./aicr for a UAT release + cell. Fails closed if the binary is unattested or the attestation does not + verify — a release cell never runs an unverified binary. The released binary + self-resolves its own version's validator images (pkg/validator/catalog + rewrites :latest -> :) and snapshot-agent (ghcr.io/nvidia/aicr:), + so a release cell builds no images. Shared by uat-aws.yaml and uat-gcp.yaml. + +inputs: + aicr-version: + description: 'Released aicr version tag to install (e.g. v1.2.3).' + required: true + +runs: + using: composite + steps: + # verify-blob-attestation needs no OIDC token (verification, not signing); + # it fetches only the Sigstore trusted root, so the caller's contents:read + # ceiling suffices. Pinned to the same SHA the release/build workflows use. + - name: Install Cosign + uses: sigstore/cosign-installer@6f9f17788090df1f26f669e9d70d6ae9567deba6 # v4.1.2 + - name: Download, verify, and install released aicr + shell: bash + env: + GH_TOKEN: ${{ github.token }} + AICR_VERSION: ${{ inputs.aicr-version }} + REPO: ${{ github.repository }} + run: | + set -euo pipefail + # The binary asset drops the leading v (aicr_1.2.3_linux_amd64.tar.gz) + # while image tags keep it; match the asset with a glob. + gh release download "$AICR_VERSION" --repo "$REPO" \ + --pattern 'aicr_*_linux_amd64.tar.gz' --pattern 'aicr_checksums.txt' --dir ./_rel + + # Pin provenance to NVIDIA CI's release workflow AND the exact requested + # tag. The signer SAN is the on-tag.yaml workflow that ran `cosign + # attest-blob` at that tag; binding the specific tag (not refs/tags/.*) + # stops a binary attested for one release from satisfying a different + # one. Validate the tag first (also rejects injection into the regex), + # then escape its dots so they match literally, and anchor the pattern. + if [[ ! "$AICR_VERSION" =~ ^v[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z][0-9A-Za-z.]*)?$ ]]; then + echo "::error::aicr-version must be a release tag like v1.2.3 (or v1.2.3-rc1); got '${AICR_VERSION}'." >&2 + exit 1 + fi + identity_tag="${AICR_VERSION//./\\.}" + identity_re="^https://github.com/${REPO}/.github/workflows/on-tag\\.yaml@refs/tags/${identity_tag}$" + + ( + cd ./_rel + set -- aicr_*_linux_amd64.tar.gz + asset="$1" + + # (1) Integrity: the tarball matches the release's published checksum. + # Necessary but not sufficient — aicr_checksums.txt ships in the same + # release, so it only catches download corruption, not a tampered + # release. Provenance (below) is the authenticity check. + grep " ${asset}\$" aicr_checksums.txt | sha256sum -c - + + # Extract the binary (required) plus its SLSA provenance bundle, which + # goreleaser ships inside the archive (archives.files, strip_parent). + tar -xzf "${asset}" aicr + tar -xzf "${asset}" aicr-attestation.sigstore.json 2>/dev/null || true + + # (2) Provenance: the exact binary bytes were built by NVIDIA's release + # workflow. cosign attest-blob's subject is the binary digest, so this + # binds authenticity to the installed artifact — not to the checksums + # manifest. Fail closed: a UAT release cell must not run an unattested + # or unverifiable binary. + if [ ! -f aicr-attestation.sigstore.json ]; then + echo "::error::release ${AICR_VERSION} ships no binary attestation (aicr-attestation.sigstore.json); refusing to run UAT against an unattested binary." >&2 + exit 1 + fi + # 3-attempt retry absorbs transient Sigstore trusted-root fetch flakes + # (mirrors the attest-blob retry in .goreleaser.yaml; see #1249). + for n in 1 2 3; do + if cosign verify-blob-attestation \ + --bundle aicr-attestation.sigstore.json \ + --type https://slsa.dev/provenance/v1 \ + --certificate-oidc-issuer https://token.actions.githubusercontent.com \ + --certificate-identity-regexp "$identity_re" \ + aicr; then + break + fi + if [ "$n" -eq 3 ]; then + echo "::error::cosign provenance verification failed for ${AICR_VERSION} after 3 attempts." >&2 + exit 1 + fi + echo "cosign verify attempt ${n} failed; retrying" >&2 + sleep $((n * 5)) + done + ) + + install -m 0755 ./_rel/aicr ./aicr + ./aicr --version diff --git a/.github/workflows/uat-aws.yaml b/.github/workflows/uat-aws.yaml index bcda04ad7..5de5933c6 100644 --- a/.github/workflows/uat-aws.yaml +++ b/.github/workflows/uat-aws.yaml @@ -28,6 +28,10 @@ on: description: 'Reservation name (lease key) this run targets; used for labeling.' type: string required: true + aicr_version: + description: 'Released aicr version to install (e.g. v1.2.3); empty = build from source (main tip).' + type: string + default: '' cluster_config_path: description: 'Cluster-config the EKS actuator provisions from (from infra/uat/reservations.yaml).' type: string @@ -102,6 +106,9 @@ jobs: # Build from source - name: Setup Go + # Main cells only: release cells install a prebuilt binary and must not + # depend on the source Go toolchain (a broken Go pin shouldn't fail them). + if: inputs.aicr_version == '' && inputs.skip_tests != true uses: actions/setup-go@924ae3a1cded613372ab5595356fb5720e22ba16 # v6.5.0 with: go-version: '${{ steps.versions.outputs.go }}' @@ -110,19 +117,35 @@ jobs: go.sum vendor/modules.txt + # Main cell (aicr_version empty): build the aicr binary from the + # checked-out source (tip). Release cells install the released binary + # instead (next step). - name: Build aicr binary - if: inputs.skip_tests != true + if: inputs.aicr_version == '' && inputs.skip_tests != true env: GOFLAGS: -mod=vendor run: | go build -o ./aicr ./cmd/aicr ./aicr --version + # Release cell (aicr_version set): install the RELEASED aicr binary at that + # version instead of building. The released binary self-resolves its own + # version's images — validators via pkg/validator/catalog (:latest -> + # :) and the snapshot-agent via ghcr.io/nvidia/aicr: — so + # the validator/agent build+push steps below are skipped for release cells. + - name: Install released aicr + if: inputs.aicr_version != '' && inputs.skip_tests != true + uses: ./.github/actions/install-aicr-release + with: + aicr-version: ${{ inputs.aicr_version }} + - name: Authenticate to GHCR uses: ./.github/actions/ghcr-login + # Main cell only: release cells use the released validator images the + # released binary self-resolves to (skipped when aicr_version is set). - name: Build and push validator images - if: inputs.skip_tests != true + if: inputs.aicr_version == '' && inputs.skip_tests != true run: | GO_VERSION="$(cat .go-version)" # CHAINSAW_* args dropped in #1236 — the deployment validator @@ -191,7 +214,10 @@ jobs: install_helmfile: 'true' helmfile_version: '${{ steps.versions.outputs.helmfile }}' helmfile_sha256: '${{ steps.versions.outputs.helmfile_sha256_linux_amd64 }}' - install_ko: 'true' + # ko builds the snapshot-agent image (main cells only). Release cells + # install a prebuilt binary and build no images, so skip ko — a bad ko + # pin or download must not fail a release cell before UAT starts. + install_ko: ${{ inputs.aicr_version == '' && inputs.skip_tests != true && 'true' || 'false' }} ko_version: ${{ steps.versions.outputs.ko }} # The snapshot agent runs in-cluster as its own image, separate from the @@ -204,7 +230,7 @@ jobs: # arch: the GPU worker (p5.48xlarge) is amd64. Lives in the aicr-validators # namespace (not the release ghcr.io/nvidia/aicr repo); cleaned up below. - name: Build and push snapshot-agent image - if: inputs.skip_tests != true + if: inputs.aicr_version == '' && inputs.skip_tests != true env: GOFLAGS: -mod=vendor KO_DOCKER_REPO: ghcr.io/nvidia/aicr-validators/agent @@ -267,9 +293,10 @@ jobs: env: AICR_BIN: ${{ github.workspace }}/aicr RUN_ID: ${{ github.run_id }} - # Snapshot agent = PR-built image (see "Build and push snapshot-agent - # image"), not the stale released default. Honored by `aicr snapshot`. - AICR_IMAGE: ghcr.io/nvidia/aicr-validators/agent:${{ env.VALIDATOR_TAG }} + # Snapshot agent: main cell = the PR-built image (see "Build and push + # snapshot-agent image"); release cell = the released aicr image at that + # version. Honored by `aicr snapshot`. + AICR_IMAGE: ${{ inputs.aicr_version == '' && format('ghcr.io/nvidia/aicr-validators/agent:{0}', env.VALIDATOR_TAG) || format('ghcr.io/nvidia/aicr:{0}', inputs.aicr_version) }} run: ./tests/uat/aws/run prep "${TEST_CONFIG}" - name: UAT - install (helmfile apply) @@ -379,11 +406,13 @@ jobs: if: always() env: SUMMARY_RESERVATION: ${{ inputs.reservation }} + SUMMARY_AICR_VERSION: ${{ inputs.aicr_version }} run: | { echo "## UAT Results (AWS)" echo "" printf '**Reservation:** `%s`\n' "$SUMMARY_RESERVATION" + printf '**AICR version:** `%s`\n' "${SUMMARY_AICR_VERSION:-main (build from source)}" echo "**Build:** \`${{ github.sha }}\` (branch: \`${{ github.ref_name }}\`)" echo "" echo "| Phase | Status |" @@ -405,8 +434,10 @@ jobs: } >> "$GITHUB_STEP_SUMMARY" # Cleanup (after artifacts so a failed cleanup does not lose evidence). + # Main cell only — release cells reference shared, permanent released image + # tags and must not delete them. - name: Cleanup validator image - if: always() && inputs.skip_tests != true + if: always() && inputs.aicr_version == '' && inputs.skip_tests != true env: GH_TOKEN: ${{ github.token }} run: | diff --git a/.github/workflows/uat-gcp.yaml b/.github/workflows/uat-gcp.yaml index 83491b6ac..140c1adff 100644 --- a/.github/workflows/uat-gcp.yaml +++ b/.github/workflows/uat-gcp.yaml @@ -28,6 +28,10 @@ on: description: 'Reservation name (lease key) this run targets; used for labeling.' type: string required: true + aicr_version: + description: 'Released aicr version to install (e.g. v1.2.3); empty = build from source (main tip).' + type: string + default: '' cluster_config_path: description: 'Cluster-config the GKE actuator provisions from (from infra/uat/reservations.yaml).' type: string @@ -100,6 +104,9 @@ jobs: # Build from source - name: Setup Go + # Main cells only: release cells install a prebuilt binary and must not + # depend on the source Go toolchain (a broken Go pin shouldn't fail them). + if: inputs.aicr_version == '' && inputs.skip_tests != true uses: actions/setup-go@924ae3a1cded613372ab5595356fb5720e22ba16 # v6.5.0 with: go-version: '${{ steps.versions.outputs.go }}' @@ -108,19 +115,35 @@ jobs: go.sum vendor/modules.txt + # Main cell (aicr_version empty): build the aicr binary from the + # checked-out source (tip). Release cells install the released binary + # instead (next step). - name: Build aicr binary - if: inputs.skip_tests != true + if: inputs.aicr_version == '' && inputs.skip_tests != true env: GOFLAGS: -mod=vendor run: | go build -o ./aicr ./cmd/aicr ./aicr --version + # Release cell (aicr_version set): install the RELEASED aicr binary at that + # version instead of building. The released binary self-resolves its own + # version's images — validators via pkg/validator/catalog (:latest -> + # :) and the snapshot-agent via ghcr.io/nvidia/aicr: — so + # the validator/agent build+push steps below are skipped for release cells. + - name: Install released aicr + if: inputs.aicr_version != '' && inputs.skip_tests != true + uses: ./.github/actions/install-aicr-release + with: + aicr-version: ${{ inputs.aicr_version }} + - name: Authenticate to GHCR uses: ./.github/actions/ghcr-login + # Main cell only: release cells use the released validator images the + # released binary self-resolves to (skipped when aicr_version is set). - name: Build and push validator images - if: inputs.skip_tests != true + if: inputs.aicr_version == '' && inputs.skip_tests != true run: | GO_VERSION="$(cat .go-version)" # CHAINSAW_* args dropped in #1236 — the deployment validator @@ -159,7 +182,10 @@ jobs: install_helmfile: 'true' helmfile_version: '${{ steps.versions.outputs.helmfile }}' helmfile_sha256: '${{ steps.versions.outputs.helmfile_sha256_linux_amd64 }}' - install_ko: 'true' + # ko builds the snapshot-agent image (main cells only). Release cells + # install a prebuilt binary and build no images, so skip ko — a bad ko + # pin or download must not fail a release cell before UAT starts. + install_ko: ${{ inputs.aicr_version == '' && inputs.skip_tests != true && 'true' || 'false' }} ko_version: ${{ steps.versions.outputs.ko }} # The snapshot agent runs in-cluster as its own image, separate from the @@ -173,7 +199,7 @@ jobs: # aicr-validators namespace (not the release ghcr.io/nvidia/aicr repo); # cleaned up below. - name: Build and push snapshot-agent image - if: inputs.skip_tests != true + if: inputs.aicr_version == '' && inputs.skip_tests != true env: GOFLAGS: -mod=vendor KO_DOCKER_REPO: ghcr.io/nvidia/aicr-validators/agent @@ -237,9 +263,10 @@ jobs: env: AICR_BIN: ${{ github.workspace }}/aicr RUN_ID: ${{ github.run_id }} - # Snapshot agent = PR-built image (see "Build and push snapshot-agent - # image"), not the stale released default. Honored by `aicr snapshot`. - AICR_IMAGE: ghcr.io/nvidia/aicr-validators/agent:${{ env.VALIDATOR_TAG }} + # Snapshot agent: main cell = the PR-built image (see "Build and push + # snapshot-agent image"); release cell = the released aicr image at that + # version. Honored by `aicr snapshot`. + AICR_IMAGE: ${{ inputs.aicr_version == '' && format('ghcr.io/nvidia/aicr-validators/agent:{0}', env.VALIDATOR_TAG) || format('ghcr.io/nvidia/aicr:{0}', inputs.aicr_version) }} run: ./tests/uat/gcp/run prep "${TEST_CONFIG}" - name: UAT - install (helmfile apply) @@ -352,11 +379,13 @@ jobs: if: always() env: SUMMARY_RESERVATION: ${{ inputs.reservation }} + SUMMARY_AICR_VERSION: ${{ inputs.aicr_version }} run: | { echo "## UAT Results (GCP)" echo "" printf '**Reservation:** `%s`\n' "$SUMMARY_RESERVATION" + printf '**AICR version:** `%s`\n' "${SUMMARY_AICR_VERSION:-main (build from source)}" echo "**Build:** \`${{ github.sha }}\` (branch: \`${{ github.ref_name }}\`)" echo "" echo "| Phase | Status |" @@ -377,8 +406,10 @@ jobs: } >> "$GITHUB_STEP_SUMMARY" # Cleanup (after artifacts so a failed cleanup does not lose evidence). + # Main cell only — release cells reference shared, permanent released image + # tags and must not delete them. - name: Cleanup validator image - if: always() && inputs.skip_tests != true + if: always() && inputs.aicr_version == '' && inputs.skip_tests != true env: GH_TOKEN: ${{ github.token }} run: | diff --git a/.github/workflows/uat-nightly-batch.yaml b/.github/workflows/uat-nightly-batch.yaml index cebebaaa1..626160139 100644 --- a/.github/workflows/uat-nightly-batch.yaml +++ b/.github/workflows/uat-nightly-batch.yaml @@ -14,20 +14,34 @@ name: UAT Nightly Batch -# The night side of the day/night cycle (#1274, DC1): on a nightly cron, run -# one UAT pass per reservation. Reservations are enumerated from the -# checked-in registry (infra/uat/reservations.yaml) so adding a reservation -# needs no change here. Each cell invokes the shared uat-run.yaml, which holds -# the per-reservation lease — so different reservations (independent hardware) -# run in parallel while runs on the SAME reservation serialize. +# The night side of the day/night cycle (#1274, DC1 + DC5): on a nightly cron, +# run the version matrix per reservation. For each reservation the controller +# expands an ordered, time-boxed schedule — main first, then the previous N +# stable releases in descending semver order (uat-broker schedule) — and runs +# the cells SEQUENTIALLY, dispatching the shared uat-run.yaml per cell and +# waiting for it before the next. Different reservations run in parallel +# (independent hardware); uat-run.yaml's per-reservation lease serializes runs +# that share a reservation. When the time-box closes, remaining cells are +# dropped oldest-release-first (they are ordered oldest-last, so the controller +# simply stops). Dispatch (not workflow_call) keeps uat-run.yaml top-level, so +# its own lease + superseded observer apply and the reusable-nesting depth stays +# at 3 (uat-run -> uat-{aws,gcp} -> evidence-ingest). # -# PR-2 scope: one main (build-from-source) pass per reservation. The -# time-boxed, main-first, previous-N-stable version matrix (uat-broker -# schedule) lands in the follow-up (DC1 PR-3 / DC5's versioned install). +# aicr_version is threaded per cell; the release cells install the released +# aicr + validator/agent images at that version (DC5), main builds from source. on: schedule: - cron: '0 4 * * *' # 04:00 UTC daily (evening in US Pacific); default branch only - workflow_dispatch: {} + workflow_dispatch: + inputs: + previous_n: + description: 'Previous stable releases below main to run per reservation (0 = main only).' + type: string + default: '2' + deadline_offset_hours: + description: 'Time-box: hours after batch start to stop dispatching new cells. Keep below the drive job timeout (GitHub caps a hosted job at 6h) so the graceful drop-oldest is reachable.' + type: string + default: '5' permissions: contents: read @@ -47,6 +61,9 @@ jobs: runs-on: ubuntu-latest outputs: reservations: ${{ steps.list.outputs.reservations }} + # Single batch-start timestamp shared by every matrix leg, so the time-box + # deadline is identical regardless of when each leg's runner starts. + batch_start: ${{ steps.list.outputs.batch_start }} steps: - name: Checkout uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 @@ -78,25 +95,162 @@ jobs: json=$(printf '%s\n' "${names[@]}" | jq -R . | jq -cs .) echo "reservations=${json}" >> "$GITHUB_OUTPUT" echo "Reservations: ${json}" + # Capture the batch start once; every leg derives its deadline from this. + echo "batch_start=$(date +%s)" >> "$GITHUB_OUTPUT" - # One UAT pass per reservation. Different reservations are independent - # hardware, so they run in parallel; uat-run.yaml's per-reservation lease - # serializes any run that shares a reservation (e.g. a concurrent ad-hoc - # dispatch). The caller grants the permission superset uat-run.yaml's nested - # pipeline + evidence-ingest need. - run: + # One matrix leg per reservation (parallel). Each leg drives its own version + # matrix sequentially. It only DISPATCHES uat-run.yaml (workflow_dispatch is + # exempt from the GITHUB_TOKEN recursion rule and always creates a run), so + # this job needs actions:write to dispatch + read to watch; the dispatched + # uat-run.yaml runs top-level with its own permissions. + drive: needs: enumerate + runs-on: ubuntu-latest + # The controller blocks on `gh run watch` per cell sequentially, so this job + # runs as long as the batch. GitHub hard-caps a hosted job at 6h; set an + # explicit timeout just under that so the job ends cleanly, and keep + # deadline_offset_hours below it so the graceful drop-oldest fires first — + # otherwise the job is killed mid-cell and the leg hard-fails. + timeout-minutes: 350 strategy: fail-fast: false matrix: reservation: ${{ fromJSON(needs.enumerate.outputs.reservations) }} - # Superset ceiling for the nested pipeline (uat-run.yaml → uat-{aws,gcp} - # → evidence-ingest); a reusable workflow's jobs cannot exceed it. permissions: - contents: read # checkout inside the called pipeline - actions: read # required by the nested uat-{aws,gcp} + evidence-ingest jobs - id-token: write # cloud OIDC/WIF: provision, keyless cosign, and GCS evidence publish - packages: write # push validator + snapshot-agent images to GHCR - uses: ./.github/workflows/uat-run.yaml - with: - reservation: ${{ matrix.reservation }} + contents: read # checkout + build uat-broker + actions: write # dispatch uat-run.yaml and watch the dispatched runs + steps: + - name: Checkout + uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + with: + persist-credentials: false + fetch-depth: 0 # full tag history for `git tag` -> uat-broker schedule + - name: Load versions + id: versions + uses: ./.github/actions/load-versions + - name: Setup Go + uses: actions/setup-go@924ae3a1cded613372ab5595356fb5720e22ba16 # v6.5.0 + with: + go-version: '${{ steps.versions.outputs.go }}' + cache: true + cache-dependency-path: | + go.sum + vendor/modules.txt + - name: Run version-matrix cells (sequential, time-boxed) + env: + GH_TOKEN: ${{ github.token }} + GOFLAGS: -mod=vendor + REPO: ${{ github.repository }} + REF: ${{ github.ref_name }} + RESERVATION: ${{ matrix.reservation }} + PREVIOUS_N: ${{ inputs.previous_n || '2' }} + DEADLINE_OFFSET_HOURS: ${{ inputs.deadline_offset_hours || '5' }} + BATCH_START: ${{ needs.enumerate.outputs.batch_start }} + run: | + set -euo pipefail + # Poll budget for finding a dispatched run (gh workflow run returns no + # run id): POLL_MAX_ATTEMPTS x POLL_INTERVAL_SECONDS seconds. + POLL_MAX_ATTEMPTS=24 + POLL_INTERVAL_SECONDS=5 + go build -o ./bin/uat-broker ./tools/uat-broker + + # Validate previous_n before the broker consumes it. `--previous-n` is + # an int flag so it rejects non-integers, but it would accept a + # negative and mis-shape the schedule; reject non-integer or negative + # values here too, symmetric with the deadline_offset_hours guard. + if [[ ! "$PREVIOUS_N" =~ ^[0-9]+$ ]]; then + echo "::error::previous_n must be a non-negative integer; got '${PREVIOUS_N}'." >&2 + exit 1 + fi + + # Ordered cells for this reservation: main first, then the previous-N + # stable releases in descending semver order (oldest last). + schedule=$(git tag -l 'v*' | ./bin/uat-broker schedule \ + --reservations "$RESERVATION" --previous-n "$PREVIOUS_N") + mapfile -t versions < <(jq -r --arg r "$RESERVATION" '.[$r][].aicr_version' <<<"$schedule") + echo "Cells for ${RESERVATION}: ${versions[*]:-}" + + # Reject a non-integer / negative deadline_offset_hours before the + # arithmetic below — workflow_dispatch passes it as a free-form string + # and Bash would silently accept e.g. -1, moving the cutoff before + # batch start and dropping every cell immediately. + if [[ ! "$DEADLINE_OFFSET_HOURS" =~ ^[0-9]+$ ]]; then + echo "::error::deadline_offset_hours must be a non-negative integer; got '${DEADLINE_OFFSET_HOURS}'." >&2 + exit 1 + fi + + # Anchor the time-box to the single batch-start timestamp (captured + # once in the enumerate job) so every reservation leg shares one cutoff + # regardless of when its runner started. + deadline=$(( BATCH_START + DEADLINE_OFFSET_HOURS * 3600 )) + cell=0 + leg_failed="" + for ver in "${versions[@]}"; do + if (( $(date +%s) >= deadline )); then + echo "::notice::Time-box reached — dropping remaining cells for ${RESERVATION} (oldest release first)." + break + fi + cell=$((cell + 1)) + label="${ver:-main}" + # Unique per-dispatch key (controller run id + attempt + reservation + # + cell index) so the resolver below watches THIS run and never a + # concurrent manual dispatch of the same reservation/version. + dispatch_key="${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}-${RESERVATION}-${cell}" + title="UAT ${RESERVATION} @ ${label} #${dispatch_key}" + echo "::group::${title}" + + gh workflow run uat-run.yaml --repo "$REPO" --ref "$REF" \ + -f reservation="$RESERVATION" -f aicr_version="$ver" \ + -f dispatch_key="$dispatch_key" + + # Resolve the dispatched run by its run-name — this title MUST match + # uat-run.yaml's `run-name:` (which appends the same dispatch_key). + # dispatch_key makes the title globally unique, so an exact title + # match is unambiguous; no createdAt filter (a runner clock running + # ahead of GitHub's API clock would falsely reject this dispatch). + run_id="" + for _ in $(seq 1 "$POLL_MAX_ATTEMPTS"); do + sleep "$POLL_INTERVAL_SECONDS" + run_id=$(gh run list --repo "$REPO" --workflow uat-run.yaml \ + --event workflow_dispatch --limit 50 \ + --json databaseId,displayTitle,createdAt 2>/dev/null \ + | jq -r --arg t "$title" \ + 'map(select(.displayTitle == $t)) + | sort_by(.createdAt) | .[-1].databaseId // empty' || true) + [[ -n "$run_id" ]] && break + done + if [[ -z "$run_id" ]]; then + # Fail the leg rather than continue: an unwatched run can still + # hold the reservation's pending slot and let a later (lower- + # priority) cell supersede it, breaking version ordering. + echo "::error::Dispatched ${title} but could not resolve its run id; stopping this reservation leg to preserve version ordering." + echo "::endgroup::" + exit 1 + fi + + echo "Waiting on run ${run_id} ..." + # --exit-status is intentionally not fatal here; classify the run's + # conclusion below so we can distinguish a benign superseded run from + # a real failure without aborting mid-loop. + gh run watch "$run_id" --repo "$REPO" --exit-status || true + + conclusion=$(gh run view "$run_id" --repo "$REPO" --json conclusion --jq '.conclusion' 2>/dev/null || echo "") + njobs=$(gh api "repos/${REPO}/actions/runs/${run_id}/jobs" --jq '.total_count' 2>/dev/null || echo "") + if [[ "$conclusion" == "cancelled" && "$njobs" == "0" ]]; then + # Benign: the single-slot reservation lease dropped a pending run. + # Surface it (must not be silent) but do not fail the leg — the + # superseded-run observer complements this (DC1 superseded surfacing). + echo "::warning::${title} was superseded while pending (dropped by the reservation lease); re-dispatch when the reservation frees." + elif [[ "$conclusion" != "success" ]]; then + # Real cell failure/timeout — record it so the leg (and thus the + # nightly batch) fails, but keep testing the remaining versions. + echo "::error::${title} finished with conclusion '${conclusion:-unknown}'." + leg_failed=1 + fi + echo "::endgroup::" + done + + if [[ -n "$leg_failed" ]]; then + echo "::error::One or more version cells failed for ${RESERVATION}." + exit 1 + fi diff --git a/.github/workflows/uat-run.yaml b/.github/workflows/uat-run.yaml index 3255a0bdc..c2f776c55 100644 --- a/.github/workflows/uat-run.yaml +++ b/.github/workflows/uat-run.yaml @@ -14,6 +14,15 @@ name: UAT Run +# A distinct run title per dispatch so the nightly controller can identify the +# run it dispatched — `gh workflow run` does not return a run id, so the +# controller matches on this display title. The controller supplies a unique +# dispatch_key (its run id + cell index) that is appended here, so a concurrent +# manual dispatch of the same reservation/version can never be mistaken for the +# controller's run. A manual dispatch omits the key and the suffix is dropped. +# Quoted so YAML does not treat the ` #` in the dispatch_key suffix as a comment. +run-name: "UAT ${{ inputs.reservation }} @ ${{ inputs.aicr_version || 'main' }}${{ inputs.dispatch_key != '' && format(' #{0}', inputs.dispatch_key) || '' }}" + # The shared UAT dispatch surface and reservation-lease owner (#1274, DC1). # A human (workflow_dispatch) or the nightly batch (workflow_call) requests a # run against a named reservation; this workflow holds the per-reservation @@ -29,6 +38,14 @@ on: description: 'Reservation name from infra/uat/reservations.yaml (e.g. aws-h100, gcp-h100).' type: string required: true + aicr_version: + description: 'Released aicr version to test (e.g. v1.2.3); empty = build from source (main tip).' + type: string + default: '' + dispatch_key: + description: 'Unique correlation key set by the nightly controller so it can find this run; leave empty for manual runs.' + type: string + default: '' skip_delete: description: 'Skip cluster teardown (manual debugging only).' type: boolean @@ -42,6 +59,12 @@ on: reservation: type: string required: true + aicr_version: + type: string + default: '' + dispatch_key: + type: string + default: '' skip_delete: type: boolean default: false @@ -129,6 +152,7 @@ jobs: uses: ./.github/workflows/uat-aws.yaml with: reservation: ${{ inputs.reservation }} + aicr_version: ${{ inputs.aicr_version }} cluster_config_path: ${{ needs.resolve.outputs.cluster_config_path }} test_config_dir: ${{ needs.resolve.outputs.test_config_dir }} skip_delete: ${{ inputs.skip_delete }} @@ -147,6 +171,7 @@ jobs: uses: ./.github/workflows/uat-gcp.yaml with: reservation: ${{ inputs.reservation }} + aicr_version: ${{ inputs.aicr_version }} cluster_config_path: ${{ needs.resolve.outputs.cluster_config_path }} test_config_dir: ${{ needs.resolve.outputs.test_config_dir }} skip_delete: ${{ inputs.skip_delete }} diff --git a/docs/contributor/uat.md b/docs/contributor/uat.md index afcaf73bd..6c6965183 100644 --- a/docs/contributor/uat.md +++ b/docs/contributor/uat.md @@ -6,7 +6,7 @@ Each reserved GPU pool follows a daily cycle, with every phase acquiring the *same* per-reservation lease so CI and human use never overlap on one reservation: -- **Night — the nightly batch.** On a cron, `uat-nightly-batch.yaml` runs one UAT pass per reservation (provision → CUJ → evidence → publish → teardown). +- **Night — the nightly batch.** On a cron, `uat-nightly-batch.yaml` runs the [version matrix](#the-version-matrix) per reservation — `main` plus the previous N stable releases — each cell a full provision → CUJ → evidence → publish → teardown. - **Morning — handoff.** Once the batch drains a reservation, the daytime human-access deployment is stood up on it (owned by DC2/DC8; not yet implemented). - **Day — human use.** The daytime cluster is used outside CI. - **Evening — teardown.** The daytime cluster is torn down before the next night batch. @@ -35,6 +35,21 @@ This replaces the previous behavior, where a second run hitting a busy AWS reser **The one-in-progress-plus-one-pending limit.** GitHub concurrency holds at most one in-progress run plus one pending run per group. If a *third* run is queued for a reservation that already has one in-progress and one pending, GitHub cancels the older pending run and the newest takes its place. At launch this is acceptable: there are two reservations, each contended by at most the nightly cron plus an occasional ad-hoc dispatch. A run cancelled this way is *superseded*, not failed. So that a dropped request is never silent, the `uat-superseded-notice.yaml` observer watches for it: triggered on `workflow_run: completed` for `UAT Run`, it classifies a cancelled run that never started a job as a supersede (versus a genuine mid-run cancel) and emits a job-summary entry plus a `::warning`. (The nightly controller reconciles the same signal synchronously for the cells it dispatches; a DC6 regression guard, #1279, will exercise the observer.) If deeper queuing is ever needed (many requesters per reservation), the escalation path is the *Deferred* standing broker service — a pull-based queue rather than GitHub concurrency — recorded in the epic (#1264). +## The version matrix + +The nightly batch runs a **cross-version regression** per reservation: `main` (built from source at tip) plus the previous **N** stable releases, so an older stable `aicr` is re-checked against today's cluster. `uat-broker schedule` orders the cells `main`-first, then releases in descending semver order; the controller runs them **sequentially** on the reservation (each cell dispatched through `uat-run.yaml`, so they share the lease) and **time-boxes** the batch — once the deadline passes it stops dispatching, so the in-flight cell finishes and the remaining (oldest) releases are dropped, guaranteeing `main` and the freshest releases always land. + +**Release cells install released artifacts, not source.** A `main` cell builds the `aicr` binary + validator/agent images from the checked-out tree. A release cell (`aicr_version=vX.Y.Z`) instead downloads the released `aicr` binary at that tag; the released binary self-resolves its own version's validator images (`…/aicr-validators/:vX.Y.Z`) and snapshot agent (`ghcr.io/nvidia/aicr:vX.Y.Z`), so no images are built for release cells. Each run's summary records its `aicr_version` (`main` or the tag). + +**Release cells verify what they install.** The `install-aicr-release` composite action does two checks before a downloaded binary is used, and **fails closed** on either: (1) *integrity* — the archive matches its `aicr_checksums.txt` entry; and (2) *provenance* — `cosign verify-blob-attestation` validates the SLSA Build Provenance v1 attestation goreleaser ships inside the archive (`aicr-attestation.sigstore.json`). The verifier does not trust *any* NVIDIA release signer: it derives the certificate-identity regexp from the requested `aicr_version`, so **only the attestation for that exact release tag** is accepted (`on-tag.yaml@refs/tags/`, issuer `token.actions.githubusercontent.com`) — an attestation for a different tag is rejected. The attestation's subject is the binary's own digest, so this also binds authenticity to the exact bytes that run — not to the same-release checksums manifest. A release whose binary is unattested, or whose attestation does not verify, aborts the cell rather than running an unverified `aicr`. + +**Tunables** — workflow inputs on `uat-nightly-batch.yaml` (these are the scheduled-run defaults): + +- `previous_n` — stable releases below `main` to run per reservation (default `2`; `0` = `main` only). +- `deadline_offset_hours` — hours after batch start to stop dispatching new cells (default `5`). The controller job watches each cell sequentially and GitHub caps a hosted job at 6h, so this stays below that ceiling (and the job's own `timeout-minutes`) to keep the graceful drop-oldest reachable rather than being killed mid-cell. + +To test a single released version by hand: `gh workflow run uat-run.yaml --repo NVIDIA/aicr --ref main -f reservation=aws-h100 -f aicr_version=v1.2.3`. (`--ref main` dispatches the nightly-path revision of the workflow, not your feature branch's.) + ## Adding a reservation Reservations are data, not code. To onboard a new reserved pool, add a row to `infra/uat/reservations.yaml`: @@ -55,7 +70,6 @@ The values in this file are identifiers, **not secrets** — a reservation-id gr ## Roadmap -What ships now is the lease, the data-driven dispatch surface, and a main-only nightly batch. Still to come: +What ships now is the lease, the data-driven dispatch surface, the time-boxed nightly version matrix (`main` + previous-N stable releases, release cells installing released artifacts), and superseded-run surfacing (the controller flags a dropped cell inline; the `uat-superseded-notice.yaml` observer catches ad-hoc dropped runs). Still to come: -- **Version matrix (DC1 follow-up / DC5).** The nightly batch will expand into a time-boxed, `main`-first, previous-N-stable-releases schedule (`uat-broker schedule`), each release row installing the released `aicr` images at that version. - **Per-intent shaping + daytime deployment (DC2 / DC8).** Selecting the test config and recipe by `intent`, and the morning-handoff daytime human-access cluster.