Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
111 changes: 111 additions & 0 deletions .github/actions/install-aicr-release/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,111 @@
# Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

name: Install released aicr
description: >-
Download a released aicr binary, verify its integrity (checksum) and
provenance (SLSA Build Provenance v1 attestation, keyless-signed by NVIDIA's
on-tag.yaml release workflow), and install it to ./aicr for a UAT release
cell. Fails closed if the binary is unattested or the attestation does not
verify — a release cell never runs an unverified binary. The released binary
self-resolves its own version's validator images (pkg/validator/catalog
rewrites :latest -> :<version>) and snapshot-agent (ghcr.io/nvidia/aicr:<version>),
so a release cell builds no images. Shared by uat-aws.yaml and uat-gcp.yaml.

inputs:
aicr-version:
description: 'Released aicr version tag to install (e.g. v1.2.3).'
required: true

runs:
using: composite
steps:
# verify-blob-attestation needs no OIDC token (verification, not signing);
# it fetches only the Sigstore trusted root, so the caller's contents:read
# ceiling suffices. Pinned to the same SHA the release/build workflows use.
- name: Install Cosign
uses: sigstore/cosign-installer@6f9f17788090df1f26f669e9d70d6ae9567deba6 # v4.1.2
- name: Download, verify, and install released aicr
shell: bash
env:
GH_TOKEN: ${{ github.token }}
AICR_VERSION: ${{ inputs.aicr-version }}
REPO: ${{ github.repository }}
run: |
set -euo pipefail
# The binary asset drops the leading v (aicr_1.2.3_linux_amd64.tar.gz)
# while image tags keep it; match the asset with a glob.
gh release download "$AICR_VERSION" --repo "$REPO" \
--pattern 'aicr_*_linux_amd64.tar.gz' --pattern 'aicr_checksums.txt' --dir ./_rel

# Pin provenance to NVIDIA CI's release workflow AND the exact requested
# tag. The signer SAN is the on-tag.yaml workflow that ran `cosign
# attest-blob` at that tag; binding the specific tag (not refs/tags/.*)
# stops a binary attested for one release from satisfying a different
# one. Validate the tag first (also rejects injection into the regex),
# then escape its dots so they match literally, and anchor the pattern.
if [[ ! "$AICR_VERSION" =~ ^v[0-9]+\.[0-9]+\.[0-9]+(-[0-9A-Za-z][0-9A-Za-z.]*)?$ ]]; then
echo "::error::aicr-version must be a release tag like v1.2.3 (or v1.2.3-rc1); got '${AICR_VERSION}'." >&2
exit 1
fi
identity_tag="${AICR_VERSION//./\\.}"
identity_re="^https://github.com/${REPO}/.github/workflows/on-tag\\.yaml@refs/tags/${identity_tag}$"

(
cd ./_rel
set -- aicr_*_linux_amd64.tar.gz
asset="$1"

# (1) Integrity: the tarball matches the release's published checksum.
# Necessary but not sufficient — aicr_checksums.txt ships in the same
# release, so it only catches download corruption, not a tampered
# release. Provenance (below) is the authenticity check.
grep " ${asset}\$" aicr_checksums.txt | sha256sum -c -

# Extract the binary (required) plus its SLSA provenance bundle, which
# goreleaser ships inside the archive (archives.files, strip_parent).
tar -xzf "${asset}" aicr
tar -xzf "${asset}" aicr-attestation.sigstore.json 2>/dev/null || true

# (2) Provenance: the exact binary bytes were built by NVIDIA's release
# workflow. cosign attest-blob's subject is the binary digest, so this
# binds authenticity to the installed artifact — not to the checksums
# manifest. Fail closed: a UAT release cell must not run an unattested
# or unverifiable binary.
if [ ! -f aicr-attestation.sigstore.json ]; then
echo "::error::release ${AICR_VERSION} ships no binary attestation (aicr-attestation.sigstore.json); refusing to run UAT against an unattested binary." >&2
exit 1
fi
# 3-attempt retry absorbs transient Sigstore trusted-root fetch flakes
# (mirrors the attest-blob retry in .goreleaser.yaml; see #1249).
for n in 1 2 3; do
if cosign verify-blob-attestation \
--bundle aicr-attestation.sigstore.json \
--type https://slsa.dev/provenance/v1 \
--certificate-oidc-issuer https://token.actions.githubusercontent.com \
--certificate-identity-regexp "$identity_re" \
aicr; then
break
fi
if [ "$n" -eq 3 ]; then
echo "::error::cosign provenance verification failed for ${AICR_VERSION} after 3 attempts." >&2
exit 1
fi
echo "cosign verify attempt ${n} failed; retrying" >&2
sleep $((n * 5))
done
)

install -m 0755 ./_rel/aicr ./aicr
./aicr --version
47 changes: 39 additions & 8 deletions .github/workflows/uat-aws.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,10 @@ on:
description: 'Reservation name (lease key) this run targets; used for labeling.'
type: string
required: true
aicr_version:
description: 'Released aicr version to install (e.g. v1.2.3); empty = build from source (main tip).'
type: string
default: ''
cluster_config_path:
description: 'Cluster-config the EKS actuator provisions from (from infra/uat/reservations.yaml).'
type: string
Expand Down Expand Up @@ -102,6 +106,9 @@ jobs:

# Build from source
- name: Setup Go
# Main cells only: release cells install a prebuilt binary and must not
# depend on the source Go toolchain (a broken Go pin shouldn't fail them).
if: inputs.aicr_version == '' && inputs.skip_tests != true
uses: actions/setup-go@924ae3a1cded613372ab5595356fb5720e22ba16 # v6.5.0
with:
go-version: '${{ steps.versions.outputs.go }}'
Expand All @@ -110,19 +117,35 @@ jobs:
go.sum
vendor/modules.txt

# Main cell (aicr_version empty): build the aicr binary from the
# checked-out source (tip). Release cells install the released binary
# instead (next step).
- name: Build aicr binary
if: inputs.skip_tests != true
if: inputs.aicr_version == '' && inputs.skip_tests != true
env:
GOFLAGS: -mod=vendor
run: |
go build -o ./aicr ./cmd/aicr
./aicr --version

# Release cell (aicr_version set): install the RELEASED aicr binary at that
# version instead of building. The released binary self-resolves its own
# version's images — validators via pkg/validator/catalog (:latest ->
# :<version>) and the snapshot-agent via ghcr.io/nvidia/aicr:<version> — so
# the validator/agent build+push steps below are skipped for release cells.
- name: Install released aicr
if: inputs.aicr_version != '' && inputs.skip_tests != true
uses: ./.github/actions/install-aicr-release
with:
aicr-version: ${{ inputs.aicr_version }}

- name: Authenticate to GHCR
uses: ./.github/actions/ghcr-login

# Main cell only: release cells use the released validator images the
# released binary self-resolves to (skipped when aicr_version is set).
- name: Build and push validator images
if: inputs.skip_tests != true
if: inputs.aicr_version == '' && inputs.skip_tests != true
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Comment thread
coderabbitai[bot] marked this conversation as resolved.
run: |
GO_VERSION="$(cat .go-version)"
# CHAINSAW_* args dropped in #1236 — the deployment validator
Expand Down Expand Up @@ -191,7 +214,10 @@ jobs:
install_helmfile: 'true'
helmfile_version: '${{ steps.versions.outputs.helmfile }}'
helmfile_sha256: '${{ steps.versions.outputs.helmfile_sha256_linux_amd64 }}'
install_ko: 'true'
# ko builds the snapshot-agent image (main cells only). Release cells
# install a prebuilt binary and build no images, so skip ko — a bad ko
# pin or download must not fail a release cell before UAT starts.
install_ko: ${{ inputs.aicr_version == '' && inputs.skip_tests != true && 'true' || 'false' }}
ko_version: ${{ steps.versions.outputs.ko }}

# The snapshot agent runs in-cluster as its own image, separate from the
Expand All @@ -204,7 +230,7 @@ jobs:
# arch: the GPU worker (p5.48xlarge) is amd64. Lives in the aicr-validators
# namespace (not the release ghcr.io/nvidia/aicr repo); cleaned up below.
- name: Build and push snapshot-agent image
if: inputs.skip_tests != true
if: inputs.aicr_version == '' && inputs.skip_tests != true
env:
GOFLAGS: -mod=vendor
KO_DOCKER_REPO: ghcr.io/nvidia/aicr-validators/agent
Expand Down Expand Up @@ -267,9 +293,10 @@ jobs:
env:
AICR_BIN: ${{ github.workspace }}/aicr
RUN_ID: ${{ github.run_id }}
# Snapshot agent = PR-built image (see "Build and push snapshot-agent
# image"), not the stale released default. Honored by `aicr snapshot`.
AICR_IMAGE: ghcr.io/nvidia/aicr-validators/agent:${{ env.VALIDATOR_TAG }}
# Snapshot agent: main cell = the PR-built image (see "Build and push
# snapshot-agent image"); release cell = the released aicr image at that
# version. Honored by `aicr snapshot`.
AICR_IMAGE: ${{ inputs.aicr_version == '' && format('ghcr.io/nvidia/aicr-validators/agent:{0}', env.VALIDATOR_TAG) || format('ghcr.io/nvidia/aicr:{0}', inputs.aicr_version) }}
run: ./tests/uat/aws/run prep "${TEST_CONFIG}"

- name: UAT - install (helmfile apply)
Expand Down Expand Up @@ -379,11 +406,13 @@ jobs:
if: always()
env:
SUMMARY_RESERVATION: ${{ inputs.reservation }}
SUMMARY_AICR_VERSION: ${{ inputs.aicr_version }}
run: |
{
echo "## UAT Results (AWS)"
echo ""
printf '**Reservation:** `%s`\n' "$SUMMARY_RESERVATION"
printf '**AICR version:** `%s`\n' "${SUMMARY_AICR_VERSION:-main (build from source)}"
echo "**Build:** \`${{ github.sha }}\` (branch: \`${{ github.ref_name }}\`)"
echo ""
echo "| Phase | Status |"
Expand All @@ -405,8 +434,10 @@ jobs:
} >> "$GITHUB_STEP_SUMMARY"

# Cleanup (after artifacts so a failed cleanup does not lose evidence).
# Main cell only — release cells reference shared, permanent released image
# tags and must not delete them.
- name: Cleanup validator image
if: always() && inputs.skip_tests != true
if: always() && inputs.aicr_version == '' && inputs.skip_tests != true
env:
GH_TOKEN: ${{ github.token }}
run: |
Expand Down
47 changes: 39 additions & 8 deletions .github/workflows/uat-gcp.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,10 @@ on:
description: 'Reservation name (lease key) this run targets; used for labeling.'
type: string
required: true
aicr_version:
description: 'Released aicr version to install (e.g. v1.2.3); empty = build from source (main tip).'
type: string
default: ''
cluster_config_path:
description: 'Cluster-config the GKE actuator provisions from (from infra/uat/reservations.yaml).'
type: string
Expand Down Expand Up @@ -100,6 +104,9 @@ jobs:

# Build from source
- name: Setup Go
# Main cells only: release cells install a prebuilt binary and must not
# depend on the source Go toolchain (a broken Go pin shouldn't fail them).
if: inputs.aicr_version == '' && inputs.skip_tests != true
uses: actions/setup-go@924ae3a1cded613372ab5595356fb5720e22ba16 # v6.5.0
with:
go-version: '${{ steps.versions.outputs.go }}'
Expand All @@ -108,19 +115,35 @@ jobs:
go.sum
vendor/modules.txt

# Main cell (aicr_version empty): build the aicr binary from the
# checked-out source (tip). Release cells install the released binary
# instead (next step).
- name: Build aicr binary
if: inputs.skip_tests != true
if: inputs.aicr_version == '' && inputs.skip_tests != true
env:
GOFLAGS: -mod=vendor
run: |
go build -o ./aicr ./cmd/aicr
./aicr --version

# Release cell (aicr_version set): install the RELEASED aicr binary at that
# version instead of building. The released binary self-resolves its own
# version's images — validators via pkg/validator/catalog (:latest ->
# :<version>) and the snapshot-agent via ghcr.io/nvidia/aicr:<version> — so
# the validator/agent build+push steps below are skipped for release cells.
- name: Install released aicr
if: inputs.aicr_version != '' && inputs.skip_tests != true
uses: ./.github/actions/install-aicr-release
with:
aicr-version: ${{ inputs.aicr_version }}

- name: Authenticate to GHCR
uses: ./.github/actions/ghcr-login

# Main cell only: release cells use the released validator images the
# released binary self-resolves to (skipped when aicr_version is set).
- name: Build and push validator images
if: inputs.skip_tests != true
if: inputs.aicr_version == '' && inputs.skip_tests != true
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Comment thread
coderabbitai[bot] marked this conversation as resolved.
run: |
GO_VERSION="$(cat .go-version)"
# CHAINSAW_* args dropped in #1236 — the deployment validator
Expand Down Expand Up @@ -159,7 +182,10 @@ jobs:
install_helmfile: 'true'
helmfile_version: '${{ steps.versions.outputs.helmfile }}'
helmfile_sha256: '${{ steps.versions.outputs.helmfile_sha256_linux_amd64 }}'
install_ko: 'true'
# ko builds the snapshot-agent image (main cells only). Release cells
# install a prebuilt binary and build no images, so skip ko — a bad ko
# pin or download must not fail a release cell before UAT starts.
install_ko: ${{ inputs.aicr_version == '' && inputs.skip_tests != true && 'true' || 'false' }}
ko_version: ${{ steps.versions.outputs.ko }}

# The snapshot agent runs in-cluster as its own image, separate from the
Expand All @@ -173,7 +199,7 @@ jobs:
# aicr-validators namespace (not the release ghcr.io/nvidia/aicr repo);
# cleaned up below.
- name: Build and push snapshot-agent image
if: inputs.skip_tests != true
if: inputs.aicr_version == '' && inputs.skip_tests != true
env:
GOFLAGS: -mod=vendor
KO_DOCKER_REPO: ghcr.io/nvidia/aicr-validators/agent
Expand Down Expand Up @@ -237,9 +263,10 @@ jobs:
env:
AICR_BIN: ${{ github.workspace }}/aicr
RUN_ID: ${{ github.run_id }}
# Snapshot agent = PR-built image (see "Build and push snapshot-agent
# image"), not the stale released default. Honored by `aicr snapshot`.
AICR_IMAGE: ghcr.io/nvidia/aicr-validators/agent:${{ env.VALIDATOR_TAG }}
# Snapshot agent: main cell = the PR-built image (see "Build and push
# snapshot-agent image"); release cell = the released aicr image at that
# version. Honored by `aicr snapshot`.
AICR_IMAGE: ${{ inputs.aicr_version == '' && format('ghcr.io/nvidia/aicr-validators/agent:{0}', env.VALIDATOR_TAG) || format('ghcr.io/nvidia/aicr:{0}', inputs.aicr_version) }}
run: ./tests/uat/gcp/run prep "${TEST_CONFIG}"

- name: UAT - install (helmfile apply)
Expand Down Expand Up @@ -352,11 +379,13 @@ jobs:
if: always()
env:
SUMMARY_RESERVATION: ${{ inputs.reservation }}
SUMMARY_AICR_VERSION: ${{ inputs.aicr_version }}
run: |
{
echo "## UAT Results (GCP)"
echo ""
printf '**Reservation:** `%s`\n' "$SUMMARY_RESERVATION"
printf '**AICR version:** `%s`\n' "${SUMMARY_AICR_VERSION:-main (build from source)}"
echo "**Build:** \`${{ github.sha }}\` (branch: \`${{ github.ref_name }}\`)"
echo ""
echo "| Phase | Status |"
Expand All @@ -377,8 +406,10 @@ jobs:
} >> "$GITHUB_STEP_SUMMARY"

# Cleanup (after artifacts so a failed cleanup does not lose evidence).
# Main cell only — release cells reference shared, permanent released image
# tags and must not delete them.
- name: Cleanup validator image
if: always() && inputs.skip_tests != true
if: always() && inputs.aicr_version == '' && inputs.skip_tests != true
env:
GH_TOKEN: ${{ github.token }}
run: |
Expand Down
Loading
Loading