feat(bundler): wire DRA eviction for GPU driver upgrades (#2401)

Signed-off-by: Manav Kamleshbhai Dhamani <35000278+MDhamani@users.noreply.github.com>
Co-authored-by: Mark Chmarny <mchmarny@users.noreply.github.com>

Author: MDhamani

.github/workflows/gpu-h100-kind-runtime-test.yaml

@@ -85,6 +85,16 @@ jobs:
           control_plane_resource_patches: 'true'
           control_plane_leader_election_tuning: 'true'
 
+      - name: Label nodes for the DRA kubelet plugin
+        shell: bash
+        env:
+          DRA_EVICTION_NODE_LABEL: nvidia.com/dra-kubelet-plugin=true
+        run: >-
+          kubectl label nodes
+          --selector='!node-role.kubernetes.io/control-plane'
+          "${DRA_EVICTION_NODE_LABEL}"
+          --overwrite
+
       - name: Build AICR CLI
         timeout-minutes: 10
         uses: ./.github/actions/aicr-build

api/aicr/v1/server.yaml

@@ -1285,6 +1285,7 @@ paths:
         - $ref: "#/components/parameters/SystemNodeToleration"
         - $ref: "#/components/parameters/AcceleratedNodeSelector"
         - $ref: "#/components/parameters/AcceleratedNodeToleration"
+        - $ref: "#/components/parameters/DRAEvictionNodeLabel"
         - $ref: "#/components/parameters/BundleDeployer"
         - $ref: "#/components/parameters/BundleRepo"
         - $ref: "#/components/parameters/WorkloadGate"
@@ -1498,9 +1499,11 @@ paths:
 
             A declaration whose component is absent from the generated
             bundle is rejected with HTTP 400 (INVALID_REQUEST, "cannot
-            take effect"); no path is exempt. Certain gate-verified paths
-            on present components are also rejected — driver-ownership
-            paths, GPU allocation-policy keys, and, where the
+            take effect"); no path is exempt. Certain gate- or contract-owned
+            paths on present components are also rejected — driver-ownership
+            paths, GPU allocation-policy keys, the DRA eviction paths
+            kubeletPlugin.nodeSelector and driver.manager.env when both
+            contract components are enabled, and, where the
             corresponding NVSentinel gate applies on the recipe's
             platform and configuration, the NVSentinel
             remedy/consumer/runtime-class paths — because an install-time
@@ -1570,6 +1573,7 @@ paths:
           style: form
           explode: true
           example: ["nvidia.com/gpu=present:NoSchedule"]
+        - $ref: "#/components/parameters/DRAEvictionNodeLabel"
         - name: deployer
           in: query
           required: false
@@ -2094,11 +2098,13 @@ components:
       description: >-
         Repeatable install-time component:path declaration. Rejected with
         HTTP 400 when the component is absent from the generated bundle;
-        no path is exempt. Certain gate-verified paths on present
+        no path is exempt. Certain gate- or contract-owned paths on present
         components are also rejected (driver-ownership paths, GPU
-        allocation-policy keys, and — where the corresponding NVSentinel
-        gate applies on the recipe's platform and configuration — the
-        NVSentinel remedy/consumer/runtime-class paths).
+        allocation-policy keys, the DRA eviction paths
+        kubeletPlugin.nodeSelector and driver.manager.env when both contract
+        components are enabled, and — where the corresponding NVSentinel gate
+        applies on the recipe's platform and configuration — the NVSentinel
+        remedy/consumer/runtime-class paths).
       schema:
         type: array
         items:
@@ -2149,6 +2155,18 @@ components:
           type: string
       style: form
       explode: true
+    DRAEvictionNodeLabel:
+      name: dra-eviction-node-label
+      in: query
+      required: false
+      description: >
+        Single key=value node label coordinating NVIDIA DRA kubelet-plugin
+        eviction with GPU Operator driver upgrades. Applied only when both
+        components are enabled.
+      schema:
+        type: string
+        default: nvidia.com/dra-kubelet-plugin=true
+      example: "example.com/dra-ready=enabled"
     BundleDeployer:
       name: deployer
       in: query

docs/contributor/component.md

@@ -233,6 +233,18 @@ Merge order is base → `valuesFile` → overlay `overrides` →
 recipe default. The node-class flags (`--system-node-selector` et al.)
 write into their routed paths at bundle time, alongside `--set`.
 
+The narrow exception is a **bundler-owned derived integration value**:
+when AICR renders both sides of a cross-chart contract and allowing either
+side to drift would produce an invalid bundle, the bundler may enforce that
+contract after ordinary overrides. The DRA/GPU Operator integration is the
+worked example: AICR merges the configured DRA eviction label into
+`kubeletPlugin.nodeSelector` and writes the same label key to GPU Operator's
+`NODE_LABEL_FOR_GPU_POD_EVICTION`. This exception must remain gated on both
+components being enabled, must reject dynamic declarations that could move
+either managed path to install-time configuration, and must be covered across
+every deployer. Ordinary workload placement still uses registry paths and must
+not grow a special flag.
+
 **Deciding where a knob belongs:**
 
 | Situation | Where it goes | Why |

docs/user/api-reference.md

@@ -549,11 +549,12 @@ Generate deployment bundles from a recipe.
 |-----------|------|---------|-------------|
 | `bundlers` | string | (all) | Comma-delimited list of recipe component names to bundle (e.g. `gpu-operator,network-operator`). Whitespace around names is trimmed. Components not listed are skipped as if disabled (their dependency edges are treated as satisfied externally). A name the recipe does not declare, or one that is disabled (by the recipe or a `set` `enabled=false` override), is rejected with HTTP 400. |
 | `set` | string[] | | Value overrides (format: `bundler:path.to.field=value`). Repeat for multiple. The reserved prefix `deployer:` carries Argo CD Application options for `deployer=argocd` and `deployer=argocd-helm` (`namePrefix`, `destinationServer`, `project`, `cascadeDelete`), e.g. `set=deployer:namePrefix=tenant-a-`. Unknown `deployer:` keys — or the prefix with any other deployer — are rejected with HTTP 400. An override whose component is absent from the generated bundle is rejected with HTTP 400 (`INVALID_REQUEST`, "cannot take effect") rather than silently discarded; the scalar `enabled=false` spelling is exempt on a declared component. See [Overrides that cannot take effect are rejected](bundling.md#overrides-that-cannot-take-effect-are-rejected) and the CLI reference's [Argo CD Deployer Options](cli-reference.md#argo-cd-deployer-options) for full semantics. |
-| `dynamic` | string[] | | Declare value paths as install-time parameters (format: `component:path.to.field`). Repeat for multiple. Supported with `deployer=helm`, `deployer=argocd-helm`, `deployer=flux`, and `deployer=helmfile`. A declaration whose component is absent from the generated bundle is rejected with HTTP 400 (`INVALID_REQUEST`, "cannot take effect"); no path is exempt. Certain gate-verified paths on **present** components are also rejected — driver-ownership paths, GPU allocation-policy keys, and, where the corresponding NVSentinel gate applies on the recipe's platform and configuration, the NVSentinel remedy/consumer/runtime-class paths; see [NVSentinel on provider-installed-driver platforms](component-catalog.md#nvsentinel-on-provider-installed-driver-platforms). See [Overrides that cannot take effect are rejected](bundling.md#overrides-that-cannot-take-effect-are-rejected). |
+| `dynamic` | string[] | | Declare value paths as install-time parameters (format: `component:path.to.field`). Repeat for multiple. Supported with `deployer=helm`, `deployer=argocd-helm`, `deployer=flux`, and `deployer=helmfile`. A declaration whose component is absent from the generated bundle is rejected with HTTP 400 (`INVALID_REQUEST`, "cannot take effect"); no path is exempt. Certain gate- or contract-owned paths on **present** components are also rejected — driver-ownership paths, GPU allocation-policy keys, the DRA eviction paths `kubeletPlugin.nodeSelector` and `driver.manager.env` when both contract components are enabled, and, where the corresponding NVSentinel gate applies on the recipe's platform and configuration, the NVSentinel remedy/consumer/runtime-class paths; see [NVSentinel on provider-installed-driver platforms](component-catalog.md#nvsentinel-on-provider-installed-driver-platforms). See [Overrides that cannot take effect are rejected](bundling.md#overrides-that-cannot-take-effect-are-rejected). |
 | `system-node-selector` | string[] | | Node selectors for system components (format: `key=value`). Repeat for multiple. |
 | `system-node-toleration` | string[] | | Tolerations for system components (format: `key=value:effect`). Repeat for multiple. |
 | `accelerated-node-selector` | string[] | | Node selectors for GPU nodes (format: `key=value`). Repeat for multiple. |
 | `accelerated-node-toleration` | string[] | | Tolerations for GPU nodes (format: `key=value:effect`). Repeat for multiple. |
+| `dra-eviction-node-label` | string | `nvidia.com/dra-kubelet-plugin=true` | Single node label coordinating DRA kubelet-plugin eviction with GPU Operator driver upgrades (format: `key=value`). Applied only when both components are enabled. Nodes used for DRA GPU allocation must carry the same label. |
 | `nodes` | int | 0 | Estimated number of GPU nodes (0 = unset). Written to Helm value paths declared in the registry under `nodeScheduling.nodeCountPaths`. |
 | `vendor-charts` | bool | false | Pull upstream Helm chart bytes into the bundle at bundle time so the artifact is fully self-contained and air-gap deployable. Each vendored chart is recorded in `provenance.yaml` with name, version, source URL, and SHA256. Trades the upstream CVE-yank fail-loud signal for offline deployability — see the CLI reference's "Vendoring Charts for Air-Gap" section for the full tradeoff. Requires the `helm` binary on the API server's `$PATH`. **The server-side vendor path is opt-in and off by default** — the operator must set `AICR_ALLOW_VENDOR_CHARTS=true`, otherwise `vendor-charts=true` returns `400 vendor-charts is not enabled on this server`. Even when enabled, repository hosts that resolve to loopback, link-local, private, or cloud-metadata IPs are rejected with `400 INVALID_REQUEST`, and vendored artifacts are capped at 64 MiB. **Private HTTP(S) repository credentials:** the aicrd pre-check sends `HELM_REPOSITORY_USERNAME`/`HELM_REPOSITORY_PASSWORD` (as HTTP Basic auth) ONLY when `AICR_HELM_REPOSITORY_HOST` is set to that repository's exact host, the request scheme is `https`, and the request host matches (case-insensitive). All three conditions must hold — an operator setting only the username/password env vars will get no credentials attached, preventing a caller-supplied `Repository` URL from harvesting the operator's helm credentials. (Note: the upstream `helm pull --repo` subprocess does not itself read these env vars — private HTTP repos require a prior `helm repo add --username --password` in the aicrd image or an SDK-based puller.) OCI credentials flow through the standard docker config (`~/.docker/config.json` or `$DOCKER_CONFIG`), exactly like `helm pull oci://...`. If prerequisites are missing the request fails with a structured error code (`SERVICE_UNAVAILABLE` / HTTP 503 for missing helm). The index pre-check surfaces upstream HTTP status by class: `404` → `NOT_FOUND` / HTTP 404, `401`/`403` → `UNAUTHORIZED` / HTTP 401, `408`/`429` → `SERVICE_UNAVAILABLE` / HTTP 503 (retryable), other `4xx` → `INVALID_REQUEST` / HTTP 400, `5xx` → `SERVICE_UNAVAILABLE` / HTTP 503. |
 | `serial` | bool | false | Sequence components strictly one at a time in deployment order, disabling the parallel rollout of independent components. Affects `deployer=argocd`, `argocd-helm`, `flux`, and `helmfile` (`helm` is already serial): argocd falls back to a linear sync-wave per folder, flux chains each `HelmRelease` `dependsOn` to the previous component, and helmfile chains every release via `needs:` into one linear apply order. An escape hatch for reproducing the pre-parallelism ordering or bisecting a rollout. |
@@ -713,6 +714,12 @@ curl -X POST "http://localhost:8080/v1/bundle?system-node-selector=nodeGroup=sys
   -d @recipe.json \
   -o bundles.zip
 
+# Override the shared DRA eviction label when bundling DRA with GPU Operator
+curl -X POST "http://localhost:8080/v1/bundle?dra-eviction-node-label=example.com%2Fdra-ready%3Denabled" \
+  -H "Content-Type: application/json" \
+  -d @recipe.json \
+  -o bundles.zip
+
 # Generate bundles from a saved (fully-hydrated) recipe
 curl -X POST "http://localhost:8080/v1/bundle" \
   -H "Content-Type: application/json" \

docs/user/bundling.md

@@ -179,6 +179,49 @@ aicr bundle --recipe recipe.yaml \
   --output ./bundles
 ```
 
+## Prepare DRA nodes before applying upgraded bundles
+
+> **Upgrade requirement:** A bundle containing both `gpu-operator` and
+> `nvidia-dra-driver-gpu` now selects DRA kubelet-plugin nodes with
+> `nvidia.com/dra-kubelet-plugin=true` by default. The same applies to the
+> corresponding `-ocp` components. Before applying the first newly generated
+> bundle to an existing deployment, label every GPU node that must run the DRA
+> kubelet plugin. Applying the bundle first can reduce the DaemonSet to zero
+> eligible nodes, interrupting ComputeDomain/IMEX and any whole-GPU resources
+> advertised through DRA.
+
+Label the intended nodes, confirm the selector matches at least one node, and
+then apply the bundle:
+
+```bash
+kubectl label node <node-name> nvidia.com/dra-kubelet-plugin=true
+kubectl get nodes -l nvidia.com/dra-kubelet-plugin=true
+```
+
+If the cluster uses a different convention, generate the bundle with
+`--dra-eviction-node-label key=value` and apply that exact pair to the nodes.
+AICR gives the full pair to the DRA node selector, but GPU Operator's Driver
+Manager receives only the label key because its eviction contract matches and
+temporarily removes the label by key.
+
+After installation and after every GPU driver upgrade, monitor the kubelet
+plugin DaemonSet until all desired pods are ready. This also catches a Driver
+Manager rollout that did not restore the eviction label:
+
+```bash
+kubectl -n nvidia-dra-driver get daemonset \
+  nvidia-dra-driver-gpu-kubelet-plugin
+```
+
+The integration is not rendered when either component is absent. A dynamic
+declaration intersecting `kubeletPlugin.nodeSelector` or `driver.manager.env`
+is rejected because moving either path to install-time configuration would let
+the two halves drift independently. See
+[DRA Driver Upgrade Eviction](cli-reference.md#dra-driver-upgrade-eviction) for
+configuration details and NVIDIA's
+[GPU Operator DRA installation guide](https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/26.3/dra-intro-install.html)
+for the upstream contract.
+
 ## Produce an offline (vendored) bundle
 
 `--vendor-charts` pulls upstream Helm chart bytes into the bundle at bundle

docs/user/cli-config.md

@@ -156,6 +156,7 @@ spec:
         nodeGroup: gpu-worker
       acceleratedNodeTolerations:
         - nvidia.com/gpu=present:NoSchedule
+      draEvictionNodeLabel: nvidia.com/dra-kubelet-plugin=true
       workloadGate: ""
       workloadSelector: {}
       nodes: 2
@@ -265,7 +266,7 @@ Inputs to `aicr bundle`.
 | `deployment.set` / `.dynamic` | []string | Value overrides, `key:path=value` |
 | `deployment.vendorCharts` | bool | Vendor charts into the bundle |
 | `deployment.appName` | string | Argo CD parent `Application` name override (multi-bundle installs sharing a namespace) |
-| `scheduling.*` | object | `systemNodeSelector`/`Tolerations`, `acceleratedNodeSelector`/`Tolerations`, `workloadGate`, `workloadSelector`, `nodes`, `storageClass`, `sharedStorageClass`. Selectors are YAML maps; tolerations use the CLI's `key=value:effect` strings |
+| `scheduling.*` | object | `systemNodeSelector`/`Tolerations`, `acceleratedNodeSelector`/`Tolerations`, `draEvictionNodeLabel`, `workloadGate`, `workloadSelector`, `nodes`, `storageClass`, `sharedStorageClass`. Selectors are YAML maps; tolerations use the CLI's `key=value:effect` strings. `draEvictionNodeLabel` accepts one `key=value` label and defaults to `nvidia.com/dra-kubelet-plugin=true`; AICR applies it only when DRA and GPU Operator are both enabled. |
 | `attestation.enabled` | bool | Enable bundle attestation (signing); keyless OIDC by default, KMS-backed when `signingKey` is set |
 | `attestation.certificateIdentityRegexp` | string | Expected signer identity |
 | `attestation.oidcDeviceFlow` | bool | Device-code flow for headless signing |