fix(bundler): make the DRA eviction node label opt-in (#2471)

Signed-off-by: Yuan Chen <yuanchen97@gmail.com>

Author: yuanchen8911

.github/workflows/gpu-h100-kind-runtime-test.yaml

@@ -85,16 +85,6 @@ jobs:
           control_plane_resource_patches: 'true'
           control_plane_leader_election_tuning: 'true'
 
-      - name: Label nodes for the DRA kubelet plugin
-        shell: bash
-        env:
-          DRA_EVICTION_NODE_LABEL: nvidia.com/dra-kubelet-plugin=true
-        run: >-
-          kubectl label nodes
-          --selector='!node-role.kubernetes.io/control-plane'
-          "${DRA_EVICTION_NODE_LABEL}"
-          --overwrite
-
       - name: Build AICR CLI
         timeout-minutes: 10
         uses: ./.github/actions/aicr-build

api/aicr/v1/server.baseline.yaml

@@ -1038,7 +1038,8 @@ components:
         components are also rejected (driver-ownership paths, GPU
         allocation-policy keys, the DRA eviction paths
         kubeletPlugin.nodeSelector and driver.manager.env when both contract
-        components are enabled, and — where the corresponding NVSentinel gate
+        components are enabled and dra-eviction-node-label is supplied, and —
+        where the corresponding NVSentinel gate
         applies on the recipe's platform and configuration — the NVSentinel
         remedy/consumer/runtime-class paths).
       schema:
@@ -1096,12 +1097,15 @@ components:
       in: query
       required: false
       description: >
-        Single key=value node label coordinating NVIDIA DRA kubelet-plugin
-        eviction with GPU Operator driver upgrades. Applied only when both
-        components are enabled.
+        Opt in to NVIDIA DRA kubelet-plugin eviction coordination with GPU
+        Operator driver upgrades, as a single key=value node label. Omit the
+        parameter to inject nothing: the kubelet plugin then needs no extra
+        node label. When set, and both components are enabled, the plugin runs
+        only on GPU nodes carrying this label. Unlabeled nodes are silently
+        excluded; if no GPU node carries it the kubelet-plugin DaemonSet
+        reaches DESIRED=0.
       schema:
         type: string
-        default: nvidia.com/dra-kubelet-plugin=true
       example: "example.com/dra-ready=enabled"
     BundleDeployer:
       name: deployer

api/aicr/v1/server.yaml

@@ -1025,7 +1025,8 @@ components:
         components are also rejected (driver-ownership paths, GPU
         allocation-policy keys, the DRA eviction paths
         kubeletPlugin.nodeSelector and driver.manager.env when both contract
-        components are enabled, and — where the corresponding NVSentinel gate
+        components are enabled and dra-eviction-node-label is supplied, and —
+        where the corresponding NVSentinel gate
         applies on the recipe's platform and configuration — the NVSentinel
         remedy/consumer/runtime-class paths).
       schema:
@@ -1083,12 +1084,15 @@ components:
       in: query
       required: false
       description: >
-        Single key=value node label coordinating NVIDIA DRA kubelet-plugin
-        eviction with GPU Operator driver upgrades. Applied only when both
-        components are enabled.
+        Opt in to NVIDIA DRA kubelet-plugin eviction coordination with GPU
+        Operator driver upgrades, as a single key=value node label. Omit the
+        parameter to inject nothing: the kubelet plugin then needs no extra
+        node label. When set, and both components are enabled, the plugin runs
+        only on GPU nodes carrying this label. Unlabeled nodes are silently
+        excluded; if no GPU node carries it the kubelet-plugin DaemonSet
+        reaches DESIRED=0.
       schema:
         type: string
-        default: nvidia.com/dra-kubelet-plugin=true
       example: "example.com/dra-ready=enabled"
     BundleDeployer:
       name: deployer

docs/contributor/component.md

@@ -237,14 +237,33 @@ The narrow exception is a **bundler-owned derived integration value**:
 when AICR renders both sides of a cross-chart contract and allowing either
 side to drift would produce an invalid bundle, the bundler may enforce that
 contract after ordinary overrides. The DRA/GPU Operator integration is the
-worked example: AICR merges the configured DRA eviction label into
+worked example: when opted in, AICR merges the configured DRA eviction label into
 `kubeletPlugin.nodeSelector` and writes the same label key to GPU Operator's
 `NODE_LABEL_FOR_GPU_POD_EVICTION`. This exception must remain gated on both
 components being enabled, must reject dynamic declarations that could move
 either managed path to install-time configuration, and must be covered across
 every deployer. Ordinary workload placement still uses registry paths and must
 not grow a special flag.
 
+**A bundler-owned contract that imposes cluster state must be opt-in.** The DRA
+eviction contract also illustrates the limit of the exception. Its selector is
+an exact-match `nodeSelector`, so rendering it unconditionally made a node label
+a precondition for the kubelet plugin to run at all — and an unlabeled node is
+then silently excluded, with `DESIRED=0` only when no node carries the label,
+and a successful `helm upgrade` either way. AICR does not
+own node labels, cannot verify them at bundle time, and generation is offline,
+so it cannot even warn accurately about what a given cluster carries.
+
+The rule that follows: a derived integration value may enforce consistency
+between two charts AICR renders, but when it also requires state outside those
+charts, the enforcement is opt-in. Absent the opt-in, AICR writes neither side
+and owns neither path — no injection, no override protection, no dynamic-path
+rejection. The single configuration surface that carries the state (here
+`--dra-eviction-node-label` and its config/API/Go equivalents) doubles as the
+opt-in, so there is no second boolean and no ambiguous combination. See
+[DRA Driver Upgrade Eviction](../user/cli-reference.md#dra-driver-upgrade-eviction)
+for the user-facing contract and issue #2469 for the reasoning.
+
 **Deciding where a knob belongs:**
 
 | Situation | Where it goes | Why |

docs/integrator/aks-gpu-setup.md

@@ -122,23 +122,28 @@ spec:
           enabled: false
 ```
 
-### Label GPU nodes for the DRA kubelet plugin
-
-A bundle that enables both `gpu-operator` and `nvidia-dra-driver-gpu` schedules
-the DRA kubelet plugin only on nodes labeled
-`nvidia.com/dra-kubelet-plugin=true` (or the pair given to
-`aicr bundle --dra-eviction-node-label`). Set it **in the node pool
-definition**, with the other required node labels — an ad hoc
-`kubectl label node` does not survive node replacement, recycling,
-autoscaling, or a pool scaled from zero, so later nodes arrive unlabeled.
-
-An unlabeled GPU node fails silently: it runs no kubelet plugin and publishes
-no `ResourceSlices`, and neither Helm nor the bundle's `deploy.sh` reports an
-error. With no labeled GPU node at all the DaemonSet sits at `DESIRED=0`; with
-only some labeled, those nodes work while the rest silently lack DRA.
-This applies to existing clusters too — adding the selector during an
-upgrade removes a plugin that was previously working.
-See [Prepare DRA nodes before applying upgraded bundles](../user/bundling.md#prepare-dra-nodes-before-applying-upgraded-bundles).
+### Label GPU nodes for the DRA kubelet plugin (opt-in only)
+
+DRA eviction coordination is opt-in and **does nothing on the default AKS
+profile**. Under `gpuStack=azure-managed` the node image installs the driver, so
+GPU Operator sets `driver.enabled=false`, deploys no driver pod and runs no
+Driver Manager — there is nothing to coordinate with, and no node label is
+needed. Bundles generated without `--dra-eviction-node-label` add no such
+selector, and the DRA kubelet plugin runs on every accelerated node.
+
+The label matters only if you both run an operator-managed driver (see the
+`gpuStack=operator-managed` procedure below) and generate the bundle with
+`aicr bundle --dra-eviction-node-label key=value`. In that case set it **in the
+node pool definition**, with the other required node labels — an ad hoc
+`kubectl label node` does not survive node replacement, recycling, autoscaling,
+or a pool scaled from zero, so later nodes arrive unlabeled.
+
+When you have opted in, an unlabeled GPU node fails silently: it runs no kubelet
+plugin and publishes no `ResourceSlices`, and neither Helm nor the bundle's
+`deploy.sh` reports an error. With no labeled GPU node at all the DaemonSet sits
+at `DESIRED=0`; with only some labeled, those nodes work while the rest silently
+lack DRA.
+See [Prepare DRA nodes when opting in to eviction coordination](../user/bundling.md#prepare-dra-nodes-when-opting-in-to-eviction-coordination).
 
 ## GPU Driver Setup
 
@@ -258,8 +263,7 @@ az aks nodepool add \
   --resource-group <rg> \
   --name gpupool \
   --node-vm-size Standard_ND96isr_H100_v5 \
-  --node-count 1 \
-  --labels nvidia.com/dra-kubelet-plugin=true
+  --node-count 1
 ```
 
 No changes to AICR recipes are needed — this is the AKS family's `gpuStack`
@@ -591,17 +595,40 @@ az aks nodepool add \
   --name gpupool \
   --node-vm-size Standard_ND96isr_H100_v5 \
   --gpu-driver none \
-  --node-count 1 \
-  --labels nvidia.com/dra-kubelet-plugin=true
+  --node-count 1
 ```
 
+Add the eviction label to this pool only if you also opt in at bundle time. The
+node label and the flag value must be the **same `key=value` pair** — the flag
+selects the convention, and AICR renders exactly what you pass:
+
+`az aks nodepool update --labels` **replaces** the pool's entire user-label map
+rather than merging, so repeat every label the pool already carries or they are
+dropped — including the accelerated-node selector the bundle relies on:
+
+```shell
+az aks nodepool update \
+  --cluster-name <cluster> --resource-group <rg> --name gpupool \
+  --labels nodeGroup=gpu-worker nvidia.com/dra-kubelet-plugin=true
+```
+
+Prefer setting both at pool creation time (`az aks nodepool add --labels ...`)
+so there is no map to preserve.
+
+Then pass the same pair to `aicr bundle` in the generation step below. See
+[Label GPU nodes for the DRA kubelet plugin (opt-in only)](#label-gpu-nodes-for-the-dra-kubelet-plugin-opt-in-only).
+
 Then select the mode at recipe generation time with the `gpuStack`
 configuration profile — one flag flips every ownership path together:
 
 ```shell
 aicr recipe --service aks --accelerator h100 --os ubuntu --intent training \
   --profile gpuStack=operator-managed -o recipe.yaml
-aicr bundle -r recipe.yaml -o ./bundles
+# AKS requires a keyed accelerated-node toleration; add
+# --dra-eviction-node-label only if you opted in and labelled the pool.
+aicr bundle -r recipe.yaml -o ./bundles \
+  --accelerated-node-toleration nvidia.com/gpu:NoSchedule \
+  --dra-eviction-node-label nvidia.com/dra-kubelet-plugin=true
 ```
 
 The `operator-managed` value sets `driver.enabled=true`, `toolkit.enabled=true`,

docs/integrator/gke-gpu-setup.md

@@ -107,7 +107,6 @@ gcloud container node-pools create POOL_NAME \
   --num-nodes=1 \
   --machine-type=a3-highgpu-8g \
   --accelerator type=nvidia-h100-80gb,count=8,gpu-driver-version=default \
-  --node-labels="nvidia.com/dra-kubelet-plugin=true"
 ```
 
 Two flags deserve care:
@@ -231,7 +230,7 @@ gcloud container node-pools create POOL_NAME \
   --num-nodes=1 \
   --machine-type=a3-highgpu-8g \
   --accelerator type=nvidia-h100-80gb,count=8,gpu-driver-version=disabled \
-  --node-labels="gke-no-default-nvidia-gpu-device-plugin=true,nvidia.com/dra-kubelet-plugin=true"
+  --node-labels="gke-no-default-nvidia-gpu-device-plugin=true"
 ```
 
 The `--machine-type` and `--num-nodes` cautions from the default-profile
@@ -294,7 +293,7 @@ gcloud container node-pools describe POOL_NAME \
 gcloud container node-pools update POOL_NAME \
   --cluster CLUSTER_NAME \
   --location=LOCATION \
-  --node-labels="EXISTING_KEY_1=EXISTING_VALUE_1,gke-no-default-nvidia-gpu-device-plugin=true,nvidia.com/dra-kubelet-plugin=true"
+  --node-labels="EXISTING_KEY_1=EXISTING_VALUE_1,gke-no-default-nvidia-gpu-device-plugin=true"
 ```
 
 Replace `EXISTING_KEY_…=EXISTING_VALUE_…` with every label the `describe`
@@ -337,7 +336,7 @@ allocatable `nvidia.com/gpu`):
 
 ```bash
 kubectl get nodes -l cloud.google.com/gke-accelerator \
-  -L gke-no-default-nvidia-gpu-device-plugin,nvidia.com/dra-kubelet-plugin
+  -L gke-no-default-nvidia-gpu-device-plugin
 kubectl get pods -n kube-system -l k8s-app=nvidia-gpu-device-plugin -o wide
 kubectl get pods -n gpu-operator -l app=nvidia-device-plugin-daemonset -o wide
 kubectl get nodes -l cloud.google.com/gke-accelerator \
@@ -379,23 +378,35 @@ regenerate with `--profile gpuStack=bundle-installer`, and deploy the
 bundle. Nodes with a loaded driver are untouched (the installer's fast path
 skips them); the bundle takes over provisioning for new and rebooted nodes.
 
-## Label GPU nodes for the DRA kubelet plugin
+## DRA kubelet plugin node labels are not required on GKE
+
+DRA eviction coordination is opt-in, and by default it does nothing on
+supported GKE recipes. Every GKE overlay is COS, and both the default and
+`gpuStack=bundle-installer` profiles keep GPU Operator's `driver.enabled=false`
+— `bundle-installer` provisions the driver through a separate
+`gcp-driver-installer` DaemonSet, not a GPU Operator driver pod. With no GPU
+Operator driver pod there is no Driver Manager to coordinate with.
+
+So bundles generated for GKE add no eviction node selector, and the DRA kubelet
+plugin runs on every accelerated node with no extra label. Nothing in the node
+pool commands below needs `nvidia.com/dra-kubelet-plugin`.
+
+If you opt in anyway, AICR renders the selector and every GPU node must carry
+the **same `key=value` pair** you passed, set in the node pool definition:
 
-A bundle that enables both `gpu-operator` and `nvidia-dra-driver-gpu` schedules
-the DRA kubelet plugin only on nodes labeled
-`nvidia.com/dra-kubelet-plugin=true` (or the pair given to
-`aicr bundle --dra-eviction-node-label`). Set it **in the node pool
-definition**, with the other required node labels — an ad hoc
-`kubectl label node` does not survive node replacement, recycling,
-autoscaling, or a pool scaled from zero, so later nodes arrive unlabeled.
+```bash
+aicr bundle --recipe recipe.yaml \
+  --dra-eviction-node-label nvidia.com/dra-kubelet-plugin=true \
+  --output bundle
+
+gcloud container node-pools create <pool> --cluster <cluster> \
+  --node-labels="nvidia.com/dra-kubelet-plugin=true"
+```
 
-An unlabeled GPU node fails silently: it runs no kubelet plugin and publishes
-no `ResourceSlices`, and neither Helm nor the bundle's `deploy.sh` reports an
-error. With no labeled GPU node at all the DaemonSet sits at `DESIRED=0`; with
-only some labeled, those nodes work while the rest silently lack DRA.
+See [DRA Driver Upgrade Eviction](../user/cli-reference.md#dra-driver-upgrade-eviction).
 This applies to existing clusters too — adding the selector during an
 upgrade removes a plugin that was previously working.
-See [Prepare DRA nodes before applying upgraded bundles](../user/bundling.md#prepare-dra-nodes-before-applying-upgraded-bundles).
+See [Prepare DRA nodes when opting in to eviction coordination](../user/bundling.md#prepare-dra-nodes-when-opting-in-to-eviction-coordination).
 
 ## Troubleshooting