Update `k8s-cc-manager` image to support Blackwell GPUs

[msanft]

The default image for the k8s-cc-manager is over 2 years old:

gpu-operator/deployments/gpu-operator/values.yaml

Lines 500 to 511 in 75d1b96

	ccManager:
	enabled: false
	defaultMode: "off"
	repository: nvcr.io/nvidia/cloud-native
	image: k8s-cc-manager
	version: v0.1.1
	imagePullPolicy: IfNotPresent
	imagePullSecrets: []
	env:
	- name: CC_CAPABLE_DEVICE_IDS
	value: "0x2339,0x2331,0x2330,0x2324,0x2322,0x233d"
	resources: {}

It would be great if it could be bumped to one that supports Blackwell GPUs. I tested ghcr.io/nvidia/k8s-cc-manager:d80a3a8a-distroless, which is the latest release. I can confirm this to work with B200s.

Along that, the CC_CAPABLE_DEVICE_IDS environment variable needs to be updated.

[zvonkok]

@msanft We have new images that lift this restriction. We're going to publish a new deployment guide soon-ish. In the meantime, you can update the image to point to nvcr.io/nvidia/cloud-native/k8s-cc-manager:v0.2.0 via ClusterPolicy.

[manuelh-dev]

@msanft - some documentation for this is on the way, see the current draft here: kata-containers/kata-containers#12257 - might be good to give this a try.

[msanft]

Thank you @zvonkok @manuelh-dev!

This worked quite well. The only thing we did is strip the P_GPU_ALIAS environment variable since we need per-GPU annotations for heterogenous clusters (i.e. nvidia.com/GB100_B200 instead of nvidia.com/pgpu). Is there another recommended way of achieving this in your reference deployment guide?

Furthermore, we had to do these two little patches for Blackwell support:

# Support Blackwell GPUs in nvidia-cc-manager
current_ids=$(kubectl get daemonset nvidia-cc-manager -n gpu-operator -o jsonpath='{.spec.template.spec.containers[?(@.name=="nvidia-cc-manager")].env[?(@.name=="CC_CAPABLE_DEVICE_IDS")].value}')
kubectl set env daemonset/nvidia-cc-manager -n gpu-operator -c nvidia-cc-manager CC_CAPABLE_DEVICE_IDS="${current_ids},0x2901"

# Deploy nvidia-cc-manager to nodes with Blackwell GPUs
kubectl patch nodefeaturerule nvidia-nfd-nodefeaturerules --type='json' -p='[
  {
    "op": "add",
    "path": "/spec/rules/10",
    "value": {
      "name": "NVIDIA B200",
      "labels": {
        "nvidia.com/gpu.B200": "true",
        "nvidia.com/gpu.family": "blackwell"
      },
      "matchFeatures": [
        {
          "feature": "pci.device",
          "matchExpressions": {
            "device": {"op": "In", "value": ["2901"]},
            "vendor": {"op": "In", "value": ["10de"]}
          }
        }
      ]
    }
  },
  {
    "op": "add",
    "path": "/spec/rules/11/matchAny/0/matchFeatures/0/matchExpressions/nvidia.com~1gpu.family/value/-",
    "value": "blackwell"
  },
  {
    "op": "add",
    "path": "/spec/rules/11/matchAny/1/matchFeatures/0/matchExpressions/nvidia.com~1gpu.family/value/-",
    "value": "blackwell"
  }
]'

The latter one should be solved by #1973 if my comment there is taken into account.

[manuelh-dev]

Great to hear, @msanft , thanks for exercising this and reporting the findings!

We made this change to the NVIDIA documentation: NVIDIA/cloud-native-docs#334 - does this look good to you?

On the node feature rules: We are working on addressing the current improper node labelling. This may either get solved by the referenced PR or via another way (ack to your comment in the PR). We will keep you in the loop as we make progress.

[msanft]

We made this change to the NVIDIA documentation: NVIDIA/cloud-native-docs#334 - does this look good to you?

Yes, thank you.

On the node feature rules: We are working on addressing the current improper node labelling. This may either get solved by the referenced PR or via another way (ack to your comment in the PR). We will keep you in the loop as we make progress.

Excellent, thanks!