Skip to content

Commit 8703d8c

Browse files
authored
chore(topograph): Updated the recipes to use topograph v1.0.0 (#2254)
Signed-off-by: Ravi Shankar <ravish@nvidia.com>
1 parent af9099f commit 8703d8c

10 files changed

Lines changed: 92 additions & 137 deletions

File tree

docs/integrator/recipe-development.md

Lines changed: 6 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -171,11 +171,10 @@ Leaves that need topology-aware scheduling can optionally add `slinky-topograph`
171171
- slinky-slurm-operator-crds
172172
- slinky-slurm # deploy after Slurm so Helm already owns config-extra
173173
overrides:
174-
global:
175-
provider:
176-
name: gcp # cloud provider: gcp | aws | oci | nebius | …
177-
engine:
178-
name: slinky # scheduler consumer: slinky | slurm | k8s | graph
174+
provider:
175+
name: gcp # cloud provider: gcp | aws | oci | nebius | …
176+
engine:
177+
name: slinky # scheduler consumer: slinky | slurm | k8s | graph
179178
180179
- name: slinky-slurm
181180
type: Helm
@@ -205,9 +204,8 @@ For cloud providers (gcp, aws, oci, nebius, …), `slinky-topograph` requires IA
205204

206205
```yaml
207206
overrides:
208-
global:
209-
provider:
210-
name: gcp
207+
provider:
208+
name: gcp
211209
serviceAccount:
212210
annotations:
213211
iam.gke.io/gcp-service-account: <sa-name>@<project-id>.iam.gserviceaccount.com

docs/user/component-catalog.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -43,7 +43,7 @@ The source of truth is [`recipes/registry.yaml`](https://github.com/NVIDIA/aicr/
4343
| **slinky-slurm-operator-crds** | Custom Resource Definitions for the SchedMD Slinky Slurm operator. Installs the `slinky.slurm.net` CRDs (Controller, NodeSet, LoginSet, Accounting, RestApi, Token). Installed separately to support CRD lifecycle management. | [Slinky Slurm Operator](https://github.com/SlinkyProject/slurm-operator) |
4444
| **slinky-slurm-operator** | SchedMD Slinky Slurm operator and admission webhook. Manages the lifecycle of Slurm clusters declared via Slinky CRs (Controller, NodeSet, LoginSet, Accounting, RestApi, Token). AICR's system node-selector and toleration bundle flags apply to both deployments; affinity remains available through component values or typed overrides. | [Slinky Slurm Operator](https://github.com/SlinkyProject/slurm-operator) |
4545
| **slinky-slurm** | Slinky-managed Slurm cluster instance: Controller (slurmctld) + LoginSet (sackd/sshd) + NodeSet (slurmd) + RestApi (slurmrestd), with SlurmDBD derived from the recipe's typed accounting mode. Reconciled by `slinky-slurm-operator`. See [Slurm Accounting](slinky-slurm-accounting.md), [Slurm Enroot Configuration](slinky-slurm-enroot.md), and [Slurm Shared Storage](slinky-slurm-storage.md). | [Slinky Slurm Cluster Chart](https://github.com/SlinkyProject/slurm-operator/tree/main/helm/slurm) |
46-
| **slinky-topograph** | Slinky/Slurm-scoped instance of Topograph — queries cloud provider topology APIs (GCP, AWS, OCI …) to generate Slurm `topology.conf`, enabling topology-aware placement decisions in the Slinky-managed scheduler. **Not installed by default**; leaf overlays opt in by adding an explicit `componentRef` entry for `slinky-topograph` — the `componentRef` is what schedules the release; `dependencyRefs` alone does not install anything. That `componentRef` declares `slinky-slurm` as a `dependencyRef` to deploy **after** it: `slinky-slurm` renders and owns the `slinky-slurm-config-extra` ConfigMap (from its `configFiles`, mounted into slurmctld via the Controller CR's `configFileRefs`), and Topograph patches only that ConfigMap's `topology.conf` key on each sync, preserving the chart-owned `cgroup.conf`/`gres.conf` keys — Helm has to own the ConfigMap first. `TopologyPlugin: topology/tree` is set per-leaf via `slinky-slurm`'s `controller.extraConfMap`. Includes the `node-observer` sub-chart, which watches the topograph API pod and regenerates topology on restarts or selected node/pod changes. Requires cloud provider IAM access (e.g. GCP `roles/compute.viewer` for Workload Identity). | [Topograph](https://github.com/NVIDIA/topograph) |
46+
| **slinky-topograph** | Slinky/Slurm-scoped instance of Topograph — queries cloud provider topology APIs (GCP, AWS, OCI …) to generate Slurm `topology.conf`, enabling topology-aware placement decisions in the Slinky-managed scheduler. **Not installed by default**; leaf overlays opt in by adding an explicit `componentRef` entry for `slinky-topograph` — the `componentRef` is what schedules the release; `dependencyRefs` alone does not install anything. That `componentRef` declares `slinky-slurm` as a `dependencyRef` to deploy **after** it: `slinky-slurm` renders and owns the `slinky-slurm-config-extra` ConfigMap (from its `configFiles`, mounted into slurmctld via the Controller CR's `configFileRefs`), and Topograph patches only that ConfigMap's `topology.conf` key on each sync, preserving the chart-owned `cgroup.conf`/`gres.conf` keys — Helm has to own the ConfigMap first. `TopologyPlugin: topology/tree` is set per-leaf via `slinky-slurm`'s `controller.extraConfMap`. Includes the `node-observer` component, which watches the topograph API pod and regenerates topology on restarts or selected node/pod changes. Requires cloud provider IAM access (e.g. GCP `roles/compute.viewer` for Workload Identity). | [Topograph](https://github.com/NVIDIA/topograph) |
4747
| **nfd-ocp-olm** | OLM installer for Node Feature Discovery on OpenShift. Creates the OperatorGroup and Subscription resources that install NFD via the Operator Lifecycle Manager. Paired with `nfd-ocp`. OCP-specific. | [Node Feature Discovery (Certified)](https://catalog.redhat.com/software/container-stacks/detail/5ec53e8c110f56bd24f5f8db) |
4848
| **nfd-ocp** | Node Feature Discovery CR for OpenShift. Configures NFD's operand (worker, topology updater) via a NodeFeatureDiscovery custom resource. Deployed after `nfd-ocp-olm`. OCP-specific. | [Node Feature Discovery](https://github.com/kubernetes-sigs/node-feature-discovery) |
4949
| **gpu-operator-ocp-olm** | OLM installer for the GPU Operator on OpenShift. Creates the OperatorGroup and Subscription resources that install the certified GPU Operator via the Operator Lifecycle Manager. Paired with `gpu-operator-ocp`. OCP-specific. | [NVIDIA GPU Operator (Certified)](https://catalog.redhat.com/software/container-stacks/detail/5e7b210b8a3c1e00013d636d) |

docs/user/container-images.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -72,7 +72,7 @@ _Rendering fidelity:_ `catalog-parity: charts are rendered with the shared recip
7272
| slinky-slurm | helm | slurm | 1.2.0 | 5 |
7373
| slinky-slurm-operator | helm | slurm-operator | 1.2.0 | 2 |
7474
| slinky-slurm-operator-crds | helm | slurm-operator-crds | 1.2.0 | 0 |
75-
| slinky-topograph | helm | topograph/topograph | 0.5.0 | 1 |
75+
| slinky-topograph | helm | topograph/topograph | 1.0.0 | 1 |
7676
| slurm-accounting-mariadb | helm | mariadb-cluster | 26.6.0 | 0 |
7777

7878
## Version variants
@@ -320,7 +320,7 @@ _No images extracted._
320320

321321
### slinky-topograph
322322

323-
- `ghcr.io/nvidia/topograph:v0.5.0`
323+
- `ghcr.io/nvidia/topograph:v1.0.0`
324324

325325
### slurm-accounting-mariadb
326326

pkg/bundler/testdata/stock_render_golden.yaml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -29,10 +29,10 @@ h100-eks-ubuntu-training-kubeflow: 8c1fe9721baba50d2d383e30d883c92f8c2cb99a881ed
2929
h100-eks-ubuntu-training-slurm: 082a43adc19c769b31313e09c685c18b779321b2dbcbc67a8f5699dd5ae61826
3030
h100-gke-cos-inference-dynamo: a06816f00f95b6d8dd134b38ae0d2d8c5c36ab6d643181c4c9651ac409e7b5a5
3131
h100-gke-cos-training-kubeflow: ac2c6cf43293c88d04528382cb5ba189ff906489b2c95ce636a6d0f29be5f43f
32-
h100-gke-cos-training-slurm: 9029621222d4c5f5cd49bc380e0aaa9ea6dadf7debbac45fd301ece293c29bd8
32+
h100-gke-cos-training-slurm: 8da5bafdf36f1c9118f75f32b1e0fad1d1b101daadb89036c4564c6b91dc4b00
3333
h100-kind-inference-dynamo: 32d919b1af2f780a989dd044203cb7220593b081c822227b91810249f4cad821
3434
h100-kind-training-kubeflow: dd2ded798dbc2463fafd3da93126dc431f2e42410e060e7cc4e0be4cd4d39fa9
35-
h100-kind-training-slurm: 96d9710ffbf51c1cd62d2434c816601aff8969f44dc02fba3f645fbc78b0dbd9
35+
h100-kind-training-slurm: 1b7147dd701c02e3db15032fbe7f34de87839bea71f0cfbaa5f5463e599e5b3f
3636
h200-any: b9e3f4593b7cced8ab233b44d52488a6fac5eeac3f9ca0a7cb1ee214672f72a1
3737
h200-eks-inference: b6454b224e2287eec1f8608bf539679cf8a7fbc7016bebdc139afc944315a414
3838
h200-eks-training: 9c0c79822ed70e2010158af6eba120ee0d8f942f1fafd9d981f342394cae0929

pkg/recipe/testdata/catalog_parity_golden.yaml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -29,10 +29,10 @@ h100-eks-ubuntu-training-kubeflow: d0dc8163f513470f8cdcb7906d2ff6eef630185c3469a
2929
h100-eks-ubuntu-training-slurm: 04fb61932ca46d4b0ae9f232d0c40dbc9b02872430091849c44dcf43356f366f
3030
h100-gke-cos-inference-dynamo: b5eb7a000dd238471584b7daf062f1867da4cd3c2468343a1f79582f6ec1f38a
3131
h100-gke-cos-training-kubeflow: ed11489b48c4a036b4e3d336ec41b1776e286f15a8e0caa75315e623074e4385
32-
h100-gke-cos-training-slurm: 32bb46642f1f234796d4e4ed0791e04f7d8c2a3ddb95312f8051a34a052b1988
32+
h100-gke-cos-training-slurm: 43cc50555b76ddadf175a45df49fe4f50713c1df2a40afd74a9a421eca8ba8bd
3333
h100-kind-inference-dynamo: f665ecad511b4492eb79f6de19d4ccb9e2721fcdf34ead85711b00b52678e1f0
3434
h100-kind-training-kubeflow: da0ba5fd7872854337cfca957880cbe124317c154b61608a8b21c55d2172d6d8
35-
h100-kind-training-slurm: 678a468fd348bbd2922cf5a282abef060b285410699bbca16cfbd5a7a890dc8b
35+
h100-kind-training-slurm: 184c5240e6956fe0f2e3310922044ecdfbb8cb4f0df815d696b561cb49e60bf1
3636
h200-any: b6bdb82e355993c99840147c3846656aff004a1fb4253fe2d48b85bfd860ce4d
3737
h200-eks-inference: c3c801df830173e967a366f31462ace88079656a0a660deafa09d7d45ee91583
3838
h200-eks-training: 3cb65fd9e45d7f6fbf440469c7a27ad975b634850f6ebd2566831c54dfa7a2ba

recipes/components/slinky-topograph/manifests/kind-test-model.yaml

Lines changed: 0 additions & 43 deletions
This file was deleted.

recipes/components/slinky-topograph/values.yaml

Lines changed: 39 additions & 40 deletions
Original file line numberDiff line numberDiff line change
@@ -22,44 +22,45 @@
2222
# it deploys after slinky-slurm so that chart owns the config-extra
2323
# ConfigMap before Topograph patches it.
2424
#
25-
# global.provider.name is the REQUIRED per-overlay override — set it in the
26-
# consuming overlay's overrides: block. global.engine.name is pinned to
27-
# slinky below: the chart schema requires global.engine, but this shared
28-
# values file already satisfies it, so overlays do not need to repeat it
29-
# (the slurm leaves still set it explicitly, which is harmless).
30-
# Valid providers (chart values.schema.json @ v0.5.0): aws, aws-sim, cw,
31-
# dra, dsx-sim, gcp, gcp-sim, infiniband-bm, infiniband-k8s, lambdai,
25+
# provider.name is the REQUIRED per-overlay override — set it in the
26+
# consuming overlay's overrides: block. engine.name already defaults to
27+
# slinky below: the chart schema requires engine, but this shared values
28+
# file already satisfies it, so overlays do not need to repeat it (the
29+
# slurm leaves still set it explicitly, which is harmless).
30+
# Valid providers (chart values.schema.json @ v1.0.0): aws, aws-sim, dra,
31+
# dsx-sim, gcp, gcp-sim, infiniband-bm, infiniband-k8s, lambdai,
3232
# lambdai-sim, nebius, nebius-sim, netq, nscale, nscale-sim, oci,
3333
# oci-imds, oci-sim, test
34-
# Valid engines: graph, k8s, slinky, slurm
34+
# Valid engines: graph, k8s, nfd, slinky, slurm
3535
# Example (GKE + Slinky):
3636
# overrides:
37-
# global:
38-
# provider:
39-
# name: gcp
37+
# provider:
38+
# name: gcp
4039

4140
# Stabilize the Deployment name to "slinky-topograph"; health-check assertions
42-
# and dependencyRef consumers rely on it.
41+
# and dependencyRef consumers rely on it. nodeObserver and nodeDataBroker
42+
# Deployment/DaemonSet names derive from this fullname (no separate
43+
# fullnameOverride for those components — they are rendered from the same
44+
# chart, not standalone sub-charts).
4345
fullnameOverride: slinky-topograph
4446

45-
global:
46-
service:
47-
type: ClusterIP
48-
port: 49021
49-
engine:
50-
name: slinky
51-
params:
52-
plugin: topology/tree
53-
podSelector:
54-
matchLabels:
55-
app.kubernetes.io/name: slurmd
56-
# Chart-owned ConfigMap rendered by the slurm chart from .Values.configFiles
57-
# (<fullname>-config-extra) and mounted into slurmctld via the Controller
58-
# CR's configFileRefs. Topograph patches ONLY the topology.conf key
59-
# (create-or-update with key-level merge), preserving cgroup.conf/gres.conf.
60-
topologyConfigmapName: slinky-slurm-config-extra
61-
topologyConfigPath: topology.conf
62-
namespace: slurm
47+
service:
48+
type: ClusterIP
49+
port: 49021
50+
engine:
51+
name: slinky
52+
params:
53+
plugin: topology/tree
54+
podSelector:
55+
matchLabels:
56+
app.kubernetes.io/name: slurmd
57+
# Chart-owned ConfigMap rendered by the slurm chart from .Values.configFiles
58+
# (<fullname>-config-extra) and mounted into slurmctld via the Controller
59+
# CR's configFileRefs. Topograph patches ONLY the topology.conf key
60+
# (create-or-update with key-level merge), preserving cgroup.conf/gres.conf.
61+
topologyConfigmapName: slinky-slurm-config-extra
62+
topologyConfigPath: topology.conf
63+
namespace: slurm
6364

6465
# Topology request batching delay. Topograph waits this long after the first
6566
# request before querying the provider API, collecting parallel requests into
@@ -72,19 +73,17 @@ serviceAccount:
7273
create: true
7374
name: slinky-topograph
7475

75-
# node-observer sub-chart — watches the Topograph API pod and triggers
76-
# topology regeneration on API restarts or selected node/pod changes.
77-
# Bundled as a local chart dependency; always deployed alongside topograph.
78-
node-observer:
79-
# Stabilize the Deployment name for health-check assertions.
80-
fullnameOverride: slinky-topograph-node-observer
76+
# Node Observer — watches the Topograph API pod and triggers topology
77+
# regeneration on API restarts or selected node/pod changes. Rendered from
78+
# the same chart as topograph (not a sub-chart); always deployed alongside it.
79+
nodeObserver:
8180
topograph:
8281
trigger:
8382
podSelector:
8483
matchLabels:
8584
app.kubernetes.io/name: slurmd
8685

87-
# node-data-broker sub-chart
88-
# Bundled as a local chart dependency.
89-
node-data-broker:
90-
fullnameOverride: slinky-topograph-node-data-broker
86+
# Node Data Broker — rendered from the same chart as topograph (not a
87+
# sub-chart). Enabled by default; leaves that don't need it set
88+
# nodeDataBroker.enabled: false in their overlay overrides.
89+
nodeDataBroker: {}

recipes/overlays/h100-gke-cos-training-slurm.yaml

Lines changed: 4 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -102,11 +102,10 @@ spec:
102102
- slinky-slurm-operator-crds
103103
- slinky-slurm
104104
overrides:
105-
global:
106-
provider:
107-
name: gcp
108-
engine:
109-
name: slinky
105+
provider:
106+
name: gcp
107+
engine:
108+
name: slinky
110109

111110
- name: slinky-slurm
112111
type: Helm

recipes/overlays/h100-kind-training-slurm.yaml

Lines changed: 29 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -82,36 +82,36 @@ spec:
8282
- slinky-slurm-operator
8383
- slinky-slurm-operator-crds
8484
- slinky-slurm
85-
preManifestFiles:
86-
- components/slinky-topograph/manifests/kind-test-model.yaml
8785
overrides:
88-
global:
89-
provider:
90-
name: test
91-
params:
92-
modelFileName: /etc/topograph/models/kind-slinky.yaml
93-
engine:
94-
name: slinky
95-
# Disable the node-data-broker in the kind environment.
96-
node-data-broker:
86+
provider:
87+
name: test
88+
params:
89+
# Bare filename only: the v1.0.0 test provider's model loader
90+
# resolves this against topograph's embedded tests/models/*.yaml
91+
# (compiled into the binary) and rejects any path component —
92+
# a mounted ConfigMap file cannot be referenced this way. Use one
93+
# of topograph's built-in fixtures (tests/models/small-tree.yaml
94+
# upstream) instead of a custom mounted model.
95+
modelFileName: small-tree.yaml
96+
engine:
97+
name: slinky
98+
# Disable the node-data-broker in the kind environment. nodeObserver
99+
# automatically skips waiting on it when nodeDataBroker.enabled is false.
100+
nodeDataBroker:
97101
enabled: false
98-
node-observer:
99-
topograph:
100-
nodeDataBroker:
101-
enabled: false
102-
volumes:
103-
- name: test-model
104-
configMap:
105-
name: slinky-topograph-test-model
106-
volumeMounts:
107-
- name: test-model
108-
mountPath: /etc/topograph/models
109-
readOnly: true
110102
# Inline override of the registry assertFile (checks/slinky-topograph/
111103
# health-check.yaml): this leaf disables node-data-broker, so the
112-
# DaemonSet step must not run, and the test provider's mounted
113-
# kind-slinky.yaml model (references the real slinky-0 node) makes
114-
# the exact topology.conf content assertable.
104+
# DaemonSet step must not run, and the exact topology.conf content
105+
# below pins the small-tree.yaml fixture's own shape — it does NOT
106+
# validate resolution of the leaf's real one-replica worker
107+
# (slinky-0). Topograph's test provider always returns its fixture
108+
# data as a ready-made instance map, and when the engine already has
109+
# a non-empty instance map it uses that as-is instead of resolving
110+
# real pod/node names — so the real slinky-0 pod is never looked up
111+
# here. Node names (I21, I22, I25, I34-I36) are the fixture's
112+
# instance IDs; cluset compacts them to I[21-22,25] and I[34-36].
113+
# Switch order (S2 before S3) is deterministic (sorted breadth-first
114+
# traversal of the fixture tree).
115115
healthCheckAsserts: |
116116
apiVersion: chainsaw.kyverno.io/v1alpha1
117117
kind: Test
@@ -211,7 +211,9 @@ spec:
211211
namespace: slurm
212212
data:
213213
topology.conf: |
214-
SwitchName=SW1 Nodes=slinky-0
214+
SwitchName=S1 Switches=S[2-3]
215+
SwitchName=S2 Nodes=I[21-22,25]
216+
SwitchName=S3 Nodes=I[34-36]
215217
216218
- name: slinky-slurm
217219
type: Helm

recipes/registry.yaml

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -888,24 +888,24 @@ components:
888888
helm:
889889
defaultRepository: https://nvidia.github.io/topograph
890890
defaultChart: topograph/topograph
891-
defaultVersion: "0.5.0"
891+
defaultVersion: "1.0.0"
892892
defaultNamespace: topograph
893893
nodeScheduling:
894894
system:
895895
nodeSelectorPaths:
896896
- nodeSelector
897-
- node-observer.nodeSelector
897+
- nodeObserver.nodeSelector
898898
tolerationPaths:
899899
- tolerations
900-
- node-observer.tolerations
900+
- nodeObserver.tolerations
901901
accelerated:
902902
nodeSelectorPaths:
903-
# node-observer.topograph.trigger.nodeSelector is a watch filter —
903+
# nodeObserver.topograph.trigger.nodeSelector is a watch filter —
904904
# it narrows which node-change events trigger topology regeneration.
905-
- node-observer.topograph.trigger.nodeSelector
906-
- node-data-broker.nodeSelector
905+
- nodeObserver.topograph.trigger.nodeSelector
906+
- nodeDataBroker.nodeSelector
907907
tolerationPaths:
908-
- node-data-broker.tolerations
908+
- nodeDataBroker.tolerations
909909

910910
- name: gatekeeper
911911
displayName: Gatekeeper

0 commit comments

Comments
 (0)