Skip to content

Commit 2e1e29e

Browse files
committed
fix(cli): list self-hosted cluster registrations
Add an explicit ICMS-backed registration list command that uses the admin JWT, and update the self-hosted documentation and embedded agent skill to use it. Closes #1197 Signed-off-by: Mike Camp <mcamp@nvidia.com>
1 parent e12c45e commit 2e1e29e

12 files changed

Lines changed: 3641 additions & 3454 deletions

File tree

ai-tooling/user/skills/nvcf-self-managed-cli/SKILL.md

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -91,7 +91,7 @@ nvcf-cli self-hosted uninstall --no-apply --compute-plane --cluster-name=ncp-loc
9191

9292
| Env var | Token type | Used by |
9393
|---|---|---|
94-
| `NVCF_TOKEN` | Admin JWT | `function create` / `deploy` / `update` / `delete`, cluster management (`cluster register`/`rotate`/`delete`, `self-hosted` ops). Required for admin commands; preferred for the rest. |
94+
| `NVCF_TOKEN` | Admin JWT | `function create` / `deploy` / `update` / `delete`, cluster management (`cluster register`/`registration list`/`rotate`/`delete`, `self-hosted` ops). Required for admin commands; preferred for the rest. |
9595
| `NVCF_API_KEY` | `nvapi-...` API key | `function invoke` / `list` / `get`, queue details. Falls back to `NVCF_TOKEN` when unset. |
9696

9797
Token generation flow:
@@ -148,6 +148,7 @@ After `init`, the credentials live in `~/.nvcf-cli.state`, so later commands wor
148148
| `nvcf-cli self-hosted status [--cluster-name=X] [--watch] [--json]` | Snapshot dashboard of cluster identity + component health + recent events | Routine health checks; `--watch` for live |
149149
| `nvcf-cli init` | Mint admin token from API Keys service via the public api gateway | Before any cluster-management operation; idempotent |
150150
| `nvcf-cli cluster register --name=X --nca-id=Y --region=Z [--ignore-existing]` | Register a cluster JWKS+OIDC issuer with ICMS | Standalone register (without compute-plane install) |
151+
| `nvcf-cli cluster registration list --nca-id=Y [--icms-url=URL]` | List self-hosted cluster registrations from ICMS | Check registered compute-plane names and IDs with the admin token |
151152
| `nvcf-cli cluster rotate --cluster-id=ID` | Rotate cluster JWKS in ICMS | When NVCA's K8s signing key changed and PSAT verification started 401-ing |
152153
| `nvcf-cli cluster delete --cluster-id=ID` | Remove cluster registration from ICMS | **Confirm with user.** Destroys ICMS state for the cluster. |
153154
| `nvcf-cli api-key generate --description="…" --expires-in=1h` | Mint both a function API key and a task API key (default) | Before invoking functions or creating tasks; run after every `init` |
@@ -212,7 +213,7 @@ For step-by-step playbooks, load the prompt that matches the user's intent:
212213
- `nvcf-cli self-hosted down` or `uninstall` in any form — destructive. **ALWAYS run with `--plan-only` (`down`) or `--no-apply` (`uninstall`) first** and show the user what would happen. State which compute plane(s) and whether persistent state would be wiped.
213214
- `nvcf-cli self-hosted down --remove-persistent` (or `uninstall --remove-persistent`) — deletes Cassandra rows, OpenBao seal keys, sr-default user data. **Loss is unrecoverable.** Confirm explicitly that this is what the user wants.
214215
- `nvcf-cli self-hosted uninstall --control-plane --force-with-registered-clusters` — orphans every registered compute plane (PSAT auth breaks immediately). State the consequence before passing this flag.
215-
- `nvcf-cli self-hosted down --all` nukes everything. Always show the cluster list (`nvcf-cli cluster list`) and get confirmation.
216+
- `nvcf-cli self-hosted down --all` nukes everything. Always show the registered clusters (`nvcf-cli cluster registration list --nca-id=<nca-id>`) and get confirmation.
216217
- `nvcf-cli cluster delete` — removes the cluster's ICMS registration; the compute plane immediately stops being able to authenticate.
217218
- `nvcf-cli function delete` — removes a function and any active deployment.
218219
- `nvcf-cli task delete` — permanently removes the task record. Stop, state the task ID and current status, then wait for a subsequent user reply that explicitly confirms deletion of that specific task before running this command. Do not treat the user's original delete request as confirmation.
@@ -223,7 +224,7 @@ For step-by-step playbooks, load the prompt that matches the user's intent:
223224
- Run `nvcf-cli self-hosted status` before assuming a cluster exists / is healthy.
224225
- Show the planned action (cluster name, function name, GPU type, cost if known) before creating.
225226
- Before creating or deploying a container or LLM function, confirm the exact function name and container image with the user. For LLM functions, also confirm the exact model name used in `models[].name` and OpenAI `model: "<function-id>/<model-name>"`. If any value is missing, ask the user instead of guessing or submitting example placeholders.
226-
- Confirm exact resource names before deletion — match against `cluster list` / `function list` output.
227+
- Confirm exact resource names before deletion. Match against `cluster registration list` / `function list` output.
227228
- In CI / non-interactive contexts, use `--non-interactive --token=$JWT`. Never propose interactive `nvcf-cli init` when `$CI` is set.
228229

229230
**NEVER paste these into chat / logs / feedback:**
@@ -285,6 +286,7 @@ nvcf-cli self-hosted status # snapshot
285286
nvcf-cli self-hosted status --watch # live
286287
nvcf-cli init # mint admin token (clears all saved API keys)
287288
nvcf-cli cluster register … # register cluster
289+
nvcf-cli cluster registration list --nca-id=<nca-id> # list self-hosted registrations
288290
nvcf-cli api-key generate --description=… # mint both function and task API keys (run after every init)
289291
nvcf-cli api-key generate --for function … # function key only
290292
nvcf-cli api-key generate --for task … # task key only

ai-tooling/user/skills/nvcf-self-managed-cli/examples/multi-cluster.md

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,8 @@ for CTX in admin@gpu-east-1 admin@gpu-west-1 admin@gpu-eu-1; do
2424
done
2525

2626
# 3. Verify all compute planes registered:
27-
nvcf-cli cluster list --json | jq '.clusters[] | {name, region, lastHeartbeat}'
27+
nvcf-cli cluster registration list --nca-id=$NCA_ID --icms-url=$ICMS --json \
28+
| jq '.clusters[] | {name: .clusterName, status, nvcaVersion}'
2829
```
2930

3031
## Compute-plane-only operator scenario
@@ -80,7 +81,8 @@ The `clusters` array (using cluster names, not IDs) limits scheduling to those c
8081

8182
```sh
8283
# List all registered compute planes:
83-
nvcf-cli cluster list --json | jq -r '.clusters[].name' > clusters.txt
84+
nvcf-cli cluster registration list --nca-id=$NCA_ID --icms-url=$ICMS --json \
85+
| jq -r '.clusters[].clusterName' > clusters.txt
8486

8587
# Status snapshot per compute plane (assuming each context name matches):
8688
while read NAME; do

ai-tooling/user/skills/nvcf-self-managed-cli/prompts/add-compute-plane.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ User has a working NVCF control plane (running somewhere) and wants to register
77
- **kubectl context for the new compute plane** in their `KUBECONFIG`.
88
- **Public ICMS URL** of the existing control plane (e.g. `https://icms.nvcf.example.com`).
99
- **Admin JWT** for the control plane's account, OR ability to mint one via `nvcf-cli init` against the control plane's public api endpoint. (Admin tokens come from the API Keys service via the public api gateway — kubectl access to the control plane is NOT required to obtain one.)
10-
- **A unique `--cluster-name`** that doesn't collide with already-registered clusters. Use `nvcf-cli cluster list` from a control-plane context to check.
10+
- A unique `--cluster-name` that doesn't collide with already-registered clusters. Use `nvcf-cli cluster registration list --nca-id=$NCA_ID --icms-url=$ICMS` to check.
1111

1212
## Steps
1313

@@ -53,4 +53,4 @@ User has a working NVCF control plane (running somewhere) and wants to register
5353
- **Re-registering the same cluster** (re-running on the same compute plane that was previously registered): expected, no-op semantics.
5454
- **Different physical cluster but same name**: the second attempt will reuse the ICMS row, but the new compute plane's JWKS will be silently *replaced* — the old compute plane's NVCA agent will start failing PSAT auth. **Confirm with the user** that they meant to overwrite.
5555

56-
When in doubt, run `nvcf-cli cluster list` first and ask the user before proceeding with a name that already exists.
56+
When in doubt, run `nvcf-cli cluster registration list --nca-id=$NCA_ID --icms-url=$ICMS` first and ask the user before proceeding with a name that already exists.

ai-tooling/user/skills/nvcf-self-managed-cli/prompts/rotate-cluster-jwks.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ User says NVCA stopped authenticating, or PSAT auth is 401-ing against ICMS. The
1010

1111
## Steps
1212

13-
1. **Confirm the cluster ID.** `nvcf-cli cluster list` (against the control plane). Identify the row matching the user's compute plane.
13+
1. Confirm the cluster ID. Run `nvcf-cli cluster registration list --nca-id=$NCA_ID --icms-url=$ICMS`. Identify the row matching the user's compute plane.
1414

1515
2. **Rotate.** From a context that can reach the compute plane's K8s API (because rotation re-fetches the K8s API's `/openid/v1/jwks`):
1616

ai-tooling/user/skills/nvcf-self-managed-cli/reference/commands.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,7 @@ Full subcommand list. Always pair with [flags.md](flags.md) for global flags and
2828
| Command | Purpose | Notes |
2929
|---|---|---|
3030
| `cluster register --name=X --nca-id=Y [--region=Z] [--ignore-existing]` | Register a JWKS+OIDC with ICMS | Used by `up` Phase 5 internally; standalone for manual registration |
31-
| `cluster list` | List registered clusters | Output is YAML by default; `--json` for machine |
31+
| `cluster registration list --nca-id=Y [--icms-url=URL]` | List self-hosted cluster registrations from ICMS | Uses the admin token; `--json` for machine-readable output |
3232
| `cluster rotate --cluster-id=ID` | Re-fetch JWKS from K8s and PUT to ICMS | After K8s API server signing key rotation |
3333
| `cluster delete --cluster-id=ID` | Remove ICMS row | **DESTRUCTIVE: confirm with user** |
3434

docs/user/cli.md

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -547,6 +547,18 @@ overrides (`selfManaged.icmsServiceHostHeaderOverride`,
547547
values. See [self-managed-clusters](./cluster-management/self-managed.md) for how the
548548
register values feed the operator install and when host-header overrides are required.
549549

550+
List the self-hosted cluster registrations stored in ICMS with the admin token:
551+
552+
```bash
553+
./nvcf-cli cluster registration list \
554+
--nca-id <nca-id> \
555+
--icms-url "http://<GATEWAY_ADDR>"
556+
```
557+
558+
Use `--json` for machine-readable output. `cluster registration list` is distinct
559+
from `cluster list`, which lists NVCF cluster groups and requires API credentials
560+
with the corresponding NVCF API scope.
561+
550562
### General Commands
551563

552564
| Command | Description |
@@ -1072,7 +1084,7 @@ Registry credential changes take up to about 5 minutes to take effect for task c
10721084
| `function list`, `function list-ids`, `function list-versions`, `function get` | `NVCF_TOKEN` or `NVCF_API_KEY` | `list_functions` or `list_functions_details` | `NVCF_API_KEY` |
10731085
| `function queue status`, `function queue position`, `function queue details` | `NVCF_TOKEN` or `NVCF_API_KEY` | `queue_details` | `NVCF_API_KEY` |
10741086
| `registry-credential` commands | `NVCF_TOKEN` or `NVCF_API_KEY` | `manage_registry_credentials` | `NVCF_TOKEN` |
1075-
| Self-hosted cluster register, list, rotate, delete | `NVCF_TOKEN` | `cluster-management` | `NVCF_TOKEN` |
1087+
| Self-hosted cluster register, registration list, rotate, delete | `NVCF_TOKEN` | `cluster-management` | `NVCF_TOKEN` |
10761088
| `task create` | `NVCF_NVCT_API_KEY` | `launch_task` | `NVCF_NVCT_API_KEY` |
10771089
| `task list` | `NVCF_NVCT_API_KEY` | `list_tasks` | `NVCF_NVCT_API_KEY` |
10781090
| `task get` | `NVCF_NVCT_API_KEY` | `task_details` | `NVCF_NVCT_API_KEY` |

src/clis/nvcf-cli/cmd/BUILD.bazel

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@ go_library(
2727
"cluster_agent_validate.go",
2828
"cluster_dump.go",
2929
"cluster_registration.go",
30+
"cluster_registration_list.go",
3031
"cluster_utils.go",
3132
"deploy.go",
3233
"exit_code_error.go",
@@ -108,6 +109,7 @@ go_test(
108109
"cluster_agent_test.go",
109110
"cluster_agent_validate_test.go",
110111
"cluster_dump_test.go",
112+
"cluster_registration_list_test.go",
111113
"cluster_registration_test.go",
112114
"deploy_test.go",
113115
"exit_code_error_test.go",

src/clis/nvcf-cli/cmd/cluster.go

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,8 @@ var clusterCmd = &cobra.Command{
3434
Long: `Manage and query cluster resources and GPU availability.
3535
3636
Available subcommands:
37-
- list: List available cluster groups`,
37+
- list: List available cluster groups
38+
- registration: Manage self-hosted cluster registrations`,
3839
}
3940

4041
var clusterListCmd = &cobra.Command{

src/clis/nvcf-cli/cmd/cluster_registration.go

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -86,6 +86,11 @@ func initClusterRegistrationCmds() {
8686
clusterCmd.AddCommand(clusterRegisterCmd)
8787
clusterCmd.AddCommand(clusterRotateCmd)
8888
clusterCmd.AddCommand(clusterDeleteCmd)
89+
clusterCmd.AddCommand(clusterRegistrationCmd)
90+
clusterRegistrationCmd.AddCommand(clusterRegistrationListCmd)
91+
clusterRegistrationListCmd.Flags().String(clusterFlagNcaID, "", "NCA/tenant ID (required)")
92+
addClusterICMSURLFlags(clusterRegistrationListCmd)
93+
_ = clusterRegistrationListCmd.MarkFlagRequired(clusterFlagNcaID)
8994

9095
// Register flags
9196
clusterRegisterCmd.Flags().String("name", "", "Cluster name (required)")
Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,92 @@
1+
/*
2+
SPDX-FileCopyrightText: Copyright (c) NVIDIA CORPORATION & AFFILIATES. All rights reserved.
3+
SPDX-License-Identifier: Apache-2.0
4+
5+
Licensed under the Apache License, Version 2.0 (the "License");
6+
you may not use this file except in compliance with the License.
7+
You may obtain a copy of the License at
8+
9+
http://www.apache.org/licenses/LICENSE-2.0
10+
11+
Unless required by applicable law or agreed to in writing, software
12+
distributed under the License is distributed on an "AS IS" BASIS,
13+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
See the License for the specific language governing permissions and
15+
limitations under the License.
16+
*/
17+
18+
package cmd
19+
20+
import (
21+
"context"
22+
"encoding/json"
23+
"fmt"
24+
25+
"nvcf-cli/internal/client"
26+
27+
"github.com/spf13/cobra"
28+
)
29+
30+
var clusterRegistrationCmd = &cobra.Command{
31+
Use: "registration",
32+
Short: "Manage self-hosted cluster registrations",
33+
SilenceUsage: true,
34+
}
35+
36+
var clusterRegistrationListCmd = &cobra.Command{
37+
Use: "list",
38+
Short: "List self-hosted cluster registrations from ICMS",
39+
SilenceUsage: true,
40+
RunE: runClusterRegistrationList,
41+
}
42+
43+
func runClusterRegistrationList(cmd *cobra.Command, _ []string) error {
44+
ncaID, _ := cmd.Flags().GetString(clusterFlagNcaID)
45+
46+
config, err := client.LoadConfig()
47+
if err != nil {
48+
return fmt.Errorf(errFailedToLoadConfig, err)
49+
}
50+
if err := requireAdminToken(config); err != nil {
51+
return err
52+
}
53+
54+
c, err := client.NewClient(config)
55+
if err != nil {
56+
return fmt.Errorf(errFailedToCreateClient, err)
57+
}
58+
defer c.Close()
59+
60+
ctx, cancel := context.WithTimeout(context.Background(), config.DefaultTimeout)
61+
defer cancel()
62+
63+
clusters, err := c.ListClusters(ctx, getICMSURL(cmd, config), ncaID)
64+
if err != nil {
65+
return fmt.Errorf("failed to list registered clusters from ICMS: %w", err)
66+
}
67+
68+
if jsonOutput {
69+
return json.NewEncoder(cmd.OutOrStdout()).Encode(map[string]interface{}{"clusters": clusters})
70+
}
71+
72+
if len(clusters) == 0 {
73+
fmt.Fprintln(cmd.OutOrStdout(), "No registered clusters found.")
74+
return nil
75+
}
76+
77+
fmt.Fprintf(cmd.OutOrStdout(), "Found %d registered cluster(s):\n\n", len(clusters))
78+
for _, cluster := range clusters {
79+
fmt.Fprintf(cmd.OutOrStdout(), "Name: %s\n", cluster.ClusterName)
80+
fmt.Fprintf(cmd.OutOrStdout(), "ID: %s\n", cluster.ClusterID)
81+
fmt.Fprintf(cmd.OutOrStdout(), "Cluster Group ID: %s\n", cluster.ClusterGroupID)
82+
if cluster.ClusterStatus != "" {
83+
fmt.Fprintf(cmd.OutOrStdout(), "Status: %s\n", cluster.ClusterStatus)
84+
}
85+
if cluster.NVCAVersion != "" {
86+
fmt.Fprintf(cmd.OutOrStdout(), "NVCA Version: %s\n", cluster.NVCAVersion)
87+
}
88+
fmt.Fprintln(cmd.OutOrStdout(), "---")
89+
}
90+
91+
return nil
92+
}

0 commit comments

Comments
 (0)