Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 19 additions & 11 deletions src/clis/nvcf-cli/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -1730,14 +1730,21 @@ cluster identity and the system and requests namespaces.

### How drain works

`cordon-and-drain` adds the `CordonAndDrainMaintenance` feature flag and sets
`maintenanceMode: CordonAndDrain` on the NVCA `agent-config` ConfigMap, then
restarts the NVCA deployment so the change takes effect. `uncordon` reverses
both. The command returns once NVCA has been told to drain and (unless `--force`)
the restart has rolled out; it does not wait for every instance to reach zero.
Watch progress with `cluster agent list-functions --phase DRAINING`. `--timeout`
bounds the rollout wait (default 5m); a timeout is reported as a warning because
the config change is already persisted and re-running is a no-op.
`cordon-and-drain` adds the `CordonAndDrainMaintenance` feature flag to the
`NVCFBackend` CR's `spec.overrides.featureGate.values`. `uncordon` removes it.
The CLI never edits the NVCA `agent-config` ConfigMap or restarts the NVCA
deployment directly: the NVCA operator treats `agent-config` as fully
generated from the CR and reverts any direct edit on its next reconcile, so
the CLI's job is only to submit the desired state and let the operator's own
reconcile regenerate `agent-config` and roll NVCA out. The command returns
once the CR update is accepted and (unless `--force` or `--timeout 0`) the
operator's rollout has completed; it does not wait for every instance to
reach zero. Watch progress with `cluster agent list-functions --phase
DRAINING`. `--timeout` bounds the wait for the operator's rollout (default
5m); `--force` or `--timeout 0` skip the wait entirely and return right after
the CR update, leaving the operator's reconciliation to finish
asynchronously. A timeout is reported as a warning because the CR change is
already persisted and re-running is a no-op.

### How kill works

Expand All @@ -1762,9 +1769,10 @@ All maintenance commands accept `--dry-run` to preview without mutating, and
name matches (guards against a wrong `--compute-plane-context`). `kill-function`
and `kill-all` accept `--reason` for an audit note, and `--json` for automation.

These commands need write access to the target cluster: get/update on the
`agent-config` ConfigMap and the `nvca` Deployment for drain, and list/delete
(and update, with `--force`) on `ICMSRequest` CRs for kill.
These commands need write access to the target cluster: list/update on the
`NVCFBackend` CR for drain (plus read access to the `agent-config` ConfigMap
and the `nvca` Deployment, to wait for the NVCA operator's rollout), and
list/delete (and update, with `--force`) on `ICMSRequest` CRs for kill.

### Examples

Expand Down
16 changes: 9 additions & 7 deletions src/clis/nvcf-cli/cmd/cluster_agent_maintenance.go
Original file line number Diff line number Diff line change
Expand Up @@ -55,9 +55,10 @@ var clusterAgentCordonDrainCmd = &cobra.Command{
deployments, let in-flight requests complete, and scale all function instances
to zero.

This sets the CordonAndDrainMaintenance feature flag and maintenanceMode on the
NVCA agent-config ConfigMap and restarts the NVCA deployment. The command returns
once NVCA has been told to drain (and, by default, once the restart rolls out);
This sets the CordonAndDrainMaintenance feature flag on the NVCFBackend CR's
spec.overrides.featureGate.values; the NVCA operator's own reconcile then
regenerates agent-config and restarts NVCA. The command returns once the CR
update is accepted (and, by default, once the operator's rollout completes);
use "cluster agent list-functions --phase DRAINING" to watch instances wind down.

Select the cluster with --compute-plane-context, as with the inspection commands.`,
Expand All @@ -71,7 +72,8 @@ var clusterAgentUncordonCmd = &cobra.Command{
SilenceUsage: true,
Args: cobra.NoArgs,
Long: `Reverse a cordon-and-drain: remove the CordonAndDrainMaintenance feature
flag and maintenanceMode from the NVCA agent-config ConfigMap and restart NVCA so
flag from the NVCFBackend CR's spec.overrides.featureGate.values; the NVCA
operator's own reconcile then regenerates agent-config and restarts NVCA so
the cluster accepts new deployments again.`,
RunE: runClusterAgentUncordon,
}
Expand Down Expand Up @@ -447,13 +449,13 @@ func printDrainResult(cmd *cobra.Command, res *clusteragent.DrainResult, drain b
return
}
if res.DryRun {
fmt.Fprintln(w, " would update agent-config and restart NVCA")
fmt.Fprintln(w, " would update the NVCFBackend CR; the NVCA operator would then roll out the change")
return
}
if drain {
fmt.Fprintf(w, " agent-config updated (maintenanceMode=%s); NVCA restart triggered\n", orDash(res.Mode))
fmt.Fprintf(w, " NVCFBackend updated (maintenanceMode=%s)\n", orDash(res.Mode))
} else {
fmt.Fprintln(w, " agent-config updated (maintenance cleared); NVCA restart triggered")
fmt.Fprintln(w, " NVCFBackend updated (maintenance cleared)")
}
Comment thread
coderabbitai[bot] marked this conversation as resolved.
switch {
case res.RolloutComplete:
Expand Down
Loading
Loading