test(bdd): prototype portable live smoke composition

Add a target-driven live suite with explicit provider and smoke phases, bounded retry and compensation primitives, and a portable cluster-maintenance example.

Relates to #1186

Signed-off-by: Stephanie Baum <sbaum@nvidia.com>

Author: sbaum1994

tests/bdd/AGENTS.md

@@ -42,6 +42,9 @@ so `nvcf-cli` and the NVCF API remain the product-validation boundary.
   `Suite.Runner`.
 - `godog_test.go` owns the live entry points and the fake-runner
   wiring tests.
+- `live/` owns the separate portable runner. It may reuse `harness`, `dsl`,
+  and `steps`, but must not add target modes or feature selection to
+  `godog_test.go`.
 
 A step handler that does anything beyond argv assembly, ledger
 snapshot, runner invocation, and result capture is a smell. Move the
@@ -113,6 +116,9 @@ logic into `dsl/`.
   `tools/ncp-local-cluster/Makefile` and
   `deploy/stacks/self-managed/Makefile` are intentionally maintained
   so an operator can clean by hand without involving `go test`.
+- The portable `live/` suite rejects `BDD_CLEANUP_MODE`. Those cleanup
+  commands are specific to ncp-local. Remote and production targets must never
+  inherit local cleanup behavior.
 - Cleanup belongs in `harness/cleanup.go`, never in `steps/`. Do not
   introduce a `Given the cluster is freshly destroyed` Given or
   similar; the conflict precheck inside every feature Background is
@@ -223,10 +229,37 @@ multi-cluster feature:
   check that a destructive command was issued. Do not deep-equality
   the recorder; consolidating equivalent steps in the future must not
   break these tests.
+- Portable wiring tests live in `live/` and use the same fake
+  `CommandRunner`. Portable features run one file per Godog phase. The runner
+  restores step-exported environment values and ledger-backed files between
+  files so one selectable smoke cannot become an undocumented provider for the
+  next.
 - Live entry points (`TestSingleClusterUp`, `TestMultiClusterUp`,
   `TestSingleClusterHelmfile`) skip under `-short`. They build the
   CLI and exercise real `make`/`kubectl`/`helm` against k3d.
 
+## Portable live features
+
+- Target YAML is versioned, non-secret data. It supplies execution coordinates
+  and workload inputs only. It must not select features or contain destructive
+  consent.
+- Features are selected by the operator independently of the target. One
+  optional provider runs before one or more smoke features. Do not rely on
+  Godog file ordering.
+- Every portable smoke is independently selectable. It declares required
+  target fields through environment preconditions and does not depend on an
+  earlier feature's exported variables or files.
+- Every Kubernetes command exposes both kubeconfig and context in Gherkin.
+  Ambient current context is not portable and is unsafe for remote targets.
+- Cluster-wide mutations require separate invocation-time consent and use the
+  product's expected-cluster identity guard before mutation.
+- Register compensation before its destructive command. Keep the action,
+  target, and timeout visible. Compensation handlers hide only reverse-order
+  execution and continue-after-failure mechanics.
+- Do not add capability auto-detection, target-based feature skipping, or
+  product validation to the runner. Missing inputs fail the feature's visible
+  preconditions; unsupported behavior fails through the real CLI or API.
+
 ## Style
 
 - Plain ASCII only in committed text. No bold, no em dashes, no

tests/bdd/PLAN.md

@@ -119,6 +119,8 @@ refactor in every consumer; that is a feature.
 | `When I run command:` (docstring) | Multi-line form for commands that don't fit on one line. Same recording semantics. |
 | `When I successfully run command {string}` | Single-line command that must exit 0. Preserves the command result for later output assertions and records the resolved command in the successful-command cache. |
 | `When I successfully run command:` (docstring) | Multi-line form for commands that must exit 0. Uses the same result recording, interpolation, logging, and cache semantics as the single-line form. |
+| `And after this scenario I successfully run command within {string}:` (docstring) | Registers the visible command as bounded scenario compensation. Compensations run in reverse registration order after success or failure, continue past failures, and preserve product behavior in the command text. |
+| `Then within {string} this command should succeed, checking every {string}:` (docstring) | Repeats the visible command after product-level nonzero exits until it exits 0 or the visible timeout expires. Runner failures stop immediately. The successful result remains available to output assertions. |
 | `When I run command with a terminal:` (docstring) | Same as the docstring form, but stdin is attached to a pseudo-terminal so the child sees a TTY on fd 0. For commands that gate interactive-only behavior on a TTY, such as `nvcf-cli self-hosted up` (its auth-gate mints the admin token only when stdin is a terminal). No input is written; stdout and stderr are captured separately as usual. |
 | `When I export command output to environment variable {string}` | Exports the previous command's trimmed stdout under the named env var. Fails the step unless the prior command exited 0 and produced non-empty stdout. Snapshotted by the env Ledger; restored at suite teardown. |
 
@@ -158,6 +160,7 @@ original order. Repeated options and empty values are preserved.
 | `Then the command exit code should be {int}` | Last-run exit code. |
 | `Then the command output should contain {string}` | Substring match on combined stdout + stderr. |
 | `Then the command output should not contain {string}` | Negative substring match. |
+| `Then the JSON command output should contain:` (JSON docstring) | Parses the last command's stdout and the expected docstring as JSON, then requires the typed expected object fields while allowing additional actual fields. Lists remain order- and length-sensitive. |
 | `Then file {string} should exist` | |
 | `Then yaml file {string} key {string} should equal {string}` | Reads the YAML file, walks the dotted key path, compares to the value (with `${VAR}` expansion). |
 | `Then yaml file {string} key {string} should not be empty` | Same key resolution; passes if the resolved value is non-empty. Use for non-deterministic outputs (cluster IDs, identity sources) where exact-value assertions are wrong. |
@@ -206,10 +209,18 @@ them via `${VAR}` in command strings and table cells.
 | `REPO_ROOT` | `git rev-parse --show-toplevel` at suite start | Absolute path to the repo root. Required when invoking `make -C deploy/stacks/self-managed` because the Makefile's `-C` changes cwd; relative paths to fixtures from there break. |
 | `NGC_API_KEY` / `SAMPLE_NGC_ORG` / `SAMPLE_NGC_TEAM` | The operator's shell | Passed through unchanged. An environment-variable precondition step asserts they are non-empty before any scenario uses them. |
 
+The portable live runner additionally maps its selected target to stable
+`BDD_NVCF_*`, `BDD_COMPUTE_*`, `BDD_NVCA_*`, and `BDD_WORKLOAD_*` variables.
+It runs each selected feature separately and restores step-exported environment
+variables and ledger-backed file changes between features. CLI state and the
+successful bootstrap command cache remain suite-scoped. See `live/README.md`
+for the target schema and safety contract.
+
 Feature files may also export their own env vars at runtime via
 `When I export command output to environment variable {string}`. Those
-are snapshotted by the env Ledger and restored at teardown so they do
-not leak into later test binaries in the same `go test` invocation.
+are snapshotted by the env Ledger and restored at teardown. The portable live
+runner restores them after each selected feature so they cannot become an
+implicit input to another smoke.
 The EKS Helmfile feature exports `EKS_GATEWAY_ADDR` this way from
 `kubectl get gateway` after the Gateway resource is Programmed.
 
@@ -255,8 +266,9 @@ restoration ledger:
   body, mode) in memory. Repeat writes against the same path during a
   suite do not overwrite the snapshot; only the first write records.
 - At suite teardown, the runner restores every registered path to its
-  pre-suite state. Files that did not exist before are deleted; files
-  that did are rewritten with the original bytes and mode.
+  pre-suite state. The portable live runner performs the same restoration
+  between selected features. Files that did not exist before are deleted;
+  files that did are rewritten with the original bytes and mode.
 - `Config.LedgerDir` (`out/<run-id>/originals/`) is reserved for an
   on-disk variant if very large fixtures ever push memory limits.
   Today the directory is created but unused.
@@ -344,13 +356,16 @@ old `tests/bdd` tree.
 tests/bdd/
   features/                      (Gherkin, already committed)
   fixtures/                      (sample env + CLI config, already committed)
+  targets/                       (non-secret portable live coordinates)
   STEPS.md                       (this document)
 
   harness/                       (phase 1)
     config.go                    (paths, env exports)
     runner.go                    (CommandRunner around infra.Runner)
     ledger.go                    (file restoration ledger)
     cache.go                     (command-success cache)
+    deferred.go                  (bounded scenario compensation stack)
+    eventually.go                (bounded command retry)
     suite.go                     (lifecycle: build CLI, set env, teardown)
 
   dsl/                           (phase 1)
@@ -365,6 +380,8 @@ tests/bdd/
     assertion_steps.go           (exit code, output contains, file/yaml/json)
     infra_steps.go               (cluster bootstrap, image pull secret)
 
+  live/                          (portable target loader and phased runner)
+
   godog_test.go                  (phase 3 and 4: TestSingleClusterUp,
                                   TestMultiClusterUp, TestSingleClusterHelmfile,
                                   wiring tests with fakes)
@@ -432,14 +449,17 @@ func (c *CommandCache) Has(commandText string) bool
 // Suite is the top-level lifecycle owner. Built once per go test
 // invocation; runs one or more feature files.
 type Suite struct {
-    Config Config
-    Runner CommandRunner
-    Ledger *Ledger
-    Cache  *CommandCache
+    Config    Config
+    Runner    CommandRunner
+    Ledger    *Ledger
+    EnvLedger *EnvLedger
+    Cache     *CommandCache
 }
 
 func NewSuite(t *testing.T) (*Suite, error) // builds nvcf-cli, sets NVCF_CLI + REPO_ROOT
-func (s *Suite) Teardown() error            // calls Ledger.RestoreAll
+func NewSuiteWithOptions(t *testing.T, options SuiteOptions) (*Suite, error)
+func (s *Suite) RestoreFeatureState() error // restores step-owned files and env vars
+func (s *Suite) Teardown() error            // also restores the exact CLI state file
 ```
 
 #### dsl package

tests/bdd/README.md

@@ -12,7 +12,7 @@ in `AGENTS.md`.
 ```
 features/   Gherkin feature files: single-cluster CLI, multi-cluster CLI,
             single/multi-cluster Helmfile (k3d), and single/multi-cluster
-            EKS Helmfile (non-local).
+            EKS Helmfile (non-local), plus target-neutral smoke features.
 fixtures/   Starting environment YAML and CLI config the features copy from.
 harness/    Suite lifecycle: Config, CommandRunner, Ledger, CommandCache,
             Suite. Builds nvcf-cli at suite start; exports NVCF_CLI and
@@ -22,6 +22,9 @@ dsl/        Pure helpers: ${VAR} interpolation, dotted-path YAML upsert
             row matching, kubectl manifest builders.
 steps/      Godog step handlers. Every handler is a thin wrapper around a
             dsl helper or CommandRunner.Run; no domain validation.
+live/       Separate portable runner for optional provider plus independently
+            selectable smoke features. See live/README.md.
+targets/    Versioned, non-secret execution coordinates for portable smokes.
 godog_test.go
             Live entry points (TestSingleClusterUp, TestSingleClusterUpOneClick,
             TestMultiClusterUp, TestSingleClusterHelmfile, TestMultiClusterHelmfile,
@@ -44,9 +47,14 @@ cd tests/bdd
 go test -short ./...
 ```
 
-Live runs build nvcf-cli, bring up a real k3d cluster, and exercise the
-feature end to end. They require an NGC API key and sample registry
-coordinates.
+To run one smoke feature against an already installed local or remote target,
+or to compose a provider with several target-neutral smokes, see
+`live/README.md`.
+
+The install-suite live runs build nvcf-cli, bring up a real k3d cluster, and
+exercise the feature end to end. They require an NGC API key and sample
+registry coordinates. The portable live runner attaches to the target selected
+in its target file.
 
 Each `-run` argument is anchored with `^...$` so the live entry point
 runs without also matching its `...FeatureFileWiresToSteps` wiring

tests/bdd/dsl/interp.go

@@ -21,8 +21,11 @@ limitations under the License.
 package dsl
 
 import (
+	"fmt"
 	"os"
 	"regexp"
+	"sort"
+	"strings"
 )
 
 // braced matches the only environment variable form the DSL recognizes:
@@ -41,3 +44,27 @@ func Interpolate(s string) string {
 		return os.Getenv(name)
 	})
 }
+
+// InterpolateRequired expands the same syntax as Interpolate but rejects an
+// unset or empty referenced variable. Use it for lifecycle coordinates where
+// an empty expansion could point a mutation or restoration at the wrong path.
+func InterpolateRequired(s string) (string, error) {
+	missingSet := make(map[string]struct{})
+	resolved := braced.ReplaceAllStringFunc(s, func(match string) string {
+		name := match[2 : len(match)-1]
+		value := os.Getenv(name)
+		if value == "" {
+			missingSet[name] = struct{}{}
+		}
+		return value
+	})
+	if len(missingSet) == 0 {
+		return resolved, nil
+	}
+	missing := make([]string, 0, len(missingSet))
+	for name := range missingSet {
+		missing = append(missing, name)
+	}
+	sort.Strings(missing)
+	return "", fmt.Errorf("required interpolation variables are empty: %s", strings.Join(missing, ", "))
+}

tests/bdd/dsl/interp_test.go

@@ -46,3 +46,17 @@ func TestInterpolateBracedOnly(t *testing.T) {
 		})
 	}
 }
+
+func TestInterpolateRequired(t *testing.T) {
+	t.Setenv("BDD_REQUIRED", "value")
+	got, err := InterpolateRequired("prefix-${BDD_REQUIRED}")
+	if err != nil {
+		t.Fatalf("interpolate: %v", err)
+	}
+	if got != "prefix-value" {
+		t.Fatalf("got %q", got)
+	}
+	if _, err := InterpolateRequired("${BDD_REQUIRED_MISSING}/state"); err == nil {
+		t.Fatal("expected missing variable error")
+	}
+}

tests/bdd/dsl/jsoncmp.go

@@ -22,6 +22,21 @@ import (
 	"fmt"
 )
 
+// MatchJSONDocument compares an expected JSON document to command output.
+// MatchSubset requires every expected object field while allowing additional
+// actual fields. Mismatch errors identify paths without printing values.
+func MatchJSONDocument(actualJSON, expectedJSON string, mode MatchMode) error {
+	var expected any
+	if err := json.Unmarshal([]byte(Interpolate(expectedJSON)), &expected); err != nil {
+		return fmt.Errorf("parse expected json: invalid JSON")
+	}
+	var actual any
+	if err := json.Unmarshal([]byte(actualJSON), &actual); err != nil {
+		return fmt.Errorf("parse command output: invalid JSON")
+	}
+	return deepCompare(expected, actual, mode, "")
+}
+
 // JSONContainsRows parses raw as a JSON array of objects and, for each
 // row map, asserts that an object matching every (key, value) pair in
 // the row exists in the array. Extra objects in the array are

tests/bdd/dsl/jsoncmp_test.go

@@ -25,6 +25,26 @@ const helmListOutput = `[
   {"name": "api", "namespace": "nvcf", "status": "deployed"}
 ]`
 
+func TestMatchJSONDocumentContainsTypedSubset(t *testing.T) {
+	actual := `{"mode":"CordonAndDrain","configChanged":true,"count":0,"extra":"ok"}`
+	expected := `{"mode":"CordonAndDrain","configChanged":true,"count":0}`
+	if err := MatchJSONDocument(actual, expected, MatchSubset); err != nil {
+		t.Fatalf("match: %v", err)
+	}
+}
+
+func TestMatchJSONDocumentRejectsMismatchAndInvalidJSON(t *testing.T) {
+	if err := MatchJSONDocument(`{"configChanged":false}`, `{"configChanged":true}`, MatchSubset); err == nil {
+		t.Fatal("expected typed value mismatch")
+	}
+	if err := MatchJSONDocument("not json", `{}`, MatchSubset); err == nil {
+		t.Fatal("expected actual JSON parse error")
+	}
+	if err := MatchJSONDocument(`{}`, "not json", MatchSubset); err == nil {
+		t.Fatal("expected expected JSON parse error")
+	}
+}
+
 func TestJSONContainsRowsMatchesExpected(t *testing.T) {
 	rows := []map[string]string{
 		{"name": "nats", "namespace": "nats-system", "status": "deployed"},