feat(deploy): post-deploy smoke tests from project manifest (#753)

[solaawojobi00-bit]

Post-deploy smoke tests from project manifests

Problem

starforge deploy --execute stops as soon as the Stellar CLI reports success. Nothing checks that the deployed contract actually works. Broken initialisation or configuration is found later, usually by a user. A project also has no way to declare the checks it expects after a deploy.

Scenario	Before	After
Deploy succeeds, contract works	Exit 0	Smoke tests run and pass, exit 0
Deploy succeeds, contract misbehaves	Exit 0 (not caught)	SMOKE FAIL line per failing test, exit 9
Deploy fails	Deploy error (exit 6)	Unchanged; smoke tests never run
Invalid smoke test declaration	n/a	Rejected at manifest load, before deploying (exit 3)
Smoke test hangs	n/a	Killed at timeout_secs, counted as a smoke failure
Dry run, no --execute, --json	No checks	Unchanged; smoke tests never run
--skip-smoke	n/a	Smoke tests not loaded or run
Manifest without smoke_tests	Deploy as usual	Identical behaviour

Solution

• Schema. starforge-project.toml gains an optional [[smoke_tests]] array. Each entry has:

  • a unique name
  • exactly one of invoke (a contract function plus args, run as stellar contract invoke --id <new contract>) or command (a shell command, run from the manifest's directory)
  • optional assertions expect_exit (default 0), expect_output (trimmed exact match) and expect_contains
  • an optional timeout_secs (1–3600, default 60)

  Unknown keys are rejected, the same as the rest of the manifest.

• Validation before deploy. The manifest is loaded and validated before any deploy work starts, so a bad declaration can't surface after a contract is live.

• Runs only after a confirmed deploy. Tests run only once the Stellar CLI exits successfully and a contract ID is parsed from its output. If no contract ID can be parsed, the tests are skipped with a warning. Every test runs even after one fails, so one report shows everything broken.

• Separate classification. A smoke failure returns a typed SmokeTestFailure error, which maps to new exit code 9 SMOKE_TEST_FAILURE. It is matched by type before the message heuristics, so its text (which mentions a contract and a network) is never misread as an execution or network error. The deploy record stays Success, and both the error and the recovery hints say the contract is live.

• Environment. command checks receive STARFORGE_CONTRACT_ID, STARFORGE_NETWORK and STARFORGE_SOURCE. invoke arguments are passed as an argv vector with no shell, so manifest values can't inject commands.

Changes

src/utils/smoke_tests.rs (new)

• Schema: SmokeTest and InvokeCheck, both with deny_unknown_fields.
• validate(): names are non-empty and unique, exactly one of command/invoke is set, neither is empty, and the timeout is in range.
• Runner: run_all() and run_with_timeout(). The timeout is enforced by polling, and pipes are drained on background threads so chatty output can't block. This follows the existing pattern in mutate.rs.
• SmokeReport produces the SMOKE PASS / SMOKE FAIL lines. SmokeTestFailure is the typed error.

src/utils/project_config.rs

#[serde(default, skip_serializing_if = "Vec::is_empty")]
pub smoke_tests: Vec<SmokeTest>,

parse_project_lockfile_str now also runs smoke_tests::validate, so every load path enforces the rules. The field is not part of to_overlay(), so it never affects the merged config.

src/utils/exit_codes.rs
New ExitCode::SmokeTestFailure = 9 with a name and description. determine_exit_code checks the error chain for SmokeTestFailure before the message heuristics:

if err.chain().any(|c| c.is::<crate::utils::smoke_tests::SmokeTestFailure>()) {
    return ExitCode::SmokeTestFailure;
}

src/commands/deploy.rs

• Adds the --skip-smoke flag.
• load_smoke_tests() runs straight after the WASM-exists check, before optimisation, policy checks or signing.
• run_post_deploy_smoke_tests() runs after update_status(Success) and only when a contract ID was parsed.
• Non-execute runs print how many tests are declared and that they run only on an executed deploy.

src/main.rs
Adds recovery hints for smoke failures in the deploy branch. They say the contract is deployed, point to the manifest entry, and mention --skip-smoke. The next condition was reformatted to satisfy rustfmt.

Docs

• docs/SMOKE_TESTS.md (new): the schema table, when tests run, validation rules, output format and the exit-code table.
• docs/COMMAND_REFERENCE.md and docs/CONFIGURATION.md: short sections that point to it.
• docs/SUMMARY.md: adds the new page to the book.

examples/smoke-tests/ (new)
A sample project that targets the simple-counter template:

• a passing manifest with get_count == 0, increment == 1 and a contract-ID shell check
• failing/starforge-project.toml, which deliberately expects 42 and demonstrates exit 9
• a README

Tests

• tests/cli_exit_codes.rs: code 9 value, name and classification, including when wrapped in context, while a real deploy failure stays Execution.
• Unit tests in both modules (see below).

Regression Tests

Acceptance criterion / requirement	Test(s)
Manifest schema for smoke test commands/assertions	smoke_tests_parse_from_manifest, valid_declarations_pass_validation, unknown_fields_are_rejected_by_serde
Invalid declarations fail before deploy	invalid_smoke_tests_fail_manifest_validation, missing_check_is_rejected, both_command_and_invoke_is_rejected, duplicate_and_empty_names_are_rejected, out_of_range_timeout_is_rejected
Sample project demonstrates smoke tests	sample_project_manifests_are_valid (parses both example manifests)
Failures exit non-zero with clear classification	test_classify_smoke_test_failure_distinct_from_deploy_failure, failure_error_is_typed_and_says_deploy_succeeded, test_exit_code_numeric_values
Assertions evaluated correctly	passing_command_passes, wrong_exit_code_fails_with_reason, expected_nonzero_exit_can_pass, output_mismatch_fails
Timeout classified as smoke failure	timeout_is_a_smoke_failure
Deployed contract passed to checks	contract_id_is_exported_to_commands, invoke_args_target_the_deployed_contract
All failures surfaced in one report	all_tests_run_after_a_failure
Docs describe manifest schema	docs/SMOKE_TESTS.md

Testing

smoke_tests module tests, plus the manifest-level parse tests, run in isolation with the module's serde/toml/anyhow dependencies (Windows 10):

running 16 tests
test tests::both_command_and_invoke_is_rejected ... ok
test tests::duplicate_and_empty_names_are_rejected ... ok
test tests::contract_id_is_exported_to_commands ... ok
test tests::invoke_args_target_the_deployed_contract ... ok
test tests::manifest_workdir_is_its_parent ... ok
test tests::missing_check_is_rejected ... ok
test tests::out_of_range_timeout_is_rejected ... ok
test tests::expected_nonzero_exit_can_pass ... ok
test tests::failure_error_is_typed_and_says_deploy_succeeded ... ok
test tests::all_tests_run_after_a_failure ... ok
test tests::unknown_fields_are_rejected_by_serde ... ok
test tests::valid_declarations_pass_validation ... ok
test tests::output_mismatch_fails ... ok
test tests::passing_command_passes ... ok
test tests::wrong_exit_code_fails_with_reason ... ok
test tests::timeout_is_a_smoke_failure ... ok
test result: ok. 16 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.19s
test inline_and_invalid ... ok
test samples ... ok
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

• cargo clippy --all-targets -- -D warnings on the module: no warnings.
• rustfmt --check: clean on all touched hunks.
• scripts/docs-snippets.py --lint-only: no findings in the new or edited docs.

Notes for Reviewers

• Exit code 9 is new. Codes 0–8 were already taken; 3 is CONFIG_ERROR. Code 9 is checked by type before any message heuristics.
• The deploy record stays Success on a smoke failure. The contract is on-chain, so marking the deploy failed (or triggering the auto-rollback path) would misreport what happened. Only the command's exit code changes.
• deploy now loads starforge-project.toml. It uses the existing upward discovery, and only when --skip-smoke is not set. A broken manifest therefore now stops a deploy, which matches how load_effective() treats it elsewhere.
• Invoke source. invoke checks use the same --source value the deploy command uses (wallet.public_key), for consistency.
• Missing contract ID. If the Stellar CLI succeeds but no contract ID can be parsed, the tests are skipped with a warning rather than failed, because there is nothing to test against.

Closes #753

[drips-wave]

@solaawojobi00-bit Great news! 🎉 Based on an automated assessment of this PR, the linked Wave issue(s) no longer count against your application limits.

You can now already apply to more issues while waiting for a review of this PR. Keep up the great work! 🚀

Learn more about application limits