Skip to content

Evals (Layer 1): payload snapshot tests for prompt regressions (closes #46 Layer 1) - #50

Merged
lezama merged 1 commit into
mainfrom
ship/issue-46-prompt-snapshots
May 18, 2026
Merged

Evals (Layer 1): payload snapshot tests for prompt regressions (closes #46 Layer 1)#50
lezama merged 1 commit into
mainfrom
ship/issue-46-prompt-snapshots

Conversation

@lezama

@lezama lezama commented May 17, 2026

Copy link
Copy Markdown
Owner

Closes #46 (Layer 1). Layer 2 (promptfoo conversation evals) tracked in #49.

Summary

Ships golden-trace snapshot tests for the assembled provider payload. For each canonical (agent, channel) pair we freeze a user message, build the request payload openclaWP would send to the provider (system prompt + messages + tool catalog + model preference + runtime context), and diff it against a committed JSON snapshot under tests/integration/prompt-assembly/__snapshots__/. Any change to a system-prompt template line, a tool description, or a default config produces a failing snapshot diff on the next CI run.

Canonical pairs covered:

  • openclawp-loop-demo × chat — single ability tool
  • openclawp-site-introspection × whatsapp — multi-ability catalog, channel runtime context
  • openclawp-coordinator × chat — subagents-as-delegate-to-* tools
  • openclawp-workflow-drafter × chat — large structured system prompt, no tools
  • openclawp-example × whatsapp — minimal config sanity check

What this adds

  • tests/integration/prompt-assembly/PromptAssemblySnapshotTest.php — the snapshot suite. Drives registrations through the real OpenclaWP_Agent_Registrar (opt-in filters flipped on in setUp()), so production prompt edits are what fails the test.
  • tests/integration/prompt-assembly/PromptPayloadAssembler.php — pure-PHP payload builder mirroring OpenclaWP_Runner::build_turn_runner() minus the provider call. Test-only; not loaded at runtime.
  • tests/integration/prompt-assembly/bootstrap.php — minimal in-memory stubs for wp_get_ability / wp_get_agent (and friends) plus the WP AI Client FunctionDeclaration DTO stub. Keeps the suite hermetic.
  • tests/integration/prompt-assembly/__snapshots__/*.json — the five pinned snapshots, pretty-printed, sorted keys, tabs for indent, trailing newline.
  • composer test:assembly script + dedicated CI step in .github/workflows/tests.yml inside the existing phpunit job.
  • docs/dev/evals.md — update flow (UPDATE_SNAPSHOTS=1 composer test:assembly) + Layer 2 pointer to Prompt evals (Layer 2): promptfoo conversation evals in CI #49.

What this does NOT add

Performance

The new suite runs 5 tests / 15 assertions in ~3-5ms locally — well under the 2s budget called out in the task.

$ composer test:assembly
.....                                                               5 / 5 (100%)
Time: 00:00.003, Memory: 6.00 MB
OK (5 tests, 15 assertions)

Test plan

  • composer test:assembly — snapshots match HEAD.
  • php tests/smoke.php — passes (exit 0).
  • vendor/bin/phpunit --testsuite unit — 32 tests pass.
  • vendor/bin/phpunit (default) — 37 tests pass.
  • npm run lint — clean.
  • Sanity-test the failure path: tweaking OpenclaWP_Agent_Registrar's openclawp-loop-demo description from "precise assistant" → "regressed assistant" produces failing diffs on both the loop-demo--chat and coordinator--chat snapshots (the latter because the subagent description is inlined into the coordinator's delegate-to-* tool description).
  • Sanity-test the missing-snapshot path: deleting a snapshot file produces an actionable failure with the regenerate command in the assertion message.
  • Sanity-test the update path: UPDATE_SNAPSHOTS=1 composer test:assembly regenerates byte-identical snapshots when nothing changed (md5 stable).

Update flow (for reviewers)

When a future PR intentionally changes a system prompt, tool description, or default config:

UPDATE_SNAPSHOTS=1 composer test:assembly
git add tests/integration/prompt-assembly/__snapshots__

The reviewer reads the diff to confirm the prompt change is intentional. Treat snapshot churn the same way you'd treat a copy edit on a customer-facing string.

Follow-up

Layer 2 (promptfoo conversation evals against a booted wp-env, with contains / regex / latency< / cost< / llm-rubric assertions) tracked in #49.

@lezama
lezama marked this pull request as ready for review May 18, 2026 10:38
@lezama
lezama merged commit d2b5a48 into main May 18, 2026
6 checks passed
@lezama
lezama deleted the ship/issue-46-prompt-snapshots branch May 18, 2026 10:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Prompt regression evals in CI + payload snapshot tests

1 participant