Evals (Layer 1): payload snapshot tests for prompt regressions (closes #46 Layer 1) - #50
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #46 (Layer 1). Layer 2 (promptfoo conversation evals) tracked in #49.
Summary
Ships golden-trace snapshot tests for the assembled provider payload. For each canonical
(agent, channel)pair we freeze a user message, build the request payload openclaWP would send to the provider (system prompt + messages + tool catalog + model preference + runtime context), and diff it against a committed JSON snapshot undertests/integration/prompt-assembly/__snapshots__/. Any change to a system-prompt template line, a tool description, or a default config produces a failing snapshot diff on the next CI run.Canonical pairs covered:
openclawp-loop-demo×chat— single ability toolopenclawp-site-introspection×whatsapp— multi-ability catalog, channel runtime contextopenclawp-coordinator×chat— subagents-as-delegate-to-*toolsopenclawp-workflow-drafter×chat— large structured system prompt, no toolsopenclawp-example×whatsapp— minimal config sanity checkWhat this adds
tests/integration/prompt-assembly/PromptAssemblySnapshotTest.php— the snapshot suite. Drives registrations through the realOpenclaWP_Agent_Registrar(opt-in filters flipped on insetUp()), so production prompt edits are what fails the test.tests/integration/prompt-assembly/PromptPayloadAssembler.php— pure-PHP payload builder mirroringOpenclaWP_Runner::build_turn_runner()minus the provider call. Test-only; not loaded at runtime.tests/integration/prompt-assembly/bootstrap.php— minimal in-memory stubs forwp_get_ability/wp_get_agent(and friends) plus the WP AI ClientFunctionDeclarationDTO stub. Keeps the suite hermetic.tests/integration/prompt-assembly/__snapshots__/*.json— the five pinned snapshots, pretty-printed, sorted keys, tabs for indent, trailing newline.composer test:assemblyscript + dedicated CI step in.github/workflows/tests.ymlinside the existingphpunitjob.docs/dev/evals.md— update flow (UPDATE_SNAPSHOTS=1 composer test:assembly) + Layer 2 pointer to Prompt evals (Layer 2): promptfoo conversation evals in CI #49.What this does NOT add
assertSame+file_put_contentsfor snapshot updates.Performance
The new suite runs 5 tests / 15 assertions in ~3-5ms locally — well under the 2s budget called out in the task.
Test plan
composer test:assembly— snapshots match HEAD.php tests/smoke.php— passes (exit 0).vendor/bin/phpunit --testsuite unit— 32 tests pass.vendor/bin/phpunit(default) — 37 tests pass.npm run lint— clean.OpenclaWP_Agent_Registrar'sopenclawp-loop-demodescription from "precise assistant" → "regressed assistant" produces failing diffs on both theloop-demo--chatandcoordinator--chatsnapshots (the latter because the subagent description is inlined into the coordinator'sdelegate-to-*tool description).UPDATE_SNAPSHOTS=1 composer test:assemblyregenerates byte-identical snapshots when nothing changed (md5 stable).Update flow (for reviewers)
When a future PR intentionally changes a system prompt, tool description, or default config:
The reviewer reads the diff to confirm the prompt change is intentional. Treat snapshot churn the same way you'd treat a copy edit on a customer-facing string.
Follow-up
Layer 2 (promptfoo conversation evals against a booted wp-env, with
contains/regex/latency</cost</llm-rubricassertions) tracked in #49.