feat(e2e): add N-run averaging and Activation vs. Outcome tagging to test harness#68
Open
Saurav Gupta (Saurav-Gupta-9741) wants to merge 4 commits into
Conversation
…ough the workflow skills A declarative harness (e2e/workflows/harness/, barrel index.ts) where one defineScenario call is a workflow test: it drives a REAL in-process pi agent through the production wiring — no browser, no HTTP host — via the new @thinkrail/server/agent subpath export (a deliberate second entry that avoids evaluating the Bun-only host under the node-run e2e worker; recorded in packages/server/SPEC.md). Verdict model (the invariant everything serves): an LLM never decides pass. - Binding: deterministic checks + signals over the captured event log and workspace. - Input/diagnosis only: persona responder + user simulator (cheap LLM) for user-side input; the watchdog aborts-with-diagnosis; the judge grades a rubric advisorily (warn + run log, never a fail). Pieces: session lifecycle (isolated PI_CODING_AGENT_DIR, never setModel), one process-wide event log, throwaway git workspaces, an ask_user_question dialog ladder (script → persona → deterministic fallback — an unscripted interview can never hang), mid-flow presets (artifact + recorded-transcript fixtures, markdown masked as *.md.test to stay out of the host's spec graph), stop/forbid signals, budget watchdog, and a gitignored per-run log (e2e/.workflow-runs.jsonl). Run via bun run test:workflows (own Playwright config, shares the browser suite's global setup). On-demand only — needs pi auth, spends real tokens; the main config testIgnores this directory so it never enters bun run e2e / e2e:agent / CI gates. harness.unit.spec.ts covers the pure parts (no agent, no auth). Design + boundary: e2e/workflows/SPEC.md (module-workflow-tests).
routing.live.spec.ts — one scenario per row of the two routers' classification tables, all observed green. Root router (reached naturally via the always-on rule): feature → brainstorming (abort at the worker's skill read); onboarding → the setup family; pure question → no worker skill loads, answer grounded in the repo. Dispatcher (force-loaded via the app's exact /skill: seed): empty → starting-a-new-project; code-only → importing-a-codebase; specced → review/extend offer with both setup workers forbidden. One advisory judge finding kept visible on purpose: questions bypass the root router (WORKFLOW_RULE doesn't list questions), recorded in the skills SPEC as an open gap. smoke.live.spec.ts — the infra-proving scenarios: a mid-flow brainstorming round-trip (artifact preset + persona answer + user simulator + on-disk decision) and transcript continuation (a reopened recorded fixture recalls mid-flow state). fixtures/brainstorming-mid-flow/ is the first recorded fixture (session.jsonl + workspace snapshot, markdown masked as *.md.test).
…test harness Each fact in one home, cross-links instead of restatement: - module-thinkrail-workflow § Testing points at module-workflow-tests for the harness design; it keeps only what it owns (the index.test.ts pin, the app-path @agent e2e, slice-3 follow-up). - submodule-workflow-skills: rule 14 (verify-by-use) rationale updated — observation is now cheap via the harness; the gate stays suspended only for want of a durable run record. Family table rows carry per-skill observation status (routing suite / smokes / 2026-07 manual observations — asking-user-questions and writing-specs now observed by use; brainstorming's missing writing-specs reference confirmed as an observed defect). New recorded gap: questions bypass the root router (advisory judge finding, kept re-flagging until its own brainstormed fix). - architecture.md: bundled-extension list gains pi-thinkrail-workflow; repo-layout diagram gains pi-visualize + pi-thinkrail-workflow; 'Out of scope (V1)' scoped to the workflow product layer — the skill-based system ships in V1 (no runtime machinery). - goal-and-requirements.md: same scoping, so the V2 Workflow layer plan no longer contradicts the shipped extension. - CLAUDE.md: Verification section documents bun run test:workflows (on-demand, needs auth, never a gate).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #67 (Slice 2 items).
This PR introduces the first batch of Eval-style enhancements for the headless workflow-test harness.
Changes
tagproperty toCheckandSignalinterfaces. Built-in factories now apply sensible defaults (e.g.,expectSkillReaddefaults toactivation, whileexpectFiledefaults tooutcome). TherunScenariofailure output automatically groups deterministic failures by these tags.repeat: N): ExtendedScenarioDefto support arepeatparameter. To avoid polluting.workflow-runs.jsonlwith duplicate lines for identical runs,runlog.tsnow features anappendAggregatedRunRecordutility. This seamlessly aggregates pass rates, duration averages, and judge rubrics into a single unified telemetry line for the entire repeated batch.Tested locally via
bun run test:workflowsto verify the execution loop unrolls correctly and logs gracefully.