Skip to content

fix(desktop): make evaluation evidence explicit and fail closed - #123

Merged
xr843 merged 8 commits into
mainfrom
fix/desktop-evaluation-contract
Jul 30, 2026
Merged

fix(desktop): make evaluation evidence explicit and fail closed#123
xr843 merged 8 commits into
mainfrom
fix/desktop-evaluation-contract

Conversation

@xr843

@xr843 xr843 commented Jul 30, 2026

Copy link
Copy Markdown
Owner

Summary

  • emit a versioned fidelity-suite envelope with explicit mode and outcome fields
  • parse evaluation evidence into typed Rust models while retaining conservative legacy compatibility
  • keep structural dry-run evidence separate from graded evidence in coverage, filters, trends, and release gates
  • surface malformed, unsupported, mismatched, and failed evaluation evidence instead of silently treating it as absent
  • fail closed when fidelity JSONL is invalid and harden large/untrusted numeric inputs
  • document the reviewed two-PR refactor design and the PR1 implementation plan

Why

The desktop app inferred evaluation semantics from missing fields and string values. A dry run could therefore be mistaken for proof of quality, later dry runs could hide graded failures, and malformed evidence could disappear from the release decision. The new contract makes those states explicit and preserves the distinction end to end.

Impact

Release readiness now requires current passing graded evidence for every skill. Dry runs remain useful as structural validation but cannot satisfy that gate. Existing legacy trace history is still read conservatively; ambiguous legacy JSON is rejected visibly.

Validation

  • npm test — 60 Node tests passed; repository validators passed
  • python -m pytest tests/ scripts/tests/ -q — 373 passed
  • full --all --dry-run --json output parsed successfully
  • cargo fmt --check
  • cargo clippy --locked --all-targets -- -D warnings
  • cargo test --locked — 83 Rust unit tests plus all integration tests passed
  • cargo build --locked

Follow-up

PR2 will stack the runtime-reliability work on this branch: versioned/locked/atomic trace persistence, bounded subprocess execution, reliable worker completion, and explicit CLI argument handling.

@xr843
xr843 marked this pull request as ready for review July 30, 2026 04:23
@xr843
xr843 merged commit f0d8677 into main Jul 30, 2026
13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant