Skip to content

Add target-aware differential coverage summaries#24

Open
decofe wants to merge 3 commits into
mainfrom
centaur/add-matrix-rounds-1781886587
Open

Add target-aware differential coverage summaries#24
decofe wants to merge 3 commits into
mainfrom
centaur/add-matrix-rounds-1781886587

Conversation

@decofe

@decofe decofe commented Jun 19, 2026

Copy link
Copy Markdown
Member

Summary

  • tag showmap trials with target labels before differential coverage analysis and emit by_target/<name> rows
  • extend differential_coverage_summary.csv with monitoring statistics, effect size, CI, decision, and budget columns while preserving existing columns
  • add differential_coverage_sequential_state.json with a machine-readable scheduling directive for continue/eliminate/inconclusive decisions
  • disable automated winner decisions until the analysis can prove independent seed/round campaign units and use true anytime-valid inference
  • scope Holm correction to the top-level per-target family; combined/by-test rows keep their local p-values
  • remove fake adaptive savings, fake next seed directives, and the legacy point-estimate winner backdoor

Tests

  • uv run --with-requirements analysis/requirements.txt python -m unittest analysis.tests.test_differential_coverage
  • git diff --check

Prompted by: @0xalpharush

@decofe decofe force-pushed the centaur/add-matrix-rounds-1781886587 branch from 8231abb to 862051d Compare June 19, 2026 16:36
@decofe decofe changed the title Add matrix differential summary aggregation Add target-aware differential coverage summaries Jun 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants