Fix Pareto frontier strict-domination semantics - #122
Conversation
The Efficiency Frontier walk (both the plot in create_leaderboard_display and the trophy marker in get_pareto_df) sorted entries by ascending cost / descending score, then kept any point whose score cleared the running max with '>='. Because points are visited cheapest-first, a later point that only *ties* the running-max score has the same score at a higher cost and is strictly Pareto-dominated -- yet '>=' admitted it to the frontier and awarded it a trophy. Concretely, on DS-1000 two RoboPhD entries score an identical 0.8533 (900/900 same problems) at $0.0368 and $0.0519; both landed on the frontier and both got a trophy, though the $0.0519 entry is dominated by the $0.0368 one. Switch both comparisons to a strict '>' so only the cheapest entry at each score level stays on the frontier. Reported via S2 on-call. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Strict '>' fixed the reported over-inclusion (a same-score, higher-cost entry wrongly kept a trophy) but under-included the opposite edge case: two entries equal on BOTH cost and score are mutually non-dominated, yet sweeping row-by-row with score > running-max kept only the first and dropped the other. Compare each cost group against the best score reachable at a strictly lower cost instead: rows tied at their group's max score all stay on the frontier; rows below the group max (same cost, lower score) are dominated and dropped. Applies to both the plot frontier line and get_pareto_df (the trophy marker). Adds regression tests for the equal-on-both-axes and same-cost-lower-score cases. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Refined the fix after a good edge-case catch in review: strict Both frontier sites (plot line +
Behavior on the three ordered cases:
Added regression tests for the equal-on-both-axes and same-cost-lower-score cases; the existing dominated-tie tests still pass ( |
Problem
On the leaderboard, entries that are strictly Pareto-dominated can still land on the Efficiency Frontier and receive a 🏆 trophy. Reported for DS-1000 (Code & Execution): two RoboPhD submissions have the same full-precision score,
0.8533, at costs$0.0368and$0.0519. The$0.0519entry is dominated by the$0.0368entry, yet both sit on the frontier and both get a trophy.Root causes and fix
Both frontier computations in
leaderboard_transformer.pywalked rows sorted by cost ascending / score descending and used>=against the running maximum:_plot_scatter_plotly(plot frontier trace)get_pareto_df(table trophy marker)A later point that only ties the running maximum has the same score at higher cost and is strictly dominated. A simple
>replacement is not sufficient, however: it would incorrectly discard the second of two entries equal on both cost and score, even though those entries are mutually non-dominated.The corrected algorithm groups rows by full-precision cost and compares each group with the best score reachable at a strictly lower cost. It:
Metric rounding is presentation-only.
transform_raw_dataframepreserves full-precision scores and costs for Pareto calculation; this PR intentionally leaves display formatting unchanged.Verification
5 passed).HF_CONFIG=1.0.0), then a headless Chrome session navigated the rendered Gradio UI to Code & Execution → DS-1000.Before (pre-fix UI)
After (this PR)
The rendered result is exact: the dominated
0.853 @ $0.05RoboPhD row loses its trophy, while0.853 @ $0.04and the legitimately higher-scoring0.862 @ $0.13rows retain theirs.Follow-up PR #124 subsequently added three-decimal cost display, full-precision score and cost values on hover, and confidence-interval UX without changing Pareto membership.
Suggested-by: @jbragg