Skip to content

Fix Pareto frontier strict-domination semantics - #122

Merged
AmberRose2 merged 5 commits into
mainfrom
fix/pareto-frontier-strict-domination
Aug 5, 2026
Merged

AmberRose2 merged 5 commits into
mainfrom
fix/pareto-frontier-strict-domination

Conversation

@gas2own

@gas2own gas2own commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Problem

On the leaderboard, entries that are strictly Pareto-dominated can still land on the Efficiency Frontier and receive a 🏆 trophy. Reported for DS-1000 (Code & Execution): two RoboPhD submissions have the same full-precision score, 0.8533, at costs $0.0368 and $0.0519. The $0.0519 entry is dominated by the $0.0368 entry, yet both sit on the frontier and both get a trophy.

Root causes and fix

Both frontier computations in leaderboard_transformer.py walked rows sorted by cost ascending / score descending and used >= against the running maximum:

  • _plot_scatter_plotly (plot frontier trace)
  • get_pareto_df (table trophy marker)

A later point that only ties the running maximum has the same score at higher cost and is strictly dominated. A simple > replacement is not sufficient, however: it would incorrectly discard the second of two entries equal on both cost and score, even though those entries are mutually non-dominated.

The corrected algorithm groups rows by full-precision cost and compares each group with the best score reachable at a strictly lower cost. It:

  • keeps all entries tied at the maximum score for a cost group when that score beats all cheaper entries;
  • drops lower-scoring entries at the same cost; and
  • drops equal-scoring entries at a higher cost.

Metric rounding is presentation-only. transform_raw_dataframe preserves full-precision scores and costs for Pareto calculation; this PR intentionally leaves display formatting unchanged.

Verification

  • Regression tests cover both affected paths, equal-score/higher-cost domination, equal-cost/equal-score co-optima, same-cost/lower-score domination, and values that display identically at three decimals but differ at full precision (5 passed).
  • The patched app was launched locally with the real public results (HF_CONFIG=1.0.0), then a headless Chrome session navigated the rendered Gradio UI to Code & Execution → DS-1000.

Before (pre-fix UI)

Before fix: both the dominated $0.05 row and the cheaper $0.04 row have trophies

After (this PR)

After fix: the dominated $0.05 row has no trophy, while the cheaper $0.04 row retains it

The rendered result is exact: the dominated 0.853 @ $0.05 RoboPhD row loses its trophy, while 0.853 @ $0.04 and the legitimately higher-scoring 0.862 @ $0.13 rows retain theirs.

Follow-up PR #124 subsequently added three-decimal cost display, full-precision score and cost values on hover, and confidence-interval UX without changing Pareto membership.

Suggested-by: @jbragg

The Efficiency Frontier walk (both the plot in create_leaderboard_display
and the trophy marker in get_pareto_df) sorted entries by ascending cost /
descending score, then kept any point whose score cleared the running max
with '>='. Because points are visited cheapest-first, a later point that only
*ties* the running-max score has the same score at a higher cost and is
strictly Pareto-dominated -- yet '>=' admitted it to the frontier and awarded
it a trophy.

Concretely, on DS-1000 two RoboPhD entries score an identical 0.8533 (900/900
same problems) at $0.0368 and $0.0519; both landed on the frontier and both
got a trophy, though the $0.0519 entry is dominated by the $0.0368 one.

Switch both comparisons to a strict '>' so only the cheapest entry at each
score level stays on the frontier. Reported via S2 on-call.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Strict '>' fixed the reported over-inclusion (a same-score, higher-cost
entry wrongly kept a trophy) but under-included the opposite edge case:
two entries equal on BOTH cost and score are mutually non-dominated, yet
sweeping row-by-row with score > running-max kept only the first and
dropped the other.

Compare each cost group against the best score reachable at a strictly
lower cost instead: rows tied at their group's max score all stay on the
frontier; rows below the group max (same cost, lower score) are dominated
and dropped. Applies to both the plot frontier line and get_pareto_df
(the trophy marker). Adds regression tests for the equal-on-both-axes and
same-cost-lower-score cases.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@gas2own

gas2own commented Aug 5, 2026

Copy link
Copy Markdown
Contributor Author

Refined the fix after a good edge-case catch in review: strict > correctly drops a dominated tie (same score, higher cost) but was wrong for the opposite case — two entries equal on both cost and score. Those are mutually non-dominated (neither is worse on any axis), so both belong on the frontier, yet a row-by-row score > running_max sweep kept only the first and dropped the co-optimum.

Both frontier sites (plot line + get_pareto_df) now group by cost and compare each group against the best score reachable at a strictly lower cost:

  • equal cost + equal score → all such rows kept (co-optimal);
  • equal cost + lower score → dropped (dominated by the same-cost higher-score row);
  • same score + higher cost → dropped (the originally-reported bug).

Behavior on the three ordered cases:

entries (cost @ score) >= (orig) > row-sweep grouped (this PR)
A 0.037@0.853, B 0.037@0.853 (equal on both) A,B A only A,B
C 0.052@0.853 (same score, higher cost) kept ✗ dropped ✓ dropped ✓
D 0.037@0.800 (same cost, lower score) dropped ✓ dropped ✓ dropped ✓

Added regression tests for the equal-on-both-axes and same-cost-lower-score cases; the existing dominated-tie tests still pass (pytest tests/integration/test_pareto_frontier.py → 4 passed).

@gas2own gas2own changed the title Fix Pareto frontier to exclude strictly-dominated tied entries Fix Pareto frontier strict-domination semantics Aug 5, 2026
@gas2own
gas2own requested a review from AmberRose2 August 5, 2026 21:26
@AmberRose2
AmberRose2 merged commit c0eba08 into main Aug 5, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants