The performance pair for 2.2.0, and what it caught (#746) - #939
Merged
Conversation
#746 makes a measured performance column a standing condition of a release rather than a roadmap item, and 2.0.0 shipped without one. This is the pair for 2.2.0: the 1709th, tagged v2.1.0, against the 1724th, measured minutes apart in one session on one machine so that the two may be read against each other. Everything that is not the solver is flat -- parse, simplify, evaluate and the compiled-call trio within 2%, allocation byte-identical on most rows. Four Solve rows are 7% to 21% slower and allocate 10% to 20% more. **Allocation moving with the timing is what makes it real.** The previous pair is the cautionary case: a row reported +8.7% with allocation flat and re-measuring put it at +2.1%. Here four related rows move together in both, which noise does not do. Isolated to #918, the polynomial layer, by measuring the commits either side of it: allocation steps exactly once, at that commit, and is identical to v2.1.0 before it and to master after it, in all four rows. SolveEasy -- a quadratic, which never reaches the factorisation path -- is flat throughout, which is the mechanism corroborating itself. **It is a price rather than a regression.** #918 made the equation solver the polynomial layer's first consumer, which is what turned x^5 + 2x^3 - 2x^2 - 4 from three roots, one of them a float, into all five, exact. An incomplete solution set is a false answer, not a partial one, so the trade is the one AGENTS.md's first rule requires. Recorded, not fixed. And the part worth more than the rows: **none of these ten solver benchmarks benefit from #918.** They are quadratics, a substituted quadratic and a trigonometric substitution; not one factors into lower-degree pieces, so every one pays the search and none collects the answer. The column shows the change as pure cost, which is true of these inputs and false of the change. A benchmark whose polynomial does factor is owed before the next column. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#746 makes a measured performance column a standing condition of a release rather than a roadmap item, and 2.0.0 shipped without one. This is the pair for 2.2.0.
The 1709th, tagged
v2.1.0, against the 1724th — measured minutes apart in one session on one machine, so the two may be read against each other.Four solver rows moved, and the allocation moved with them
Everything else is flat —
ParseEasy+0.0%,SimplifyEasy−0.5%,EvalTrig−0.8%, theRun*trio within 2%, allocation byte-identical on most rows.Allocation moving with the timing is what makes it real. The previous pair is the cautionary case: a row reported +8.7% with allocation flat, and re-measuring put it at +2.1%. Here four related rows move together in both, which noise does not do.
It is #918, the polynomial layer, and it is a price rather than a regression
Isolated by measuring the commits either side of it. Allocation steps exactly once:
v2.1.051194ce8, before #91869f66da7, after #918The same clean step in all four rows.
SolveEasy— a quadratic, which never reaches the factorisation path — is flat throughout, which is the mechanism corroborating itself.#918 made the equation solver the polynomial layer's first consumer, which is what turned
x^5 + 2x^3 - 2x^2 - 4from three roots, one of them a float, into all five, exact. An incomplete solution set is a false answer, not a partial one, so the trade is the one AGENTS.md's first rule requires. Recorded, not fixed.What the suite cannot see, which is worth more than the rows it can
None of these ten solver benchmarks benefit from #918. They are quadratics, a substituted quadratic and a trigonometric substitution; not one factors into lower-degree pieces, so every one pays the search and none collects the answer. The column shows the change as pure cost — true of these inputs, false of the change.
A benchmark whose polynomial does factor is owed before the next column, or this row will keep reporting a correctness fix as a slowdown for as long as anyone reads it.
Documentation only — no code changes.
🤖 Generated with Claude Code