fix(fonts): deterministic subset choice in extract_embedded_fonts - #853
Conversation
yfedoseev
left a comment
There was a problem hiding this comment.
🔴 Changes requested — needs a determinism test
@ajbufort the fix logic is sound: replacing or_insert with a max over the total-order key (bytes.len(), bytes.as_slice()) genuinely removes the HashMap-iteration-order nondeterminism, and the final out.sort_by keeps output ordering stable. O(n), clean.
Blocker: the PR whose entire purpose is determinism ships zero tests (gh pr diff 853 → no #[test]; and I confirmed no sibling PR or branch of yours adds one). Please add a test that:
- builds a doc embedding two subsets of the same base font (
ABCDEF+Helvetica,GHIJKL+Helvetica) of different sizes, and - asserts the larger subset is returned regardless of insertion order, plus repeated extraction yields identical bytes.
Also (non-blocking): the "largest bytes = most glyph coverage" comment overstates it (byte size also grows with hinting/aux tables) — soften it, or tiebreak on glyph count first. And flag as a follow-up: the cp→gid / gid→width maps still accumulate across all subsets while the emitted program is now a single chosen subset (potential GID/program mismatch).
When several font subsets share a base name, get_font_set() yields them in HashMap order, so the or_insert kept a NONDETERMINISTIC subset - the bytes returned for a given PDF changed run to run. Both extract_embedded_fonts and extract_embedded_fonts_with_unicode_maps now keep the LARGEST subset (most glyph coverage), tie-broken by a byte comparison for a total-order-stable choice. The Unicode/width maps in the _with_unicode_maps variant still accumulate across all subsets, so coverage is never reduced. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> (cherry picked from commit 8831988) Signed-off-by: Anthony J. (Tony) Bufort <ajbufort@ajbconsulting.us>
Addresses review on yfedoseev#853: the fix removes HashMap-iteration nondeterminism but shipped no test, which for a determinism fix is the one thing that must not be taken on trust. Add a one-page PDF embedding two subsets of the same base font - ABCDEF+Helvetica and GHIJKL+Helvetica - with font programs of different sizes, and assert that: - the two subsets dedup to a single entry, prefix stripped - the LARGER program wins regardless of which resource slot holds it (the helper swaps them), so the choice follows the total order rather than encounter order - repeated extraction of one unchanged PDF is byte-identical Extraction is repeated 64 times because each call builds a fresh HashMap whose iteration order is independently seeded: one round would pass on the old code by luck. Verified the test EARNS its place - reverting either site to `or_insert` fails it at round 2, on both the plain and the with-unicode-maps variant (which carries its own copy of the choice, so it gets its own guard). Also soften the "largest = most glyph coverage" comment, which overstated the claim: byte size grows with hinting and auxiliary tables too, so a larger subset is not necessarily a superset. The guarantee is stability, not coverage. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Anthony J. (Tony) Bufort <ajbufort@ajbconsulting.us>
9502482 to
f300b0f
Compare
|
Thanks - the blocker was fair. A determinism fix that ships no test is asking to be taken on trust, and that is exactly the thing that should not be. 1. Determinism test (blocker) - added.
Extraction runs 64 rounds, because each call builds a fresh I checked the test actually earns its place rather than just going green: reverting either site to 2. "largest = most glyph coverage" - softened. You are right that it overstates it: byte count also grows with hinting and auxiliary tables, so a larger subset is not necessarily a superset of a smaller one. The comment now says what the total order actually buys - stability, not coverage - and stops claiming otherwise. I left the key as 3. The map/program mismatch - flagged in code, and it is worse than it looks. Noted in a comment at the site. Worth being precise about why it deserves its own PR: the maps do not merely risk a stale GID, they carry the very nondeterminism this PR removes from the program. When two subsets of one base font disagree about a codepoint's GID - which they routinely do, since each subset numbers its own glyphs - The fix is to bind the maps to the chosen subset instead of merging across all of them. That is a behaviour change - coverage shrinks where subsets are genuinely disjoint - so I have deliberately kept it out of this PR. Say the word and I will open it as a follow-up.
|
|
The determinism test the review asked for is in |
yfedoseev
left a comment
There was a problem hiding this comment.
Approved — reviewed and regression-passed; in the merge queue. (Formal approval to supersede the earlier review state and record sign-off.)
Description
When several font subsets share a base name,
get_font_set()yields them in HashMap order, so theor_insertinextract_embedded_fonts(andextract_embedded_fonts_with_unicode_maps) kept a nondeterministic subset - the bytes returned for a given PDF changed run to run. A consumer that re-runs extraction (or diffs/caches output) sees different font programs each time.Both functions now keep the largest subset (most glyph coverage), tie-broken by a byte comparison for a total-order-stable choice. The Unicode/width maps in the
_with_unicode_mapsvariant still accumulate across all subsets, so coverage is never reduced.Found while debugging why a downstream PDF->HTML converter emitted byte-different output for the same input on every run; the embedded
@font-facebytes were the source.Type of Change
Testing
cargo test --features renderinggreen on this branch (full suite). The change is localized to the two extraction functions; existing font tests pass unchanged.Checklist