feat(fonts): flag provably undecodable text layers with a structured warning

[tobocop2]

What this fixes

Implements the diagnostic agreed in #876 (fixes #876). A common subsetting pattern (Type0/Identity-H, Identity ordering, CIDToGIDMap /Identity, no ToUnicode, embedded font stripped of cmap and post) leaves a page with no glyph-to-Unicode mapping at all. Extraction can only echo raw glyph indices, and callers had no way to tell that mojibake from a clean extraction.

The diagnostic

FontInfo::has_undecodable_text_layer() detects exactly that severed shape. When such a font is used on a page, extraction emits a new undecodable_text_layer structured warning — page-scoped, once per page/font, with the font name and the §9.10.2 reference.

Text output is unchanged. The raw CID sequence stays available to callers who want it; the warning only adds the provable fact that no Unicode mapping exists in the file.

Staying silent on recoverable fonts

The predicate is structural and conservative. It proves cmap absence from the embedded program's own table directory rather than trusting the decoder's cached view, so a program the lazy accessor never inspected (e.g. a CIDFontType2 delivered as an OpenType FontFile3) is judged by what its tables actually say. Fonts with a live embedded cmap (the byte-as-GID cases of #99/#760), usable post glyph names, a usable ToUnicode, a non-Identity ordering, an explicit CIDToGIDMap stream, or an unparseable program are never flagged — when severedness can't be proven, it says nothing.

Emission and page attribution

Every page-parameterized extraction API surfaces the warning — extract_text and the conversion APIs, extract_chars, and extract_spans_with_config — each attributing it to the page it was asked about, via a diagnostic page context kept deliberately separate from the MCID scope stack (whose values are public span data and stay untouched). Auxiliary extractors with no named page (annotation appearance streams, font enumeration) stay silent rather than misreport a default page. WarningSink::push_once keeps the same page/font fact to a single entry across repeated passes and across APIs, and the per-page check is memoized by font identity, so the predicate runs at most once per font per page and adds no measurable work to the Tf hot path.

Bindings

No binding changes needed: structured_warnings() serializes the category via its stable string ("undecodable_text_layer"), so Python and the other consumers of that surface pick the new category up automatically. The new Rust-side helpers (push_structured_warning_once, WarningSink::push_once) match the existing Rust-only push surface.

Test plan

• 16 new tests: the predicate positive plus negative controls for every recoverable variant (embedded cmap, cmap-without-post reachable only via the table directory, post-only, ToUnicode, explicit CIDToGIDMap stream, no/empty/unparseable program, non-Identity ordering, simple fonts, absent /CIDToGIDMap key, present-but-empty ToUnicode); integration tests asserting text output stays byte-identical while the warning fires with correct fields; dedup across repeated extraction; a two-page attribution regression test proving extract_chars, extract_spans_with_config, and extract_text all report the true page and collapse to one warning. Fixtures are three fully synthetic blank-glyph fonts (604 B–1.2 KB) built with fontTools; the PDFs are built in-test.
• Reproducer parity: the undecodable.pdf from [Feature]: Tell callers when a page's text cannot be read, instead of returning garbled text that looks correct #876 extracts byte-identically to released 0.3.74 (81,7('67$7(6',675,&7&2857) and raises the warning on page 0.
• No-regression sweep: text + HTML extraction over 577 pinned public PDFs (arXiv, PMC OA, govdocs1, pdf.js corpus, IRS, multilingual/RTL), main vs this branch — 577/577 byte-identical.
• Precision probe over the same corpus: 7 documents flagged, all true positives with provably severed subsets — cmap/post-stripped Wingdings/ZapfDingbats (poppler's pdffonts confirms no Unicode map) and one arXiv paper whose embedded ArialMT subsets all lack cmap and post (verified table-by-table with fontTools; pdftotext emits the same mojibake there). Zero false positives, and page attribution is exact.
• make check green (fmt, clippy -D warnings incl. --features python, full test suite); clippy also clean on default features.

[yfedoseev]

@tobocop2 first: thank you for this — the diagnosis in #876 and the engineering here are genuinely excellent (the non-destructive design, the byte-identical differential sweep, the careful "only the fully-severed case" predicate). And I owe you an honest note: my earlier comment on #876 said "a PR would be very welcome" and pre-endorsed the diagnostic direction. That was a premature maintainer commitment on my part, made before we'd settled the scope — the bind that creates here is on us, not you.

After sitting with it, we've decided to solve the underlying problem a different way, and I want to lay out the difference plainly rather than just close this.

The fork: a judgment vs. a fact. This PR ships a judgment — has_undecodable_text_layer() and an undecodable_text_layer warning that assert "this text can't be read." We landed on exposing the fact instead: which ISO 32000-1 §9.10.2 mapping tier produced each span's text — ToUnicode / EncodingName / PredefinedCMap / EmbeddedCmap / ActualText, or Fallback when no tier mapped it and the value is a fabricated glyph-index echo. That work is in #893 (MappingProvenance).

Why the fact rather than the judgment:

1. Spec. §9.10.2 leaves the failed-mapping case to reader discretion ("there is no way to determine what the character code represents … a conforming reader may choose a character code of their choosing"). So "undecodable" isn't a conformance gap the library must assert — but which tier fired is exactly the spec's own cascade, so reporting it is maximally spec-aligned.

2. Completeness. The predicate here is gated on TrueType (is_truetype_font || cid_font_type == "CIDFontType2"), so a CIDFontType0 / CFF font with the identical severed condition returns "not undecodable" — a false all-clear. Provenance is captured from the mapping cascade, so it's Fallback for every font type, not just TrueType.

3. It doesn't overclaim. A severed-subset scan and an intentional glyph-index payload are structurally identical — the library can't know which. "Undecodable" asserts corruption we can't prove; "no mapping tier produced this" is just true.

4. No new API surface. Provenance rides the surfaces spans already have — serde/JSON, the C-ABI element accessor, the Python getter — rather than a new warnings subsystem. It's one fact that subsumes undecodable-detection, garbled-flagging, OCR-routing, and confidence, for every caller.

Your use case is fully served, and arguably better: instead of a boolean flag, your pipeline routes on span.provenance == Fallback (or the "fallback" label in the JSON/structured output) — which now also catches the CFF-severed pages #892 would miss. Same OCR-routing outcome, complete coverage, and you keep the raw echo for the payload case.

So I'm going to decline this PR in favor of #893, with real credit to you: #893 exists because your #876 nailed the condition and the trap (the offset-recovery dead end). I've credited the diagnosis in the CHANGELOG/PR. Thank you for pushing on this — the analysis is what made the right design obvious, and the premature-invite misstep was mine to own.

[tobocop2]

sounds good to me. thanks!