feat: mapping provenance — expose which §9.10.2 tier produced span text (all bindings) - #893
Merged
Conversation
… came from Introduces `MappingProvenance`, a fact recording which tier of the ISO 32000-1 §9.10.2 character-to-Unicode cascade produced a character's value (ActualText, ToUnicode, EncodingName, PredefinedCMap, EmbeddedCmap) — or `Fallback` when no tier mapped it and the value was chosen by fallback (CID-as-Unicode echo or U+FFFD), which §9.10.2 leaves to reader discretion. This is the API contract for exposing extraction provenance as a *fact* callers compose policy from (OCR routing, low-confidence flagging, "undecodable" page detection), rather than the library shipping a judgment/flag per use case. It mirrors the spec's own cascade, so it is complete across all font types. Variants are ordered most- to least-authoritative; `rank`/`weaker` summarise a multi-character span by its weakest member. This commit lands only the type and its unit tests; capturing it in the decode path and threading it onto spans are follow-ups (the capture point is `char_to_unicode_uncached`, which needs a behaviour-preserving refactor plus the corpus sweep before merge).
… capability Returns the most authoritative Unicode-mapping resource a font offers as a MappingProvenance fact, derived from font structure (ToUnicode / predefined collection / embedded cmap / simple-font encoding), else Fallback. Covers all font types (not just TrueType), and Fallback is exactly the fully-severed undecodable-subset case. No decode-path changes, so extraction output is unaffected. 4 unit tests via make_font.
Adds a `provenance: Option<MappingProvenance>` field to `TextSpan`, populated on the main extraction path from each span's resolved font (`FontInfo::best_mapping_provenance`). A span whose font carries no §9.10.2 mapping resource reports `Fallback` — the signal that its text is a fabricated glyph-index echo rather than read from the file. The field is `#[serde(skip)]` runtime metadata, so span JSON output is byte-identical and existing fixtures are unaffected. Every `TextSpan` literal across the crate gains `provenance: None` (the default for synthetic spans); only the extractor sets a real value. Rust-side feature is complete; binding accessors (C-ABI + per-language) are the remaining rollout. 7 provenance tests + 729 extraction/layout tests green.
Adds MappingProvenance::as_str() (shared stable labels for all bindings) and a `provenance` getter on the Python TextSpan, returning "to_unicode" / "encoding" / "predefined_cmap" / "embedded_cmap" / "actual_text" / "fallback", or None when the font is unresolvable. Compiles under --features python.
Reach every language binding with the mapping-provenance fact: - Serialize TextSpan.provenance as "provenance": "<label>" (skip when absent), so all serde/JSON-based bindings (WASM, and the JSON span surfaces) expose it automatically. Spans without a resolved font omit the field, so existing serialized output is unchanged — the full lib suite (5745 tests) stays green. - MappingProvenance now serializes with stable lowercase labels matching as_str() (to_unicode / encoding / predefined_cmap / embedded_cmap / actual_text / fallback), so serde and explicit accessors agree. - Add C-ABI `pdf_oxide_element_get_provenance(elements, index, err)` returning the label (or null when unknown), reaching the typed native bindings (C#, Java, Go, Swift, Kotlin, C++, Zig, ...). Python already has a getter. The Rust core, serde, and C-ABI surfaces now all carry provenance; per-language wrapper methods over the C-ABI call are thin follow-ups.
Expose mapping provenance to the native-binding languages: - ffi: `pdf_oxide_element_get_provenance` accessor; add `provenance` to the `elements_to_json` payload (skip when absent) so JSON-consuming bindings get it automatically. - C headers (pdf_oxide_c, php): declare the accessor. - C#: native import. Go: extern + `Element.Provenance` json field. C++: `Element.provenance` + populate from the accessor. Python: getter (prior). Dynamically-typed bindings (Ruby/PHP/Julia/R/Elixir/Clojure) surface the field automatically via the JSON payload. Remaining statically-typed struct fields (Swift/Dart/Zig/Kotlin) are mechanical mirrors.
Adds the §9.10.2 mapping-provenance label to the Swift Element struct, the Dart Element class, and a Zig ElementList.getProvenance() accessor, each reading the pdf_oxide_element_get_provenance C-ABI function.
Regenerate include/pdf_oxide_c/pdf_oxide.h (cbindgen) and php/include/pdf_oxide.h (preprocess_header.py) from src/ffi.rs so the provenance accessor is generated, not hand-written.
Mirrors the existing element_get_text ObjC wrapper for the §9.10.2 mapping-provenance C-ABI accessor.
A TextSpan serializes "provenance":"<label>" when known and omits it when
absent. This is the surface every JSON binding reads — WASM, Go, Ruby, Julia,
R, Elixir, Clojure, and Java/Kotlin via extractStructured (StructuredPage ->
StructuredRegion { spans: Vec<TextSpan> }) — so provenance reaches those
languages with no per-binding code.
Contributor
|
thank you for this! |
Prevent generated .NET bin/obj, dist-browser bundles, node_modules, and .gradle under viewer/ from being committed (root cause of an accidental bulk-add).
yfedoseev
force-pushed
the
feat/mapping-provenance
branch
from
July 18, 2026 05:08
f4538fe to
6559f16
Compare
…er non-default features/tests
The provenance field added to TextSpan is not defaulted in struct literals,
so every explicit `TextSpan { .. }` must name it. Six literals compiled only
under feature/test configurations the authoring build did not exercise
(--features python/wasm), so they were missed and broke CI's full-workspace
Clippy + Test:
error[E0063]: missing field `provenance` in initializer of `TextSpan`
--> tests/test_spacing_integration.rs
Sites fixed: src/ocr/engine.rs (ocr feature), tests/test_pipeline_html_converter.rs,
tests/test_spacing_integration.rs, tests/test_spacing_spec_compliant.rs.
Claude-Session: https://claude.ai/code/session_01VfT1dvLWaMNh4SpaXV8kcp
Signed-off-by: Yury Fedoseev <yfedoseev@gmail.com>
…> as HTML The corpus_sig module doc listed the output format and commands with angle-bracket placeholders (<relpath>, <corpus_dir>, ...) indented only 2 spaces, so rustdoc parsed them as HTML tags and failed under -D rustdoc::invalid-html-tags (implied by -D warnings in the docs job): error: unclosed HTML tag `relpath` --> src/bin/corpus_sig.rs:4 Wrapping the two blocks in ```text code fences makes rustdoc treat them literally. Claude-Session: https://claude.ai/code/session_01VfT1dvLWaMNh4SpaXV8kcp Signed-off-by: Yury Fedoseev <yfedoseev@gmail.com>
yfedoseev
added a commit
that referenced
this pull request
Jul 18, 2026
#893 added the provenance element accessor to the Dart binding but the new lookupFunction line was not dart-formatted, so Dart Bindings CI on main fails the Format check (dart format --set-exit-if-changed, exit 1). Ran dart format; the only change is wrapping the elementGetProvenance lookupFunction call. Verified: dart format --output=none --set-exit-if-changed lib test example (0 changed, exit 0). Claude-Session: https://claude.ai/code/session_01VfT1dvLWaMNh4SpaXV8kcp Signed-off-by: Yury Fedoseev <yfedoseev@gmail.com>
yfedoseev
added a commit
that referenced
this pull request
Jul 18, 2026
…nbreak main C++ & Dart CI) (#897) * fix(cpp): clang-format the provenance accessor block #893 added the provenance element accessor to the C++ header but the new lines were not clang-formatted, so C++ Bindings CI on main fails the blocking clang-format check (-Wclang-format-violations, exit 123): cpp/include/pdf_oxide/pdf_oxide.hpp:908: error: code should be clang-formatted Ran clang-format --style=file:cpp/.clang-format; the only changes are `char *p` -> `char* p` and wrapping the take_string call. Verified with clang-format --dry-run --Werror (exit 0). Claude-Session: https://claude.ai/code/session_01VfT1dvLWaMNh4SpaXV8kcp Signed-off-by: Yury Fedoseev <yfedoseev@gmail.com> * fix(dart): dart format the provenance accessor #893 added the provenance element accessor to the Dart binding but the new lookupFunction line was not dart-formatted, so Dart Bindings CI on main fails the Format check (dart format --set-exit-if-changed, exit 1). Ran dart format; the only change is wrapping the elementGetProvenance lookupFunction call. Verified: dart format --output=none --set-exit-if-changed lib test example (0 changed, exit 0). Claude-Session: https://claude.ai/code/session_01VfT1dvLWaMNh4SpaXV8kcp Signed-off-by: Yury Fedoseev <yfedoseev@gmail.com> --------- Signed-off-by: Yury Fedoseev <yfedoseev@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds mapping provenance as a first-class fact: which ISO 32000-1 §9.10.2 character-to-Unicode tier produced a span's text — or
Fallback, meaning the font carried no mapping resource and the text is a fabricated glyph-index echo, not read from the file (§9.10.2: "there is no way to determine what the character code represents … a conforming reader may choose a character code of their choosing").This is the library-correct, spec-aligned answer to "how do I know extracted text is real?": the library exposes the fact, callers compose the policy (route to OCR, flag a page, keep the raw echo). It replaces the need for per-use-case judgment flags (e.g. an "undecodable text layer" warning), and unlike a font-type-specific structural check it is complete across all font types.
What's included
MappingProvenance(spec-ordered:ActualText › ToUnicode › EncodingName › PredefinedCMap › EmbeddedCmap › Fallback) withrank/weaker/is_from_file/as_str. Serializes as stable lowercase labels.FontInfo::best_mapping_provenance()— the most authoritative resource a font offers, derived from structure, all font types.Fallback= the fully-severed undecodable-subset case.TextSpan.provenance: Option<MappingProvenance>, populated on the extraction path from each span's resolved font."provenance": "<label>"(omitted when absent) → every JSON/serde binding (WASM, JSON surfaces) exposes it automatically.span.provenancegetter.pdf_oxide_element_get_provenance(elements, index, err)→ reaches the typed native bindings (C#, Java, Go, Swift, Kotlin, C++, Zig, …).Verification
extraction_api_regression(52) +test_structured_extractiongreen.--features pythonand--features wasm(wasm lib).Design note
FontInfo::best_mapping_provenance()is intentionally a fact, not a judgment — it says "which mapping tier exists", never "this text is corrupt" (a scanned page and an intentional glyph-index payload are indistinguishable, so the library must not assert the latter). This is the primitive that lets a caller build "undecodable page" detection themselves, for all font types.Remaining (mechanical, per-language CI)
Idiomatic wrapper accessors over the C-ABI call in the typed native binding sources (a
Provenanceproperty in C#/Java/Swift/etc.). The FFI + serde surfaces make the fact available in every language today; these are thin last-mile additions best built and tested in each language's own toolchain.Closes #876 (as the fact-based alternative to the diagnostic-warning approach in #892).