fix(recovery): rebuild a Catalog from surviving pages when a truncated file lost its own - #890
Merged
yfedoseev merged 2 commits intoJul 20, 2026
Conversation
…ed file lost its own
xref reconstruction already scans a damaged file's `N G obj` markers and hunts for
a `/Type /Catalog` to anchor the page tree. When the Catalog itself did not
survive - a web crawl capped mid-stream, an incremental update that lost its tail -
it gives up ("Could not find catalog") and the whole document is a total loss,
even though its page objects are sitting right there in the surviving bytes.
Rebuild a Catalog instead of giving up, in order of fidelity:
1. a `/Type /Pages` node survived -> synthesize a Catalog pointing at it
(prefer a genuine root - a /Pages with no /Parent), preserving the file's
own tree and inherited attributes;
2. only orphaned `/Type /Page` objects -> hang them off a synthesized flat
/Pages (object-number order, a default /MediaBox as an inheritance fallback);
3. pages packed inside an object stream (PDF 1.5+ routinely put the Catalog and
page dicts in a /Type /ObjStm the offset scan cannot see) -> decompress the
surviving ObjStms, inject their objects at their real numbers so refs resolve
(the /Contents streams live uncompressed and are already in the rebuilt xref),
and anchor a real recovered Catalog, or synthesize one over the packed pages.
The synthesized/recovered objects have no byte offset, so they are threaded out of
reconstruction and seeded into the object cache, which `load_object` checks before
the xref. Error only when NEITHER a page nor a /Pages survived anywhere - there is
genuinely nothing to show.
MEASURED on 96 real corpus files, all truncated at a 5 MiB crawl cap with the
Catalog/xref/trailer gone (every one a total loss before): 74 now OPEN, 47 with
real extractable text, 4822 pages recovered, 0 per-file hangs. The remaining 22
were truncated past any surviving page. Empty-recovery paths (the vast majority of
files, which parse normally) are untouched: the synthetic vector is empty and the
open path is byte-for-byte unchanged.
Tests: three truncated-PDF fixtures - Pages-root survives, fully-orphaned pages,
and pages-inside-an-ObjStm - each asserting the pre-fix InvalidXref is gone and the
page text extracts. Revert-checked: dropping the ObjStm arm fails only the ObjStm
fixture; dropping the whole synthesis fails all three.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Anthony J. (Tony) Bufort <ajbufort@ajbconsulting.us>
ajbufort
marked this pull request as draft
July 19, 2026 18:10
yfedoseev
approved these changes
Jul 19, 2026
yfedoseev
left a comment
Owner
There was a problem hiding this comment.
Approved — reviewed and regression-passed; in the merge queue. (Formal approval to supersede the earlier review state and record sign-off.)
ajbufort
marked this pull request as ready for review
July 20, 2026 08:05
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
reconstruct_xrefalready salvages a damaged file by scanning its survivingN G objmarkers and hunting for a/Type /Catalogto anchor the page tree. When the Catalog itself did not survive it gives up with "Could not find catalog" and the whole document is a total loss - even though its page objects are sitting right there in the surviving bytes.This is the shape of a file truncated at the end: a web crawl capped mid-stream, an incremental update whose tail was lost. The xref, trailer and Catalog all live at the end of a PDF, so a tail cut destroys them while the bulk of the objects - including
/Type /Pagecontent - survives.The fix
When no Catalog is found, rebuild one instead of failing, in order of fidelity:
/Type /Pagesnode survived -> synthesize<< /Type /Catalog /Pages R >>pointing at it (preferring a genuine root - a/Pageswith no/Parent), preserving the file's own tree and inherited attributes./Type /Pageobjects -> hang them off a synthesized flat/Pages(object-number order; a default/MediaBoxas an inheritance fallback for a page that relied on its lost parent)./Type /ObjStm(PDF 1.5+ routinely compress the Catalog and page dicts, which the offset scan cannot see) -> decompress the surviving object streams and recover the pages packed inside, anchoring a real recovered Catalog or synthesizing one over them.Error only when neither a page nor a
/Pagessurvived anywhere - there is genuinely nothing to show.How the synthesized objects reach the document
A synthesized (or ObjStm-recovered) object has no byte offset, so it cannot go in the cross-reference table.
reconstruct_xrefnow returns a third tuple element - the objects to inject - andPdfDocument::openseeds them into the object cache, whichload_objectchecks before the xref. The page/Contentsstreams live uncompressed and are already in the rebuilt xref, so they resolve normally.This is purely additive: an ordinary reconstruction returns an empty synthetic vector and the open path is byte-for-byte unchanged.
Measured
96 real-world truncated files (all exactly 5 MiB, Catalog/xref/trailer gone, every one a total loss before this change):
The 22 that still fail were truncated past any surviving page. No per-file hang.
Tests
Three truncated-PDF fixtures, each asserting the pre-fix
InvalidXrefis gone and the page text extracts:recovers_when_pages_root_survives- a/Type /Pagesroot survives the cut;recovers_orphan_pages_with_no_pages_root- fully orphaned pages, flat/Pagessynthesized;recovers_pages_packed_inside_an_object_stream- pages inside a/Type /ObjStm(the fixture uses an unfiltered ObjStm, so no Flate dependency).Revert-checked: dropping the ObjStm arm fails only the ObjStm fixture; dropping the whole synthesis fails all three. Gate on this branch:
cargo test(5738 lib + 3 new + 139 doctests),cargo clippy --all-targets -- -D warnings,cargo fmt --check,cargo doc- all clean. DCO signed.