Skip to content

fix(recovery): rebuild a Catalog from surviving pages when a truncated file lost its own - #890

Merged
yfedoseev merged 2 commits into
yfedoseev:mainfrom
ajbufort:fix/reconstruct-catalog-from-orphan-pages
Jul 20, 2026
Merged

fix(recovery): rebuild a Catalog from surviving pages when a truncated file lost its own#890
yfedoseev merged 2 commits into
yfedoseev:mainfrom
ajbufort:fix/reconstruct-catalog-from-orphan-pages

Conversation

@ajbufort

Copy link
Copy Markdown
Contributor

The problem

reconstruct_xref already salvages a damaged file by scanning its surviving N G obj markers and hunting for a /Type /Catalog to anchor the page tree. When the Catalog itself did not survive it gives up with "Could not find catalog" and the whole document is a total loss - even though its page objects are sitting right there in the surviving bytes.

This is the shape of a file truncated at the end: a web crawl capped mid-stream, an incremental update whose tail was lost. The xref, trailer and Catalog all live at the end of a PDF, so a tail cut destroys them while the bulk of the objects - including /Type /Page content - survives.

The fix

When no Catalog is found, rebuild one instead of failing, in order of fidelity:

  1. A /Type /Pages node survived -> synthesize << /Type /Catalog /Pages R >> pointing at it (preferring a genuine root - a /Pages with no /Parent), preserving the file's own tree and inherited attributes.
  2. Only orphaned /Type /Page objects -> hang them off a synthesized flat /Pages (object-number order; a default /MediaBox as an inheritance fallback for a page that relied on its lost parent).
  3. Pages packed inside a /Type /ObjStm (PDF 1.5+ routinely compress the Catalog and page dicts, which the offset scan cannot see) -> decompress the surviving object streams and recover the pages packed inside, anchoring a real recovered Catalog or synthesizing one over them.

Error only when neither a page nor a /Pages survived anywhere - there is genuinely nothing to show.

How the synthesized objects reach the document

A synthesized (or ObjStm-recovered) object has no byte offset, so it cannot go in the cross-reference table. reconstruct_xref now returns a third tuple element - the objects to inject - and PdfDocument::open seeds them into the object cache, which load_object checks before the xref. The page /Contents streams live uncompressed and are already in the rebuilt xref, so they resolve normally.

This is purely additive: an ordinary reconstruction returns an empty synthetic vector and the open path is byte-for-byte unchanged.

Measured

96 real-world truncated files (all exactly 5 MiB, Catalog/xref/trailer gone, every one a total loss before this change):

before after
open 0 74
real extractable text 0 47
pages recovered 0 4822

The 22 that still fail were truncated past any surviving page. No per-file hang.

Tests

Three truncated-PDF fixtures, each asserting the pre-fix InvalidXref is gone and the page text extracts:

  • recovers_when_pages_root_survives - a /Type /Pages root survives the cut;
  • recovers_orphan_pages_with_no_pages_root - fully orphaned pages, flat /Pages synthesized;
  • recovers_pages_packed_inside_an_object_stream - pages inside a /Type /ObjStm (the fixture uses an unfiltered ObjStm, so no Flate dependency).

Revert-checked: dropping the ObjStm arm fails only the ObjStm fixture; dropping the whole synthesis fails all three. Gate on this branch: cargo test (5738 lib + 3 new + 139 doctests), cargo clippy --all-targets -- -D warnings, cargo fmt --check, cargo doc - all clean. DCO signed.

…ed file lost its own

xref reconstruction already scans a damaged file's `N G obj` markers and hunts for
a `/Type /Catalog` to anchor the page tree. When the Catalog itself did not
survive - a web crawl capped mid-stream, an incremental update that lost its tail -
it gives up ("Could not find catalog") and the whole document is a total loss,
even though its page objects are sitting right there in the surviving bytes.

Rebuild a Catalog instead of giving up, in order of fidelity:
  1. a `/Type /Pages` node survived  -> synthesize a Catalog pointing at it
     (prefer a genuine root - a /Pages with no /Parent), preserving the file's
     own tree and inherited attributes;
  2. only orphaned `/Type /Page` objects -> hang them off a synthesized flat
     /Pages (object-number order, a default /MediaBox as an inheritance fallback);
  3. pages packed inside an object stream (PDF 1.5+ routinely put the Catalog and
     page dicts in a /Type /ObjStm the offset scan cannot see) -> decompress the
     surviving ObjStms, inject their objects at their real numbers so refs resolve
     (the /Contents streams live uncompressed and are already in the rebuilt xref),
     and anchor a real recovered Catalog, or synthesize one over the packed pages.

The synthesized/recovered objects have no byte offset, so they are threaded out of
reconstruction and seeded into the object cache, which `load_object` checks before
the xref. Error only when NEITHER a page nor a /Pages survived anywhere - there is
genuinely nothing to show.

MEASURED on 96 real corpus files, all truncated at a 5 MiB crawl cap with the
Catalog/xref/trailer gone (every one a total loss before): 74 now OPEN, 47 with
real extractable text, 4822 pages recovered, 0 per-file hangs. The remaining 22
were truncated past any surviving page. Empty-recovery paths (the vast majority of
files, which parse normally) are untouched: the synthetic vector is empty and the
open path is byte-for-byte unchanged.

Tests: three truncated-PDF fixtures - Pages-root survives, fully-orphaned pages,
and pages-inside-an-ObjStm - each asserting the pre-fix InvalidXref is gone and the
page text extracts. Revert-checked: dropping the ObjStm arm fails only the ObjStm
fixture; dropping the whole synthesis fails all three.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Anthony J. (Tony) Bufort <ajbufort@ajbconsulting.us>
@ajbufort
ajbufort requested a review from yfedoseev as a code owner July 17, 2026 23:35
@ajbufort
ajbufort marked this pull request as draft July 19, 2026 18:10

@yfedoseev yfedoseev left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved — reviewed and regression-passed; in the merge queue. (Formal approval to supersede the earlier review state and record sign-off.)

@ajbufort
ajbufort marked this pull request as ready for review July 20, 2026 08:05
@yfedoseev
yfedoseev merged commit cfd0324 into yfedoseev:main Jul 20, 2026
232 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants