Scanning Q/A Evaluations Criteria and datasets

[flooie]

Using this to detail the different criteria being used to validate the quality of the output.

[flooie]

Quality + cost analysis across prompts and models

Summary

I've spent a chunk of time building a ground-truth dataset and the infrastructure to measure extraction quality across prompt + model combinations. This issue documents the methodology and the open questions on which combination to ship.

Ground-truth dataset

628 hand-curated pages, split into two corpora by use case:

Corpus	Pages	Role
short/	48	Single-page cases — denial orders, per-curiam dispositions, short orders. Smoke tests; cheap to iterate against.
long/	584	Multi-page cases across 28 volumes. Primary regression signal — meaningful divergence here is a real defect, not noise.

The corpora are deliberately small and curated rather than scraped. Each page was hand-verified, so a divergence between extraction and ground truth is grounds for investigation, not "probably ground truth drift."

the corpus was originally generated when we planned two differnet prompts so there some first pages were changed because the masking is no longer part of the process and ending pages should have stamps.

Why not a single match score

Ground truth drift. They were generated by earlier prompt and model versions, so they carry OCR artifacts the current model corrects, older tag conventions, hyphenation choices that differ, and column-wrap paragraphs split differently. A byte-for-byte match score would mostly measure that drift, not real quality.

We also can't lean on text similarity alone — a model can produce all the right words and still be wrong (footnotes placed in <opinion>, missing case-end markers, hallucinated tag names, broken footnote/footnotemark linking). And it can produce different words while being right (joined hyphenations, corrected OCR, recovered parallel citations the ground truth missed).

The methodology has to separate "the model is wrong" from "the ground truth is stale" from "valid representation difference." That's why we measure many narrow things instead of one broad thing.

Methodology — multi-dimensional scoring

Each page produces a score dict with ~20 independent dimensions, each in [0, 1]. They roll up into four composites. Full reference in tests/tests.md; here's the shape:

Composites (roll-ups)

Name	What it averages	When to use it
composite_core	10 load-bearing dimensions (next table)	The headline — if it drops, investigate
composite_tags	text_order, no_hallucinated_tags, 7 tag_* count dimensions	Schema / convention drift signal
composite_text	4 text dimensions	Word-content quality only
composite	Grand mean of every dimension	Single-number summary; noisy

composite_core — load-bearing dimensions

Ten dimensions that reflect real quality regressions. A drop here means something is actually wrong with the extraction, not that the ground truth and actual disagree on a representation convention.

Dimension	What it measures	Drops when	Importance
text_completeness	Fraction of ground truth's unique body words present in actual (set-based intersection, position-independent)	Model dropped content	critical
text_no_hallucination	Character-trigram precision — what fraction of actual trigrams appear in ground truth, whitespace-tolerant	Model invented characters or words	critical
text_sim	Character-trigram Jaccard between ground truth and actual body strings	Words diverge substantially	medium
body_order	Fraction of consecutive <opinion> element pairs in correct (col, y) reading order (L→R, then ascending y). Auto-skipped on pages with <img type="scan"> or <img type="figure" size="full-width">	Model interleaves L and R columns	high
footnote_order	Two-layer check: (1) <footnote> siblings appear in the order their <footnotemark> references appear in body (handles multi-case label resets like [74, 75, 1]); (2) within each footnote, positioned chunks follow (col, y)	Footnotes out of sequence; column-wrap violation within a single footnote	high
fn_band_purity	Counts body blocks whose y sits at/below the first footnote's y. Compared to ground truth (penalize only when actual has more than ground truth)	Model put footnote-continuation text into <opinion>	high
case_endings	Count match for <!-- Case Ending --> markers: 1 - abs(g - a) / max(g, a, 1)	Model missed or invented a case boundary	critical — downstream uses this to split volumes into individual cases
footnote_labels	F1 over footnote label set, orphan-aware (orphan="true", <p continued="true">, or unlabeled all collapse to a sentinel)	Footnotes missing or labels off	high
pagenumber	F1 over <pagenumber page="N"> values	Pagenumber missing or wrong	high
tag_footnote	Per-tag count match for <footnote>	Footnote count drift	medium

composite_tags — informational dimensions

Useful diagnostic signals but excluded from the headline because they fire on convention disagreements that aren't real quality issues — chunking differences, schema vocabulary evolution, hyphenation conventions.

Dimension	What it measures	Why it's excluded from composite_core
text_order	SequenceMatcher ratio on visual-order text (sorted (col, y) within <opinion> then within <footnotes>). Auto-skipped on scan / full-width-figure pages	Brittle on chunking conventions — same content can score 0.3 to 1.0 depending on how the model splits column-wrap paragraphs
no_hallucinated_tags	1 - 0.25 × unique_unknown_tags (tags outside the allowed schema vocabulary)	Useful, but ground truths often disagree with current schema
tag_footnotemark	Count match for <footnotemark>	Count drift; covered semantically by footnote_labels
tag_p	Count match for <p>	Very noisy across chunking conventions
tag_citation	Count match for <citation>	ground truths chunk citations inconsistently
tag_opinion	Count match for <opinion>	Multi-case pages cause noise
tag_parties	Count match for <parties>	Count drift
tag_heading	Count match for <heading>	Convention drift on caption conventions
tag_pagenumber	Count match for <pagenumber>	Almost always 1.0

Why the split matters

Some dimensions catch real defects (content drops, hallucinations, missed case boundaries, footnotes placed in body). Others are sensitive to harmless convention differences (column-wrap chunking, hyphenation, tag naming evolution). Rolling everything into one score makes everything look noisier than it is — the headline composite_core was deliberately designed to exclude dimensions that move on convention disagreements alone.

The "informational" dimensions still get printed and surfaced in flagged-page lists; they just don't drag the headline. If a regression hits both composite_core and a specific tag_*, the tag dimension is usually the more actionable lead.

How "the same content" is measured

A few decisions worth flagging because they make the scoring fairer:

• Text comparison is whitespace-tolerant and punctuation-stripped. _strip_ws lowercases, removes quotes / commas / periods / dashes / ⊥ markers, and collapses whitespace before any similarity check. OCR splits like estab lish vs establish don't register as differences.
• <br/> extracts as a space. Prevents operating<br/>Procedures from tokenizing as one word.
• <img> content is excluded from text scoring entirely. Scanned-exhibit text is image-OCR, not extraction quality. ie handwriting.
• Visual-order text separates <opinion> from <footnotes>. Each container's chunks are sorted by (col, y) independently and concatenated body-then-footnotes. Without this, the same content scores differently depending on whether the model splits a column-wrapped footnote into two <p> chunks vs joining into one — a convention difference with no quality impact.
• fn_band_purity compares to ground truth, not absolute. The check flags pages where the last body block in <opinion> sits at or below the y where footnotes begin — usually a sign the model put footnote-continuation text into <opinion> instead of <footnotes>. On a handful of layouts the heuristic fires on both sides (ground truth and actual carry the same flag), so we score the delta: only penalize when the actual has more of these flags than the ground truth does. That isolates real misclassifications from layout edge cases.
• Column-wrap footnotes are merged in-memory before scoring. Stale saves that fragment one logical footnote into <footnote label="1"> + <footnote orphan> get normalized to a single footnote so they're not penalized retroactively.
• Footnote ordering uses body-mark order, not strictly ascending labels. A page with two cases on it has body marks [74, 75, 1] (case A's 74, 75, then case B's 1). The footnote section in that order is correct, not a regression.

Surfacing failures

Beyond the numeric scores, each page emits:

• text_missed / text_added — words in ground truth but not actual, and vice versa. First five shown in the run_benchmark output for any flagged page.
• warnings — heuristic flags like "last body block sits below the footnote band y, may belong in footnotes." Shown in eval_prompt's Notes column.
• hallucinated_tags.actual_only — element names the model emitted that aren't in the allowed schema.

The combination of dimension scores + targeted word/tag diffs is usually enough to figure out why a page diverges in 30 seconds of inspection.

How to read divergences

The dimension layout makes it possible to distinguish "model is right, ground truth is stale" from "model regressed" without opening the PDF every time.

Model is right, ground truth is stale typically shows:

• text_completeness and text_no_hallucination both ≥ 0.99 (no content drop, no invention)
• One or more tag_* dimensions noticeably below 1.0 (convention drift)
• text_added listing real words from the page (recovered case names, parallel citations the ground truth missed, properly-spelled compound words from joined hyphens)

Model regressed typically shows:

• text_completeness < 0.99 (dropped content) or text_no_hallucination < 0.99 (added content not on the page)
• no_hallucinated_tags < 1.0 (invented an element name outside the schema vocabulary)
• text_added containing words that aren't on the PDF (cross-check by opening the source PDF)

Structural misclassification (footnote into body, body into footnote) typically shows:

• All four text dimensions near 1.0 (every word is somewhere)
• fn_band_purity < 1.0 (body block sitting at footnote-band y)
• footnote_labels or tag_footnote below 1.0 (footnote count off)

Chunking convention difference (not a real defect) typically shows:

• text_completeness, text_no_hallucination, text_sim all = 1.0
• text_order dragged to 0.3–0.8
• tag_p or tag_citation slightly off
• composite_core still ≥ 0.99

This last pattern is the most common and the easiest to mistake for a regression. The rule: if composite_core is fine and the text dimensions are fine, the divergence is in how content is packaged into XML, not in what's there.

Sanity steps before accepting a regression

1. Read the text_missed / text_added lists for the flagged page. Real regressions usually have the dropped/invented words right there.
2. Open the source PDF and confirm whether the actual is reading content the ground truth missed (or vice versa).
3. If the model's version is genuinely better, the ground truth is the one that should change — re-bootstrap by removing the page_NNN.xml and re-running eval_prompt.

Known limitations and caveats

Honesty about where the methodology is weak:

• Ground truth drifts. Many ground truths were written when <otherdate>, <parallelpagenumber>, and italics weren't tagged consistently. Tag-count dimensions (tag_*) often penalize the new schema's improvements. That's why composite_core excludes them — they're informational, not load-bearing.
• Word-split artifacts in ground truths. OCR sometimes splits compound words across whitespace (estab lish). text_no_hallucination is whitespace-tolerant and won't flag this, but the underlying text_added list will still surface the joined version as an "added word." Confusing on first read.
• text_order is brittle. Two failure modes drag it down without indicating a real regression: (1) the model splits or joins a column-wrap paragraph differently than the ground truth (ground truth: L-bottom <p> + R-top <p>; actual: one merged <p col="L">); (2) container reassignment moves a chunk between <opinion> and <footnote> — even though visual reading order is identical. Both produce wildly different visual-order sequences with the same words.
• <img type="scan"> pages need care. When a page is mostly a scanned exhibit with little surrounding body text, text_sim can score very low simply because there's almost no body text outside the image to compare against. body_order and text_order are auto-skipped on these pages; the other dimensions need human review.
• Run-to-run noise at temperature 0.1. Not fully deterministic. Two runs of the same prompt typically agree at >99% word-level, but a single page can flip a number read or shuffle a citation. Always look at multiple sampled pages before concluding "the prompt regressed."
• Case-ending markers on last pages. ground truths generated by older prompts often missed the <!-- Case Ending --> on the final page of a case. We've backfilled the existing curated cases, but newly imported ground truths need a similar check.

What to watch after a prompt change

These dimensions are the most informative leading indicators when iterating on the prompt:

Dimension	What a regression means
no_hallucinated_tags	Prompt isn't tight enough on the schema vocabulary — model is inventing tag names
case_endings	Combined-volume output will lose opinion boundaries — bad downstream impact
body_order / footnote_order	Model is producing content out of reading order — column rules need reinforcement
text_no_hallucination	Model is inventing words. Always verify against the PDF before treating as a regression — could be the model recovering text the ground truth missed
text_completeness	Model is dropping content. Same — verify against the PDF first
fn_band_purity	Model is misclassifying footnote continuations as body

What this enables

With the methodology in place, comparing two prompt + model combinations becomes mechanical:

1. Run extraction against long/ with combo A.
2. Run extraction against long/ with combo B.
3. Compare composite_core per-page, flag the deltas, eyeball the flagged pages to confirm whether they're real or representation noise.

Per-dimension breakouts make the comparison interpretable — "combo B regressed case_endings by 0.01 across 8 pages" is actionable in a way that "combo B is 0.003 worse overall" isn't.

Open questions

1Are there dimensions we're missing? The current set was built around regressions we actually hit. If there are failure modes the corpus regularly sees that aren't surfaced by an existing dimension, that's worth a new one.

Reference

The bar or threshold for a passing page is 0.995 and up. Across all of the
short pages, we have a composite core value of 0.99998133 and for the longer ones with footnotes our score is 0.99984761. No pages fall below the 0.995 threshold and most are perfect matches.

Citation tagging is the most inconsistent and varied across all tags and pagenumbers has no failures.

Actual current output in quality

Aggregate scores across the curated ground truth (run via tests/run_benchmark.py):

Dimension	short/ (48 pages)	long/ (584 pages)
composite_core	0.99998	0.99985
text_completeness	1.00000	0.99970
text_no_hallucination	1.00000	0.99985
text_sim	0.99981	0.99893
body_order	1.00000	1.00000
footnote_order	1.00000	1.00000
fn_band_purity	1.00000	1.00000
case_endings	1.00000	1.00000
footnote_labels	1.00000	1.00000
pagenumber	1.00000	1.00000
tag_footnote	1.00000	1.00000
composite_tags	0.98791	0.97387
text_order	0.98852	0.99588
no_hallucinated_tags	1.00000	0.99828
tag_footnotemark	1.00000	0.99454
tag_p	0.83192	0.91746
tag_citation	0.99085	0.78136
tag_opinion	1.00000	0.94569
tag_parties	0.99583	0.99626
tag_heading	1.00000	0.97443
tag_pagenumber	1.00000	1.00000
composite_text	0.99708	0.99859
composite	0.98984	0.97237

Finally

Readout: composite_core is essentially perfect on both corpora (0.99985+). All eight of the load-bearing structural dimensions (body_order, footnote_order, fn_band_purity, case_endings, footnote_labels, pagenumber, tag_footnote) score 1.0 across all 632 pages. The few text-dimension fractional drops on long/ are almost all from the OCR-hyphen / convention differences described above.

The lower scores all live in composite_tags, which is where convention drift shows up rather than real defects. tag_citation = 0.78 on long/ is the standout — citations chunk inconsistently between the model and goldens (different decisions about what constitutes one citation span vs two). tag_p and tag_opinion follow the same pattern. Neither tag_p or tag_opinion is super important.

User prompts are generated from the redactions.json and detections.json data into a fairly standardized statement. The only real variety occurs when we add the first line of the footnotes to the page. Not all pages are correctly extracted but it will be important that footnote tagging is near perfect .

[quevon24]

Nice breakdown. Heads up: I'm going off what's in the issue and haven't looked at the code, so some of this might already be covered. More questions than gaps.

A couple of things that seem worth adding

Footnote -> body linking

From what I can tell, we check that the set of footnote labels matches and that they show up in the right order, but I don't think we check that each <footnotemark>N in the body actually has a matching <footnote label="N"> (and the other way around). If a model drops a <footnote> but keeps the <footnotemark> in the body, it could still pass if the counts happen to line up. The link is kind of the whole point of a footnote, so it seems worth checking directly.

Citation correctness (not just count)

tag_citation is a count check, which is why it scores 0.78. That's chunking, not real errors. But it doesn't look like we check whether the citation content is actually right: reporter, volume, page, year. Something like "523 U.S. 91" vs "532 U.S. 91" is a totally different case. Might be worth checking the pieces of each citation match the golden, separately from how it's chunked.

Questions on a few others

Party/caption

I was thinking past just the count. Party order matters (Smith v. Jones vs Jones v. Smith flips who's the appellant), plus stuff like "et al." getting dropped or role labels like "Appellant" / "Petitioner". Do we ever expect parties to come out in a different order between the model and the golden on purpose? Or should it always match what's on the page?

Reading order inside a column

If I read it right, body_order checks pairs are in (col, y) order, which catches L/R columns getting mixed up. But two paragraphs both in column L could be swapped, and if the y values still compare correctly, it passes. Have you actually seen this happen? Paragraphs in the right column but wrong order? If it's not a real problem in practice, probably not worth adding.

Footnote content completeness

Depends on how text_completeness is set up. I couldn't tell from the description. If it lumps body and footnote words together, footnote drops get watered down because body text is usually way longer. A model could drop 30% of a footnote and only show a small overall dip. Maybe worth a separate one that only looks at words inside <footnote> elements, since you said footnote tagging is important downstream.

[grossir]

talked with @flooie about

• link to the datasets
• upload a branch with the evaluation scripts

See here
https://drive.google.com/file/d/1onh6tUnN2OzAi-uPeq4Hf-LwMvNpk3PL/view?usp=drive_link

For going more into details + asking Rachel to take a look