Skip to content

Unified single-call date extraction#466

Merged
ppinchuk merged 10 commits into
mainfrom
better-dates
Jun 5, 2026
Merged

Unified single-call date extraction#466
ppinchuk merged 10 commits into
mainfrom
better-dates

Conversation

@rajeee
Copy link
Copy Markdown
Collaborator

@rajeee rajeee commented Jun 3, 2026

Replaces the per-page-voting + URL-shortcut date extractor with a single whole-document LLM call: the URL is passed as a hint, but the document text is the source of truth.

Fixes three failure modes in the old extractor:

  • date-less URLs no longer short-circuit on a truthy {year: null} response and skip the document body
  • empty pages no longer out-vote the real date in the frequency tally
  • collection/upload-date contamination in page headers (e.g. energyzoning.org scrape stamps) no longer wins by repetition

Dev eval accuracy 0.43 → 0.94 (2 failing of 35).

Copilot AI review requested due to automatic review settings June 3, 2026 22:21
@rajeee rajeee requested review from castelao and ppinchuk as code owners June 3, 2026 22:21
@rajeee rajeee marked this pull request as draft June 3, 2026 22:23
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates COMPASS’s ordinance date extraction to use a single whole-document LLM call (with the source URL treated only as a hint) instead of the prior per-page voting + URL-shortcut approach, and updates dev evaluation artifacts to reflect the improved accuracy.

Changes:

  • Reworked DateExtractor.parse() to build one prompt from the full document text and optionally include the download URL as corroborating context.
  • Replaced the old multi-response voting _parse_date logic with a simpler validation-based parser for a single response (but see review comment re: backward compatibility with existing tests/callers).
  • Updated dev eval summary JSON and per-case CSV breakdown with new accuracy/cost/token numbers.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File Description
compass/extraction/date.py Switches to one whole-document LLM call; updates prompt/system guidance; simplifies response parsing/validation.
evals/results/dev/date_extraction_evals.json Updates aggregate dev eval metrics reflecting the new extraction approach.
evals/results/dev/date_extraction_evals_breakdown.csv Updates per-document dev eval outcomes and token/cost breakdown.

Comment thread compass/extraction/date.py
Comment thread compass/extraction/date.py Outdated
Comment thread compass/extraction/date.py Outdated
@codecov-commenter
Copy link
Copy Markdown

codecov-commenter commented Jun 4, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 61.16%. Comparing base (75cc04c) to head (5b87e43).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #466      +/-   ##
==========================================
+ Coverage   60.86%   61.16%   +0.30%     
==========================================
  Files          77       77              
  Lines        6843     6850       +7     
  Branches      670      672       +2     
==========================================
+ Hits         4165     4190      +25     
+ Misses       2561     2543      -18     
  Partials      117      117              
Flag Coverage Δ
unittests 61.16% <100.00%> (+0.30%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@rajeee rajeee marked this pull request as ready for review June 5, 2026 19:12
@ppinchuk ppinchuk merged commit ef0e438 into main Jun 5, 2026
31 checks passed
@ppinchuk ppinchuk deleted the better-dates branch June 5, 2026 20:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants