v1.1.34 — Batch 1: SEC Filing Tracker
Web app that tracks SEC EDGAR filings for crypto/digital-asset content. Extracts risk sections, generates AI summaries, presents an interactive dashboard for crypto lawyers.
Previously, every filing from the same company showed identical static text. Now each filing gets context-aware descriptions based on what the entity IS (ETF, miner, exchange, treasury company) AND what this form type MEANS:
| Entity | Form | Type | Purpose (excerpt) |
|---|---|---|---|
| IBIT | 485APOS | ETF — Post-effective prospectus amendment | ...a spot BTC ETF. Updates prospectus disclosures including risk factors and fund performance. |
| Coinbase | 8-K | Crypto Exchange — Current report | ...a cryptocurrency exchange. Discloses material event such as regulatory action, product launch, or executive change. |
| Marathon | 10-Q | Bitcoin Miner — Quarterly report | ...a BTC mining company. Quarterly mining output, revenue, and operational metrics. |
| MicroStrategy | 8-K | Corporate Treasury — Current report | ...holds BTC as a corporate treasury reserve. Material event such as additional crypto purchase, sale, or strategy change. |
Holdings are also entity-aware:
- ETF: "Holds spot BTC in trust for shareholders"
- Miner: "Mines and holds BTC as primary business output"
- Exchange: "Operates marketplace for BTC, ETH, USDT, SOL, DOGE and other digital assets"
- Company: "Holds BTC as corporate treasury reserve asset"
Previously: "found 2000, saved 0" (confusing — implies failure). Now: "Up to date: all 537 filings already processed (0.2min)" — when there are genuinely no new filings on EDGAR since the last run.
When you first run v1.1.34 on an existing database, it automatically regenerates Type/Purpose/Holdings for all existing filings using the new per-filing template system. No re-scraping required.
pip install -r requirements.txt
# Optional but recommended: set Anthropic API key for Claude summaries
# Without it, the app falls back to template summaries.
export ANTHROPIC_API_KEY=sk-ant-...
python run.py
# Open http://127.0.0.1:5000
# Click "Update Filings" to populate the databaseFocused speed + UX pass after researching the edgar.tools/edgartools/datamule ecosystems and verifying claims locally. The big-ticket "switch to the new edgartools Document API" path was tested and rejected (6-13× SLOWER than the existing BS4+lxml pipeline). Real wins this version:
_search_by_company_cik previously walked 54 companies × 5 forms serially.
Now runs through a ThreadPoolExecutor(max_workers=6). Each worker fetches
one company's filings in isolation; main thread dedups under a lock. Saves
several minutes per Update click.
Every EFTS HTTP response is cached by sha256(query|form|dates) into
~/.crypto_tracker_data/efts_cache/. TTL is 1 hour. Click "Update Filings"
a second time within the hour and the entire search phase is near-instant.
init_db() now runs a one-shot UPDATE that joins existing filings' CIKs
against COMPANY_DETAILS and fills in purpose + holdings for everything
that wasn't yet populated. No re-download, no Claude calls.
Added 11 spot Bitcoin ETF issuers (IBIT, FBTC, ARKB, BITB, HODL, BTCO, BTCW, BRRR, EZBC, GBTC, DEFI), 7 spot Ethereum ETF issuers (ETHA, FETH, ETHE, ETH mini, ETHW, EZET, ETHV), and 5 newer treasury/payments/mining names (Circle/CRCL, BitMine, ETHZilla, SOL Strategies, Bitdeer). All CIKs verified against SEC EDGAR before adding.
The All Filings page now has dedicated "Download Excel" and "CSV" buttons
in its header, with a banner explaining that the raw SQLite .db on disk
is binary and won't open in Excel. Exports respect any active filter, so
"Download Excel" downloads exactly what you're looking at — including the
full risk-section text in a cell.
The "Show Full Risk Section" button on each filing card now displays the character count so you know how big the extracted text is before expanding.
I tested and rejected several "improvements" the research agents proposed:
- edgartools
Document.get_sec_section()— 6-13× slower than our BS4+lxml path on every realistic HTML size we tried. The smarter section detection isn't worth the overhead for a batch pipeline. - edgartools
prepare_for_llm()— method signature exists but crashes withModuleNotFoundError: edgar.documents.aiin 5.33.0. Not shipped. - edgartools
filing.sgml()for HTTP savings —filing.html()already calls.sgml()internally in 5.33. We already get the one-call benefit. - datamule library — 200+ PyPI releases = high API churn risk, and it doesn't beat what edgartools already does for us.
- SEC daily bulk archive — terabytes of unrelated data to find our ~700 crypto filings. Wrong tool for targeted pulls.
v1.1.26 was 2 hours slow and had the same extraction bugs as v24. v1.1.27 is a structural rewrite that addresses the root causes:
Four tables replace the single filings table:
filings— metadata (one row per filing)filing_documents— raw cached text of each attachmentfiling_sections— extraction candidates with confidence scoresfiling_summaries— AI/template summaries
Result: re-extracting or re-summarizing no longer requires re-scraping.
Call POST /api/reprocess to regenerate sections/summaries from cached text.
Three patterns adopted from open-source SEC tooling:
- Boundary-pair regex with largest-span selection (from nlpaueb/edgar-crawler):
match
(Item 1A .. Item 1B)pairs, pick the LARGEST span. TOC entries are adjacent (small spans), body sections are huge. Kills the "extracted the TOC" bug. - Structural HTML prefilter (from alphanome-ai/sec-parser): BeautifulSoup drops TOC tables, unwraps inline tags, then regex runs on clean text.
- Dual-signal validation (from SECurityTr8Ker/FinBERT pattern): sections must pass header match AND keyword-density check. Rejects fee schedules and cross-references.
EDGAR_USE_LOCAL_DATA=True — all filing fetches are cached to disk.
Re-runs are 5-10x faster. Eliminates rate-limit timeouts.
99 search calls → 9. One combined OR-query per form instead of per-keyword.
If ANTHROPIC_API_KEY is set, each filing is summarized by Claude Haiku 4.5
with a structured prompt designed for crypto lawyers. Cost: ~$2 for 2000
filings. Template fallback if no key.
Every filing gets up to 5 extraction candidates with confidence scores. The primary one (highest confidence) drives the display, but alternates are queryable in the detail page — and you can click "Make Primary" on any alternate to override the auto-pick.
19 unit tests run offline without network. All pass:
python test_extraction.py # Offline unit tests (19 tests)
python test_extraction.py --live # + live tests against real SEC filingsTests cover: boundary-pair extraction, prospectus-header extraction, HTML cleaning (TOC removal), dual-signal validation, full pipeline on mock 10-K and mock 485APOS filings.
crypto_tracker/
config.py Version, cache settings, API keys, keywords, forms
database.py 4-table normalized schema + v_filings_display view
scraper.py Combined OR-query search + parallel processing
extractor.py 3-pattern extraction pipeline with confidence scoring
summarizer.py Claude Haiku (primary) + template (fallback)
filing_metadata.py Per-filing Type/Purpose/Holdings template engine (v1.1.34)
app.py Flask routes incl. /api/reprocess for no-rescrape
templates/ Jinja2 HTML (dashboard, filings, settings, detail)
static/css/ Dark theme
data/ SQLite + edgar_cache/ (gitignored)
run.py Entry point
test_extraction.py 19 unit tests + 5 live tests
Research sources that informed this rewrite:
- nlpaueb/edgar-crawler — boundary-pair regex pattern
- alphanome-ai/sec-parser — HTML structural prefilter
- pancak3lullz/SECurityTr8Ker — dual-signal pattern
- dgunning/edgartools — local cache + Fund API
- ProsusAI/finBERT — financial-domain validation idea
| Batch | Description | Status |
|---|---|---|
| 1 | SEC Filing Tracker | v1.1.34 current |
| 2 | Regulatory Action Tracker (SEC/CFTC litigation, no-action letters) | Planned |
| 3 | Real-Time Alerts (RSS monitors + push notifications) | Planned |
| 4 | Full SEC Coverage (beyond crypto) | Planned |
| 5 | User accounts + UI configuration + API | Planned |