TL;DR
Prod's nightly DOAB harvest (load_doab at 04:30 UTC) has been failing with HTTP 403 every run since ~2026-05-21. Root cause is Cloudflare bot-mitigation on directory.doabooks.org, not anything DOAB-application-level and not anything we changed. doab-check continues to harvest fine — almost certainly because its IP was pre-allowlisted by OAPEN long ago.
Status: filed 2026-05-26. Action needed: Eric to ask OAPEN to allowlist the three unglue.it IPs (immediate) and ideally exempt /oai/* from bot mgmt (cleaner long-term).
The ask for OAPEN
Immediate — add to allowlist on directory.doabooks.org:
| Host |
IP |
Why |
| unglue.it (prod) |
52.4.17.140 |
Nightly production DOAB harvest |
| test.unglue.it |
3.209.72.10 |
Pre-deploy testing of harvester changes |
| dj42.unglue.it |
54.165.7.142 |
Django 4.2 migration testing |
Worth considering as a follow-up — a Cloudflare path-rule (http.request.uri.path matches "^/oai/") exempting /oai/* from bot mgmt entirely. OAI-PMH is by definition a machine-to-machine protocol; bot challenges on it conflict with the protocol's design intent. Cleaner than maintaining per-partner allowlists and would help all of DOAB's OAI consumers.
Pre-drafted message text for OAPEN is in comment 2 below.
Smoking-gun evidence
Same one-line curl from 5 different network locations:
curl -sSI 'https://directory.doabooks.org/oai/request?verb=Identify' \
| grep -iE '^(HTTP|cf-mitigated|cf-ray|server)'
| Host |
Public IP |
Network |
HTTP |
Cloudflare verdict |
PoP |
| unglue.it (prod) |
52.4.17.140 |
AWS us-east-1 |
403 |
cf-mitigated: challenge |
IAD |
| test.unglue.it |
3.209.72.10 |
AWS us-east-1 |
403 |
cf-mitigated: challenge |
IAD |
| dj42.unglue.it |
54.165.7.142 |
AWS us-east-1 |
403 |
cf-mitigated: challenge |
IAD |
| Raymond's laptop |
23.93.118.183 |
residential (Sonic, Bay Area) |
403 |
cf-mitigated: challenge |
SJC |
| doab-check |
207.154.238.122 |
DigitalOcean fra1 |
200 |
(no challenge header) |
FRA |
4 of 5 IPs blocked across 2 networks and 2 Cloudflare PoPs; only doab-check passes. Rules out AWS-IP problem, geography, request shape, and TLS fingerprint. Leaves "doab-check IP is allowlisted; we're not."
Impact / what this blocks
- Daily DOAB updates to unglue.it stalled since 5/21
- Mid-June bundle (patched pyoai + DOAB backfill + Django 4.2) cannot fully validate on prod until unblocked
- doab-check independently unaffected — its own pipeline is healthy and continues to load nightly
Silver lining for now
Prod currently runs unpatched upstream pyoai 2.5.1 (no RateLimitedError, no retry-on-429 logic). The cron fails fast on the first 403 each night rather than hammering Cloudflare with retries. Once the patched pyoai fork ships to prod with the mid-June bundle, retry behavior will need to be re-validated against any remaining block.
Detailed evidence (original hypothesis writeup)
1. The 403 carries a Cloudflare challenge marker, not a DOAB application response
From prod (52.4.17.140):
< HTTP/2 403
< content-type: text/html; charset=UTF-8
< cf-mitigated: challenge
< server: cloudflare
< cf-ray: a01ebdce7b7dc562-IAD
Body starts with <!DOCTYPE html>...<title>Just a moment...</title> — Cloudflare's standard JS interstitial. cf-mitigated: challenge is the explicit Cloudflare header for "client must solve a browser/JS challenge to proceed."
2. Request shape is byte-for-byte identical between prod (fails) and doab-check (succeeds)
Both curl --max-time 15 'https://directory.doabooks.org/oai/request?verb=Identify':
| Field |
prod (52.4.17.140) |
doab-check (207.154.238.122) |
| Method/path |
GET /oai/request?verb=Identify HTTP/2 |
(same) |
| Host |
directory.doabooks.org |
(same) |
| User-Agent |
curl/7.68.0 |
curl/7.81.0 |
| Accept |
*/* |
(same) |
| Response |
HTTP/2 403 + cf-mitigated: challenge |
HTTP/2 200 + cf-cache-status: DYNAMIC |
3. The block is not AWS-specific
Probing from Raymond's home machine (residential ISP, IP 23.93.118.183, not AWS): also HTTP 403 + same cf-mitigated: challenge. Spoofing browser User-Agent + Accept-Language from prod: still 403. Rules out AWS IP reputation.
4. Pre-allowlist hypothesis fits the asymmetry
doab-check.ebookfoundation.org has been a long-standing OAPEN partner; reasonable that its IP was added to the DOAB Cloudflare allowlist when first stood upunglue.it prod IP has never (to our knowledge) been registered with OAPEN as a known harvester- The 5/21 cutover is sharp, not gradual — consistent with a Cloudflare config toggle, not gradual IP reputation drift
5. Why this isn't finger-pointing
OAPEN's tightening (probably reacting to bot pressure of their own) is entirely reasonable — unglue.it has been dealing with similar bot pressure on its own infrastructure. We just need to be added to the same trusted-partner list as doab-check.
Falsification tests (already partially run)
OAI-PMH protocol-level critique (worth raising with OAPEN as a follow-up)
OAI-PMH is, by design, a protocol for automated metadata harvesting between machines. It exists precisely so that repositories, aggregators, and discovery services can pull records from data providers programmatically. The standard clients are headless tools (pyoai, sickle, oai-harvester, language-specific libraries) — none of them can solve JavaScript browser-fingerprint challenges, and none of them should have to.
An OAI-PMH endpoint that returns Cloudflare's cf-mitigated: challenge to clean curl / Python-urllib requests is effectively non-functional as an OAI endpoint — the protocol's entire user base is "bots" in the Cloudflare sense.
The cleanest fix on OAPEN's side may not be a per-IP allowlist (brittle, creates maintenance burden as harvesters change IPs) but rather a Cloudflare path rule exempting /oai/* from bot management entirely, while leaving the human-facing pages of directory.doabooks.org protected. Cloudflare supports this via "Configuration Rules" or "WAF Custom Rules" with the path filter http.request.uri.path matches "^/oai/". That would:
- Let all legitimate OAI harvesters reach the endpoint (consistent with the protocol's design intent)
- Keep the human-facing pages protected from the bot pressure that prompted the tightening
- Eliminate per-partner allowlist maintenance
Other OAI-PMH consumers of DOAB (there are presumably many — DOAB is a major OA discovery source) are likely hitting the same wall and either silently failing or quietly asking for individual allowlists.
Provenance + history
- Original issue body and 2 follow-up comments evolved this from "hypothesis with falsification tests" → "confirmed via 4-of-5 evidence table" → "scope correction (prod + test + dj42) + OAI-PMH critique." This rewrite consolidates everything above into a TL;DR-first body. Comment 1 (4-of-5 evidence) and comment 2 (scope + critique) preserved for audit trail.
- Email to Eric drafted 2026-05-26:
~/C/src/Gluejar/EMAIL_ERIC_DOAB_BLOCK_2026-05-26.md
- Agenda item for Thursday 2026-05-28 meeting: [[2026-05-28 Eric Meeting Agenda]]
TL;DR
Prod's nightly DOAB harvest (
load_doabat 04:30 UTC) has been failing withHTTP 403every run since ~2026-05-21. Root cause is Cloudflare bot-mitigation ondirectory.doabooks.org, not anything DOAB-application-level and not anything we changed.doab-checkcontinues to harvest fine — almost certainly because its IP was pre-allowlisted by OAPEN long ago.Status: filed 2026-05-26. Action needed: Eric to ask OAPEN to allowlist the three unglue.it IPs (immediate) and ideally exempt
/oai/*from bot mgmt (cleaner long-term).The ask for OAPEN
Immediate — add to allowlist on
directory.doabooks.org:52.4.17.1403.209.72.1054.165.7.142Worth considering as a follow-up — a Cloudflare path-rule (
http.request.uri.path matches "^/oai/") exempting/oai/*from bot mgmt entirely. OAI-PMH is by definition a machine-to-machine protocol; bot challenges on it conflict with the protocol's design intent. Cleaner than maintaining per-partner allowlists and would help all of DOAB's OAI consumers.Pre-drafted message text for OAPEN is in comment 2 below.
Smoking-gun evidence
Same one-line
curlfrom 5 different network locations:52.4.17.140403cf-mitigated: challenge3.209.72.10403cf-mitigated: challenge54.165.7.142403cf-mitigated: challenge23.93.118.183403cf-mitigated: challenge207.154.238.1222004 of 5 IPs blocked across 2 networks and 2 Cloudflare PoPs; only doab-check passes. Rules out AWS-IP problem, geography, request shape, and TLS fingerprint. Leaves "doab-check IP is allowlisted; we're not."
Impact / what this blocks
Silver lining for now
Prod currently runs unpatched upstream pyoai 2.5.1 (no
RateLimitedError, no retry-on-429 logic). The cron fails fast on the first 403 each night rather than hammering Cloudflare with retries. Once the patched pyoai fork ships to prod with the mid-June bundle, retry behavior will need to be re-validated against any remaining block.Detailed evidence (original hypothesis writeup)
1. The 403 carries a Cloudflare challenge marker, not a DOAB application response
From prod (
52.4.17.140):Body starts with
<!DOCTYPE html>...<title>Just a moment...</title>— Cloudflare's standard JS interstitial.cf-mitigated: challengeis the explicit Cloudflare header for "client must solve a browser/JS challenge to proceed."2. Request shape is byte-for-byte identical between prod (fails) and doab-check (succeeds)
Both
curl --max-time 15 'https://directory.doabooks.org/oai/request?verb=Identify':GET /oai/request?verb=Identify HTTP/2directory.doabooks.orgcurl/7.68.0curl/7.81.0*/*cf-mitigated: challengecf-cache-status: DYNAMIC3. The block is not AWS-specific
Probing from Raymond's home machine (residential ISP, IP
23.93.118.183, not AWS): alsoHTTP 403+ samecf-mitigated: challenge. Spoofing browser User-Agent + Accept-Language from prod: still 403. Rules out AWS IP reputation.4. Pre-allowlist hypothesis fits the asymmetry
doab-check.ebookfoundation.orghas been a long-standing OAPEN partner; reasonable that its IP was added to the DOAB Cloudflare allowlist when first stood upunglue.itprod IP has never (to our knowledge) been registered with OAPEN as a known harvester5. Why this isn't finger-pointing
OAPEN's tightening (probably reacting to bot pressure of their own) is entirely reasonable — unglue.it has been dealing with similar bot pressure on its own infrastructure. We just need to be added to the same trusted-partner list as doab-check.
Falsification tests (already partially run)
https://directory.doabooks.org/oai/request?verb=Identifyin Safari — Raymond to record result. (If browser loads cleanly while curl fails, confirms "challenges non-browsers" framing.)OAI-PMH protocol-level critique (worth raising with OAPEN as a follow-up)
OAI-PMH is, by design, a protocol for automated metadata harvesting between machines. It exists precisely so that repositories, aggregators, and discovery services can pull records from data providers programmatically. The standard clients are headless tools (pyoai, sickle, oai-harvester, language-specific libraries) — none of them can solve JavaScript browser-fingerprint challenges, and none of them should have to.
An OAI-PMH endpoint that returns Cloudflare's
cf-mitigated: challengeto cleancurl/Python-urllibrequests is effectively non-functional as an OAI endpoint — the protocol's entire user base is "bots" in the Cloudflare sense.The cleanest fix on OAPEN's side may not be a per-IP allowlist (brittle, creates maintenance burden as harvesters change IPs) but rather a Cloudflare path rule exempting
/oai/*from bot management entirely, while leaving the human-facing pages ofdirectory.doabooks.orgprotected. Cloudflare supports this via "Configuration Rules" or "WAF Custom Rules" with the path filterhttp.request.uri.path matches "^/oai/". That would:Other OAI-PMH consumers of DOAB (there are presumably many — DOAB is a major OA discovery source) are likely hitting the same wall and either silently failing or quietly asking for individual allowlists.
Provenance + history
~/C/src/Gluejar/EMAIL_ERIC_DOAB_BLOCK_2026-05-26.md