Concurrent requests cause spurious failures, empty extractions, and occasional full hangs

[makothap]

Summary

Under concurrent requests, a single engine instance produces spurious
failures and silent empty extractions for PDFs that process perfectly in
isolation — and under higher concurrency it can hang entirely (idle, not
crashed) until restarted. Serializing requests eliminates the problem
completely.

Observed on the current main (f361d1e, v0.0.35), self-built image,
DGX Spark (aarch64, GB10 GPU, CUDA 13 — but nothing below looks
architecture-specific).

Evidence (100-PDF batch, identical corpus, varying only client concurrency)

Concurrent requests in flight	FAILED	Empty extractions (0 segments)	Notes
1 (serialized)	0/100	0	clean baseline
2	4/100	~33
4	19/100	~55
8	24/100	~41	plus a ~19-minute full hang (below)

• Failures are generic engine-side errors from
  src/drivers/rest/catch_exceptions.py — FileNotFoundError somewhere in
  the request path surfaces as 404 "No xml file", everything else as 422
  "Error see traceback". The pattern (plus the isolation test below) is
  consistent with a shared/temp-file race in the request path
  (PDFAnalysisServiceAdapter / FileSystemRepository).
• Empty extractions are silent: the request returns 200 with zero
  segments for documents that yield full segment lists when processed alone.
• Isolation test: a PDF that failed with 0 segments under load was
  re-POSTed to /markdown standalone — clean, full extraction. The only
  variable was concurrent load.

The hang (8 concurrent requests)

Six requests sat open at ~0% GPU / ~0.6% container CPU for ~19 minutes —
engine idle but alive (GET / still responded instantly). A container
restart cleared it immediately; the stuck requests errored out and client
retries succeeded within a minute.

What we're not asking

We've worked around this client-side (serializing requests) and it's fine
for our use. Filing because (a) nothing in the README suggests requests must
be serialized, so others will hit this, and (b) if there's an intended
concurrency model (e.g. single-request-per-instance), documenting it — or
guarding with an internal lock/queue — would save users the debugging trip.
Happy to run diagnostics or test patches on our setup (we process real
batches through a self-built image and can reproduce reliably).

Environment

• Image built from main @ f361d1e with
  BUILDER_IMAGE=nvidia/cuda:13.0.3-cudnn-devel-ubuntu24.04,
  TORCH_INDEX_URL=.../cu130, TORCH_CUDA_ARCH_LIST="12.0;12.1"
• Host: NVIDIA DGX Spark (aarch64, GB10, driver 580 / CUDA 13.0), Docker +
  nvidia-container-toolkit
• Client: HTTP multipart POSTs to /markdown and /html (with
  output_file and, on our patched build, segment_boxes — failures occur
  with and without the patch; the baseline run used stock endpoints)