Ladybug version
v0.18.3
What operating system are you using?
Windows 11
What happened?
Reproducible fail-stop native crash (0xC0000005) during WAL checkpoint under concurrent reader/GC load — localized to the non-blocking checkpoint path
Authorship disclosure: this report was written by Claude, Anthropic's AI, working as the developer on the project that found these behaviors. The test campaigns and evidence are the joint work of the AI and the project's owner, who reviewed this text and filed it.
Summary
Under sustained writes through a single writer with concurrent readers, ladybug intermittently dies with a native access violation (reproduced on 0.18.1 and re-confirmed on 0.18.3, the current release, on its publication day). Across roughly ten hours of rate-controlled testing (~83,000 audited committed writes), every crash landed exactly at a WAL checkpoint fold event — none anywhere else — and a control at a threshold where no fold occurs during a trial produced zero crashes in 20,000 writes per engine. The crash is a race between the checkpoint fold and concurrent reader/GC activity, not a write-rate problem. Evidence below localizes it to the non-blocking concurrent checkpoint introduced in 0.15.4.2 (PR #371, "feat: non-blocking concurrent checkpoint (WAL rotation + MVCC snapshot)"): the last blocking-checkpoint version (0.15.3), given repeated fold events under the same load texture, stalls but never dies — writes and folds cannot overlap there by construction.
The practical severity is high: the fold moment is precisely when large records sit unfolded in the journal, so a mid-fold death usually leaves the database file unrecoverable (see companion issue on WAL replay of large records, which we can file separately).
Environment
- ladybug 0.18.1 and 0.18.3 (Python wheels), Python 3.13.14, Windows 11
- Hardware: desktop, Intel Core Ultra 9 285K, 64 GB RAM
- One writer (all writes serialized through a single connection on a dedicated worker thread), ~20 reader threads polling on their own connections, some
QueryResult objects deliberately abandoned to the Python garbage collector (this mirrors our application's real access pattern)
- Also reproduced once in ordinary production use (real photo-ingest workload, not the test harness) on day one:
python.exe faulting in VCRUNTIME140.dll, exception 0xC0000005, leaving a 0-byte .shadow and an orphaned 16 MB .wal.checkpoint — a fold's working artifacts, still present because the fold never completed.
Reproduction
Harness shape (scripts and logs available on request):
- Lower
checkpoint_threshold to 64 KB so each trial of 250 writes crosses two fold events.
- 20 producer threads submit writes through one serialized writer connection; record mix includes a 512-dimension FLOAT vector property on every 5th write (large WAL records).
- 20 reader threads poll continuously on their own connections; a portion of results are abandoned unclosed to the GC.
- Pace writes with a token bucket at a target rate; audit every write (per-producer prefix + high-water read-back) so a crash's position in the write sequence is exact.
At 2 writes/second, 250 writes per trial: 27% of trials crash (16/60 pooled over three replicate ladders). Time-to-crash is minutes. Free-running at the engine's ceiling (~4.4 writes/s): 3% of trials.
Evidence
1. Crashes occur only at fold events. All 24 crashes (plus one hang) across five ladders landed at one of exactly two committed-write counts — 93±2 and 193±9 — which are the two fold events at the 64 KB threshold. No trial ever crashed between folds; every trial that survived the second fold reached its full 250-write budget.
1b. Confirmed on the current release. 0.18.3 (published July 21), same 20-trial ladder at the 64 KB threshold, 2 writes/s: 6 crashes / 4,281 committed writes (1.40 per 1,000), 4 of the 6 leaving an unopenable database. All six landed in the two fold bands (committed-write counts 93–101 and 194–199), none elsewhere; exit codes were the same access-violation (0xC0000005) and heap-corruption (0xC0000374) pair seen on 0.18.1. The per-trial crash rate (30%) is statistically indistinguishable from the 0.18.1 baseline (27%, 16/60).
2. No fold, no crash. At a 16 MB threshold (no fold occurs within a trial), 100 trials / 20,000 committed writes per engine: zero crashes, zero corpses, on both 0.15.3 and 0.18.1. Fold necessity confirmed. Threshold dose-response is monotonic: 64 KB → 27% of trials, 128 KB (one/two folds per trial) → 10%, 16 MB → 0%.
3. The rate relationship is inverted — slow writes are more dangerous. Pooled over 60 trials per rung at the 64 KB threshold: 2 writes/s → 27% of trials (~1.1 crashes per 1,000 committed writes); 4 writes/s → 3% (~0.1 per 1,000); ceiling rungs → 3%. Our interpretation: at slow rates the fold runs during idle gaps between writes and spends longer exposed to reader polling and GC-driven result destruction. Interactive (human-paced) use is therefore the riskiest regime, not bulk load.
4. Version localization. 0.15.3, given the same load texture and the same two folds per trial at the dangerous 2/s rate, never crashed: its blocking checkpoint stalls colliding writes for 135–160 s (our broker logs each collision explicitly: the write waits and then lands durably), so fold and write/reader activity cannot overlap. Honest caveat: because of those stalls, the 0.15.3 leg could not be dosed to equal committed-write exposure — this is a mechanistic argument plus repeated fold-event survival, not a matched-count statistical acquittal. But the mechanism is the point: the race needs overlap, and overlap is exactly what the 0.15.4.2 concurrent checkpoint introduced in PR #371.
5. Failure character. Always fail-stop native death (access-violation / heap-corruption family). Across ~83,000 audited committed writes: zero cases of the engine claiming a write had landed when it had not, zero silent corruption in surviving databases. The engine dies honestly. The damage is the corpse: a death mid-fold typically leaves the file unopenable.
A wrong turn, documented so it isn't repeated
We first hypothesized the crash was an abandoned QueryResult's finalizer performing a native close on the GC thread, racing the owning thread. A matched-intensity interleaved A/B (80 trials, alternating shielded/unshielded finalization) falsified it: 27/40 crashes unshielded vs 24/40 shielded — no effect. Shielding result finalization does not mitigate; the race is inside the fold's interaction with concurrent activity, not in the binding's close path alone.
Where we'd look
Given the localization, the interaction of the concurrent checkpoint fold with (a) reader transactions holding MVCC snapshots during rotation and (b) native-side destruction of abandoned results while the fold is in flight seems the bounded search area. Two deterministic reproductions, full ladder logs, per-trial audits, and corpse files from the synthetic reproductions are available — happy to share all of it, run variant builds, or test a patch on the same rig that produced the curves. The production corpse (with its .shadow and .wal.checkpoint artifacts) is preserved but contains private data and stays on our machine; we can run any instrumented build or diagnostic you provide against it locally and report the output.
Thank you for the fork — 0.15.4.2's concurrent checkpoint cured a real starvation problem for us (70-second write stalls under polling readers), and 0.18.x has been substantially faster and cleaner in every other respect. We'd much rather help fix this than work around it.
Are there known steps to reproduce?
No response
Ladybug version
v0.18.3
What operating system are you using?
Windows 11
What happened?
Reproducible fail-stop native crash (0xC0000005) during WAL checkpoint under concurrent reader/GC load — localized to the non-blocking checkpoint path
Authorship disclosure: this report was written by Claude, Anthropic's AI, working as the developer on the project that found these behaviors. The test campaigns and evidence are the joint work of the AI and the project's owner, who reviewed this text and filed it.
Summary
Under sustained writes through a single writer with concurrent readers,
ladybugintermittently dies with a native access violation (reproduced on 0.18.1 and re-confirmed on 0.18.3, the current release, on its publication day). Across roughly ten hours of rate-controlled testing (~83,000 audited committed writes), every crash landed exactly at a WAL checkpoint fold event — none anywhere else — and a control at a threshold where no fold occurs during a trial produced zero crashes in 20,000 writes per engine. The crash is a race between the checkpoint fold and concurrent reader/GC activity, not a write-rate problem. Evidence below localizes it to the non-blocking concurrent checkpoint introduced in 0.15.4.2 (PR #371, "feat: non-blocking concurrent checkpoint (WAL rotation + MVCC snapshot)"): the last blocking-checkpoint version (0.15.3), given repeated fold events under the same load texture, stalls but never dies — writes and folds cannot overlap there by construction.The practical severity is high: the fold moment is precisely when large records sit unfolded in the journal, so a mid-fold death usually leaves the database file unrecoverable (see companion issue on WAL replay of large records, which we can file separately).
Environment
QueryResultobjects deliberately abandoned to the Python garbage collector (this mirrors our application's real access pattern)python.exefaulting inVCRUNTIME140.dll, exception 0xC0000005, leaving a 0-byte.shadowand an orphaned 16 MB.wal.checkpoint— a fold's working artifacts, still present because the fold never completed.Reproduction
Harness shape (scripts and logs available on request):
checkpoint_thresholdto 64 KB so each trial of 250 writes crosses two fold events.At 2 writes/second, 250 writes per trial: 27% of trials crash (16/60 pooled over three replicate ladders). Time-to-crash is minutes. Free-running at the engine's ceiling (~4.4 writes/s): 3% of trials.
Evidence
1. Crashes occur only at fold events. All 24 crashes (plus one hang) across five ladders landed at one of exactly two committed-write counts — 93±2 and 193±9 — which are the two fold events at the 64 KB threshold. No trial ever crashed between folds; every trial that survived the second fold reached its full 250-write budget.
1b. Confirmed on the current release. 0.18.3 (published July 21), same 20-trial ladder at the 64 KB threshold, 2 writes/s: 6 crashes / 4,281 committed writes (1.40 per 1,000), 4 of the 6 leaving an unopenable database. All six landed in the two fold bands (committed-write counts 93–101 and 194–199), none elsewhere; exit codes were the same access-violation (0xC0000005) and heap-corruption (0xC0000374) pair seen on 0.18.1. The per-trial crash rate (30%) is statistically indistinguishable from the 0.18.1 baseline (27%, 16/60).
2. No fold, no crash. At a 16 MB threshold (no fold occurs within a trial), 100 trials / 20,000 committed writes per engine: zero crashes, zero corpses, on both 0.15.3 and 0.18.1. Fold necessity confirmed. Threshold dose-response is monotonic: 64 KB → 27% of trials, 128 KB (one/two folds per trial) → 10%, 16 MB → 0%.
3. The rate relationship is inverted — slow writes are more dangerous. Pooled over 60 trials per rung at the 64 KB threshold: 2 writes/s → 27% of trials (~1.1 crashes per 1,000 committed writes); 4 writes/s → 3% (~0.1 per 1,000); ceiling rungs → 3%. Our interpretation: at slow rates the fold runs during idle gaps between writes and spends longer exposed to reader polling and GC-driven result destruction. Interactive (human-paced) use is therefore the riskiest regime, not bulk load.
4. Version localization. 0.15.3, given the same load texture and the same two folds per trial at the dangerous 2/s rate, never crashed: its blocking checkpoint stalls colliding writes for 135–160 s (our broker logs each collision explicitly: the write waits and then lands durably), so fold and write/reader activity cannot overlap. Honest caveat: because of those stalls, the 0.15.3 leg could not be dosed to equal committed-write exposure — this is a mechanistic argument plus repeated fold-event survival, not a matched-count statistical acquittal. But the mechanism is the point: the race needs overlap, and overlap is exactly what the 0.15.4.2 concurrent checkpoint introduced in PR #371.
5. Failure character. Always fail-stop native death (access-violation / heap-corruption family). Across ~83,000 audited committed writes: zero cases of the engine claiming a write had landed when it had not, zero silent corruption in surviving databases. The engine dies honestly. The damage is the corpse: a death mid-fold typically leaves the file unopenable.
A wrong turn, documented so it isn't repeated
We first hypothesized the crash was an abandoned
QueryResult's finalizer performing a native close on the GC thread, racing the owning thread. A matched-intensity interleaved A/B (80 trials, alternating shielded/unshielded finalization) falsified it: 27/40 crashes unshielded vs 24/40 shielded — no effect. Shielding result finalization does not mitigate; the race is inside the fold's interaction with concurrent activity, not in the binding's close path alone.Where we'd look
Given the localization, the interaction of the concurrent checkpoint fold with (a) reader transactions holding MVCC snapshots during rotation and (b) native-side destruction of abandoned results while the fold is in flight seems the bounded search area. Two deterministic reproductions, full ladder logs, per-trial audits, and corpse files from the synthetic reproductions are available — happy to share all of it, run variant builds, or test a patch on the same rig that produced the curves. The production corpse (with its
.shadowand.wal.checkpointartifacts) is preserved but contains private data and stays on our machine; we can run any instrumented build or diagnostic you provide against it locally and report the output.Thank you for the fork — 0.15.4.2's concurrent checkpoint cured a real starvation problem for us (70-second write stalls under polling readers), and 0.18.x has been substantially faster and cleaner in every other respect. We'd much rather help fix this than work around it.
Are there known steps to reproduce?
No response