test(soak) #175: a soak that hunts the commit-response timeout, and the stall its first runs found (#471)

#175 (confluentinc#809) has reported `Timeout waiting for commit response` for months and
had never been reproduced, by anybody. docs/inflight/bug-177-commit-response-timeout-unreproduced.md
says so in its own title and names the experiment that would change that. This is that experiment,
the opt-in lane it runs in, and its first two runs - which found a different stall instead.

THE SCENARIO. CommitResponseTimeoutSoakIT transcribes the application code #177's reporter
posted in full - that report is closed and is kept only as the source of the shape, since it is the
cheapest route to the wedged-poller question #175 still poses: KEY ordering over a thousand
distinct keys, PERIODIC_CONSUMER_SYNC at a one-second commit interval, maxConcurrency 14, a ~100ms
user function, and half the records throwing. One instance, no churn - the report involves no
rebalance, which is the deliberate difference from every other scenario in the chaos package.

The load-bearing choice is that a poisoned record throws on EVERY attempt, not on about half of
them. The reporter's flag is a per-record header set on i % 2, so a marked record never succeeds;
under KEY ordering it head-of-line blocks its key's shard and pins that partition's committed
offset for the rest of the run. That is what makes "runs for a while and then exits" an
ACCUMULATION rather than a startup race; a per-attempt coin flip would let every record eventually
succeed and the thing being hunted would never build up. The per-attempt shape is a separate arm.

It asserts exactly one thing - that no commit-response timeout occurs - and wires no ProgressProbe,
because this workload pins its own commit watermark by design and the Class 2 lag detector would
observe continuously while saying nothing about the question. Classification is already in the
product: since #204 a message naming the poll thread's death is #100's class and a
bare timeout is the poller wedged-but-alive, carrying PollThreadStallDiagnosis's BLOCKED-vs-SLOW
verdict. The scenario adds the half the product cannot cover - a full JVM thread dump at the moment
of detection.

THE LANE. The soak tag is new and is in no suite at all: it is in the pom's excluded.groups and in
every gating wrapper. It could not join chaos, whose scenarios are calibrated detectors that gate
every PR and whose local recipe would pick a half-hour class up. docs/testing.md gains the lane,
what a soak result is (a rate under conditions, never a verdict), where it gets recorded, and that
this lane is not bin/soak-test.sh, which repeats a short test under CPU contention to measure a
flake rate.

THE RUNS, recorded as rates with conditions in the inflight note and the scenario's Calibration
status. Two thirty-minute runs, zero timeouts - and neither is a sighting-ledger entry, because in
both the assertion could not have failed after the first minute:

    failureFraction 0.5 (#177's reporter's)  seed 3747722682837130843
        90,000 produced, 451 succeeded, 237,006 failed, no findings
    failureFraction 0.03                            seed 5055695573431537469
        90,000 produced, 2,372 succeeded, 81,114 failed, no findings

WHAT THEY ACTUALLY MEASURED. Successes froze inside the first minute of each run and never moved
again, while the producer kept publishing and the failure count climbed at a rate that then held
exactly constant. A constant retry rate against a frozen success count means no new record is being
taken as work at all: the instance has stopped, not merely slowed. A stalled instance cannot reach
the exception being hunted - only PartitionState#onSuccess marks a partition dirty, onFailure is an
explicit no-op, and the control loop gates commits on isDirty, so with no success anywhere no commit
request is enqueued and ConsumerOffsetCommitter#commitAndWait, the sole thrower of the message, is
never entered. A green assertion here cannot tell "no timeout occurred" from "no commit was
attempted". This is the dirty asymmetry upstream-tell-809-833-the-hang-is-fixed.md names for this
very workload. The 0.03 arm is the control that makes the fraction the wrong knob rather than a
guess: it bought about 1.5 extra bursts and stalled identically. Duration is the wrong knob too.

ONE MECHANISM ELIMINATED, ONE NAMED. Offset-encoding back pressure is eliminated:
PartitionState#updateBlockFromEncodingResult logs every transition and neither message appears once
in either log. The named, untested candidate is the load gate - WorkManager#isSufficientlyLoaded
compares inShards minus parkedForRetry against the in-flight target, and inShards counts records
queued BEHIND a blocked shard head while only the failing head is parked. That is the silent-stall
shape the gate's own comment names against confluentinc#857. Stated as a hypothesis; the next arm
reads the gate's own DEBUG line and settles it in one run. The stall may be the better lead than the
timeout: confluentinc#833's reporter showed pc_processed_records_total FLAT across the window in
which their timeout fired, which is this state, not a busy one.

TWO DEFECTS IN THIS BRANCH'S OWN FIRST COMMIT, both silent. The soak tag had been added to the
comment above excluded.groups, to the javadoc and to docs/testing.md, and not to excluded.groups
itself, so a six-hour scenario sat in the GATING integration lane - measured, not inferred: this
PR's own Integration Tests job ran 35 minutes with the scenario's banner in its log.
QuarantinedAnnotationContractTest could not have caught it, since its list checks compare the pom
against the wrappers and a tag missing from both is consistent; it now also asserts membership of
the named opt-in lanes (pomExcludesEveryLaneTooLongToGateAMerge). And the documented run command
could not run: -am builds the parent module first and failsafe fails the reactor there before core
is reached, the trap bin/chaos-test.sh's header already owns. Both copies of the recipe now pass
-Dfailsafe.failIfNoSpecifiedTests=false and say to read the scenario's banner out of the log before
believing a green, since that flag lets a run selecting nothing exit 0.

WHAT MASTER SETTLED WHILE THIS WAS OPEN, reconciled in the merge rather than left contradictory.
The inflight vetting sweep shrank the note to the #175 half, closed the throwing-metrics
candidate against the tree now that #29 and #57 have merged, and recorded the AB-BA
wedge in upstream-175-sporadic-commit-timeouts.md as fixed by #29. So this branch's earlier
reading that candidate 2 was "owned by #29, not unowned" is withdrawn: candidate 2, the
poller wedged but alive, stays the open one with no named mechanism inside it - which is why the
intake stall these runs found is the more promising of the two leads. Arm 3 of what to run next no
longer nominates a "better wedge candidate"; it survives as the closest recorded configuration to
the live report. The note's opening sentence that no reproduction attempt existed is corrected to
name this one.

Static analysis on lines this PR wrote: StringCharset fixed with StandardCharsets.UTF_8; the
deprecated RandomUtils.nextInt() deliberately kept, because it is the topic-naming pattern in three
sibling chaos scenarios and changing one of four leaves the odd one out, so docs/refactoring.md
carries one item covering all four. Prior art recorded in the note: the unmerged branch
test/177-commit-response-timeout, which confirmed the #100 trigger reproduces when the catch
is removed and records one falsified experiment (moving brokerPollSubsystem.supervise() earlier in
controlLoop() does not help).

Co-authored-by: Claude Fable 5.1 (1M context) <noreply@anthropic.com>

Author: astubbs

bin/ci-build.sh

@@ -27,4 +27,4 @@ fi
   -Pci \
   clean verify \
   ${KAFKA_VERSION_ARG:+"$KAFKA_VERSION_ARG"} \
-  -Dexcluded.groups=performance,chaos,quarantined,lincheck
+  -Dexcluded.groups=performance,chaos,quarantined,lincheck,soak

bin/ci-integration-test.sh

@@ -131,7 +131,7 @@ set -euo pipefail
 # The groups the gating run excludes. ONE list, handed both to failsafe and to the coverage gate
 # below - a class tagged into one of these is not expected to report, and two copies of this list
 # would drift in the direction that matters: a group excluded here but not there fails every build.
-readonly EXCLUDED_GROUPS=performance,chaos,quarantined,lincheck
+readonly EXCLUDED_GROUPS=performance,chaos,quarantined,lincheck,soak
 readonly HEAVY_CLASSES="PartitionStateCommittedOffsetIT,Rebalance857CommitSyncDeadlockProbe3IT,Rebalance857CommitSyncDeadlockProbe2IT,TransactionAndCommitModeTest,MultiInstanceRebalanceTest,RebalanceEoSDeadlockTest,Rebalance857CommitSyncDeadlockProbeIT"
 
 # A class in two lists would run twice and be paid for twice, and both shards would pass. With one

bin/ci-unit-test.sh

@@ -22,5 +22,5 @@ set -euo pipefail
 ./mvnw --batch-mode \
   -Pci \
   clean test \
-  -Dexcluded.groups=performance,chaos,quarantined,lincheck \
+  -Dexcluded.groups=performance,chaos,quarantined,lincheck,soak \
   "$@"

docs/inflight/bug-177-commit-response-timeout-unreproduced.md

@@ -8,10 +8,12 @@
 **This file exists because the work had no home.** `bug-857-mirror-attributions-unconfirmed.md`
 correctly says the honest options are "reproduce and diagnose, or close on their own merits" - but it
 owns the *attribution* question, not the investigation, so "reproduce and diagnose" has sat as a
-sentence nobody could pick up. The field report has been open for months with no reproduction
-attempt: a grep of `docs/plans/` and `docs/solutions/` finds nothing targeting it. The only
-adjacent record is `unforceable-trigger-commit-lock-timeout-2026-08-07.md`, which is a *test* flake
-on the same lock and unrelated to the reporter's scenario.
+sentence nobody could pick up. The field report sat open for months with no reproduction attempt at
+all - **the attempt now exists**, as `CommitResponseTimeoutSoakIT` in the `soak` lane, and its first
+two runs are recorded at the bottom of this note. A grep of `docs/plans/` and `docs/solutions/` still
+finds nothing targeting the report; the only adjacent record is
+`unforceable-trigger-commit-lock-timeout-2026-08-07.md`, which is a *test* flake on the same lock and
+unrelated to the reporter's scenario.
 
 **The filename carries astubbs#177 for history only.** That issue was closed on 2026-09-01 without
 the reproduction or the closing comment this note asked for;
@@ -84,6 +86,131 @@ the retry tests, and the chaos suite's `ProgressProbe` plus the new `INSTANCE_ST
 whether an instance is wedged while it happens. **Do not start a parallel harness** - see
 `docs/testing.md`.
 
+**This section is now a record of the design rather than an outstanding request**: it was built as
+`CommitResponseTimeoutSoakIT`, and what its runs actually measured is the next section. Do not build
+it again.
+
+## The experiment now exists, and its first two runs measured the experiment rather than the defect
+
+`CommitResponseTimeoutSoakIT` (the `soak` lane, `docs/testing.md`) is the reproduction attempt this
+note asked for, built from the astubbs#177 shape above for the reason stated there - it is buildable,
+and it is the cheapest route to the wedged-poller question. **2026-09-07, two runs, 0 timeouts in
+2 x 30 minutes - and neither is a sighting-ledger entry, because in both the assertion could not have
+failed after the first minute.**
+
+Conditions common to both runs, so the numbers are interpretable:
+
+| Term | Value |
+|---|---|
+| Duration | 30 min each (`-Dsoak.duration=PT30M`) |
+| Shape | 1 instance, no churn, `KEY` ordering, `PERIODIC_CONSUMER_SYNC`, 1s commit interval |
+| Scale | 1000 keys over 20 partitions, `maxConcurrency` 14, ~100ms user function |
+| Poisoning | per-record and permanent (a poisoned record throws on every attempt) |
+| Produce | 1000 records every 20s - 90,000 produced per run |
+| Broker | the suite's Testcontainers Kafka on Docker |
+| Machine | maintainer's macOS arm64 workstation |
+| Assertion | exactly one - no `Timeout waiting for commit response`, and no other terminal failure |
+
+| Arm | Seed | Succeeded | Failed | Findings |
+|---|---|---|---|---|
+| `failureFraction` 0.5 (astubbs#177's reporter's) | `3747722682837130843` | 451 | 237,006 | none |
+| `failureFraction` 0.03 | `5055695573431537469` | 2,372 | 81,114 | none |
+
+Run command (the second adds `-Dsoak.failureFraction=0.03`):
+
+```
+./mvnw -Pci -pl parallel-consumer-core -am verify -DskipUTs=true \
+  -Dincluded.groups=soak -Dexcluded.groups= -Dit.test=CommitResponseTimeoutSoakIT \
+  -Dfailsafe.failIfNoSpecifiedTests=false -Dsoak.duration=PT30M
+```
+
+### What both runs measured is a total intake stall, not the absence of a timeout
+
+Successes froze - at 451 and at 2,372 - within the first ~60 seconds of each run and **never moved
+again** across the remaining 29 minutes, while the producer kept publishing and the failure count
+climbed at a rate that then held exactly constant. A constant retry rate against a frozen success
+count means no new record is being taken as work at all: the instance has stopped, not merely slowed.
+
+**A stalled instance cannot reach the exception being hunted:**
+
+- only `PartitionState#onSuccess` calls `setDirty`; `onFailure` in the same file is an explicit
+  no-op, so a failing record never marks its partition dirty;
+- the control loop gates on `shouldTryCommitNow` in `AbstractParallelEoSStreamProcessor` -
+  `isTimeToCommitNow() && wm.isDirty() && !isRebalanceInProgress.get()`;
+- so with no success anywhere, nothing is dirty, no commit request is enqueued, and
+  `ConsumerOffsetCommitter#commitAndWait` - the only place `Timeout waiting for commit response` is
+  thrown - is never entered.
+
+A green run therefore cannot distinguish "no timeout occurred" from "no commit was attempted". This
+is exactly the `dirty` asymmetry `upstream-tell-809-833-the-hang-is-fixed.md` names for this same
+workload; these runs are the measurement of it rather than evidence about the report.
+
+**Lowering the poisoned fraction does not fix it, and that is a run rather than a guess:** 0.03 bought
+about 1.5 extra bursts and stalled identically, which rules out "too many poisoned keys" and makes the
+fraction the wrong knob. Duration is the wrong knob too - the stall arrives in minute one of thirty.
+
+### What stops intake is not the documented back pressure - and the candidate is named
+
+`PartitionState#updateBlockFromEncodingResult` logs on every transition (`Offset map data too large`,
+`not allow further messages`). **Neither string appears once in either run's log**, so offset-encoding
+back pressure is eliminated.
+
+The untested candidate is the load gate. `WorkManager#isSufficientlyLoaded` compares
+`workable = inShards - parkedForRetry` against `targetAmountOfRecordsInFlight * loadingFactor`, and
+`inShards` counts records queued **behind** a blocked shard head - records that can never be worked -
+while only the failing head itself is `parkedForRetry`. A shard set full of unworkable queued records
+would therefore read as "sufficiently loaded", the broker poller would stay paused, and nothing would
+ever arrive to change it. That is the silent-stall shape the gate's own comment names against
+confluentinc#857. **This is a hypothesis, not a result.**
+
+### What to run next, in order
+
+1. **Re-run either arm with `WorkManager` at DEBUG and read the `isSufficientlyLoaded=` line at the
+   moment successes freeze.** It prints its own operands (`inShards`, `parkedForRetry`, the threshold)
+   for exactly this purpose. It either confirms or eliminates the load gate, and until it is read the
+   other arms are guesswork. One run settles it.
+2. **Per-attempt rather than per-record failure**, so records eventually succeed, the shards drain and
+   the instance keeps committing for the whole run. On this evidence it is the only shape that keeps
+   the commit path alive indefinitely - promoted from "a different mechanism" to "the first arm that
+   can falsify the assertion at all".
+3. **`gtassone`'s configuration from confluentinc#809** - 128 partitions, concurrency 64, user
+   function 100ms to minutes, `PERIODIC_CONSUMER_SYNC`. This is astubbs#175's own shape, and the
+   scenario does not have it: the workload transcribes the now-closed astubbs#177 report instead,
+   whose defect `upstream-tell-809-833-the-hang-is-fixed.md` says is already fixed.
+   `upstream-175-sporadic-commit-timeouts.md` no longer nominates it as a *wedge* candidate - see
+   the section below - but it remains the closest recorded configuration to the live report, which is
+   what this arm buys.
+
+### The stall may be the better lead than the timeout
+
+confluentinc#833's reporter - astubbs#177's, whose shape this scenario transcribes - showed
+`pc_processed_records_total` **flat** across the window in which their timeout fired, which is this
+state, not a busy one. Whoever picks this up should consider whether the reported timeout is a
+*consequence* of an intake stall rather than a peer of it.
+
+### Where the two adjacent notes leave candidate 2, and why that matters to the next arm
+
+The candidate list at the top of this note already reflects both of these. They are repeated here
+because they are what the remaining soak arms are aimed at, and each is owned by the note named:
+
+- **`upstream-tell-809-833-the-hang-is-fixed.md`** - four `astubbs#177` commits landed 2026-08-19 and
+  close candidate 1's class outright: the poller now publishes its own death
+  (`notifyPollerDied`), waiters are released with its exception, and the message reports the budget it
+  actually waited on. astubbs#177's mirror is closed; confluentinc#833 is still open upstream and
+  still unanswered, which is why its reporter's shape is still worth running here.
+- **`upstream-175-sporadic-commit-timeouts.md`** - confluentinc#809 is *not* the same defect as
+  confluentinc#833 despite the shared message, and its strand-by-strand table now closes its last
+  named strand too: **the poll thread alive but wedged - the AB-BA cycle - was fixed by astubbs#29,
+  merged 2026-09-02**, and that note records the strand as closed for this report.
+
+So candidate 2 stays the open one, with **no named mechanism left inside it**: the single wedge the
+record ever described has been fixed, so a reproduction on current code would be characterising
+something nobody has yet described. That is what makes the intake stall these runs found the more
+promising lead of the two. The branch `test/177-commit-response-timeout` (unmerged) is further prior
+art: it confirmed the astubbs#100 trigger reproduces when the catch is removed, and records one
+**falsified** experiment (moving `brokerPollSubsystem.supervise()` earlier in `controlLoop()` does not
+help - measured).
+
 ## Do not
 
 - Do not attach a closing keyword from any PR on present evidence - see

docs/inflight/test-857-churn-storm-async-stalls.md

@@ -719,3 +719,49 @@ rate-limited warning and nothing else.
 `maxInstanceStall=0ms` also separates this from the line diagnosed above: no member was stalled
 holding work. It is the rebalance dwell alone, over its bound by 632ms of 15000 - the tail shape the
 `ci-disabled-jobs-and-runner-load.md` confound predicts, not a wedge.
+
+## Sighting, 2026-09-08 - the outer wait again, with nothing gating and the Class 2 observation firing all over the topic
+
+<!-- post-merge: checked-begin -->
+`ChaosChurnStormIT.churnStormMeetsSlosAndBalancesLedger` errored after 345.8s on the `Chaos Pain
+Suite 4/4` shard of astubbs/parallel-consumer#471, at head `33906f782`. The failure is the outer
+Awaitility wait, not a detector:
+
+    failure: ConditionTimeoutException: Condition with alias 'all messages consumed under churn'
+    didn't complete within 5 minutes
+    [chaos-probe] peaks: maxRebalanceDwell=4499ms maxDrainDuration=13543ms
+                         maxLagStagnation=150030ms maxInstanceStall=0ms
+
+**Replay seed `984595272001816748`**:
+
+    ./mvnw -Pci -pl parallel-consumer-core -am verify -DskipUTs=true \
+      -Dincluded.groups=chaos -Dexcluded.groups= -Dchaos.seed=984595272001816748
+
+Job: <https://github.com/astubbs/parallel-consumer/actions/runs/34178467617/job/101912551516>.
+
+**Same signature as the 2026-09-03 sighting - and this one is not weak evidence.** That entry says
+outright that its cancelled job makes the timeout unattributable. This job ran to completion on a
+normal PR shard, so the five-minute wait genuinely expired with the fleet still short of its records.
+
+**No gating detector fired.** What fired is `CLASS2_STALL/LAG_STAGNATION`, fifty-one times, over
+twenty-three distinct partitions of the run's one topic - the observation that stopped gating on
+2026-08-25 because it cannot separate a busy fleet from a wedged one. `maxInstanceStall=0ms`
+separates this from the worker-saturation line diagnosed above on 2026-09-07: no member was accused,
+because no member was frozen while holding work. So the two instruments this file has spent a month
+calibrating both read clean, and the run still failed.
+
+**That combination is exactly the per-shard gap this file already names**, and the observation's own
+text names it too: watermarks stagnant across most of the topic while every instance keeps completing
+is covered by nothing that gates - `test-per-shard-liveness-has-no-gate.md` owns the question. This
+is the first seed on the line that pairs a clean per-instance reading with a real outer-wait failure,
+which makes it the better replay target than the ones captured for the saturation line: replay it with
+`-Dchaos.diagnoseStallRecovery=true` and the per-instance telemetry, and if the fleet is completing
+throughout while the aggregate never reaches the count, the gap is demonstrated rather than argued.
+
+**Recorded, not diagnosed, and the branch is argued against on a mechanism.** The merge that produced
+this head changes documentation plus one javadoc block in `CommitResponseTimeoutSoakIT`. The branch as
+a whole (`git diff origin/master...33906f782`) touches no main code at all: its Java is that one new
+scenario, which carries the `soak` tag and is excluded from the integration lane by this branch's own
+pom and wrapper fix, plus one unit test in `QuarantinedAnnotationContractTest`. Nothing it carries is
+loaded by the chaos lane, so it cannot reach `ChaosChurnStormIT`.
+<!-- post-merge: checked-end -->

docs/refactoring.md

@@ -934,6 +934,13 @@ rather than fixed there so the gate's scope stayed one decision.
   `getThrowableProxy()` filtering is already covered by `LogCapture.events()`, so no widening of the
   helper is needed.
 
+### Test infrastructure - `RandomUtils.nextInt()` is deprecated, and every chaos topic name uses it
+
+- **Replace the deprecated `org.apache.commons.lang3.RandomUtils.nextInt()` used to make a unique
+  topic name.** Four call sites, all in `integrationTests/chaostests` (grep `RandomUtils.nextInt()`),
+  and `-Xlint:all` warns on each. Do all four together or none: fixing one leaves a file that reads
+  as the odd one out, which is why the soak scenario that surfaced the warning deliberately did not.
+
 ### Cross-module test clones (the file-similarity backlog behind astubbs#40)
 
 Deferred half of [#40](https://github.com/astubbs/parallel-consumer/issues/40). Its first half - the

docs/testing.md

@@ -333,6 +333,46 @@ is why the chaos job summary prints the peak rather than a verdict - read it as
   javadoc carries the arithmetic, and `heavyRecordsMustNotAllShareOneKey` is the check - the pattern to
   copy is turning the conclusion into an assertion rather than a comment.
 
+## Soak lane (`@Tag("soak")`) - hours-long hunts for a named field report, in no suite at all
+
+A soak is a scenario that runs for **tens of minutes to hours** against one hypothesis, usually a
+field report nobody has reproduced. It shares the chaos suite's scaffolding (`ChaosScenarioBase`,
+`ManagedPCInstance`, `ChaosSeed`, the ambient probe) but not its job: the chaos scenarios are
+*calibrated detectors that gate every PR*, and a soak is an *experiment you run on purpose*. The tag
+is separate for exactly that reason - `soak` sits in `pom.xml`'s `excluded.groups` default, so a
+soak is in no default suite, no gating lane, and **not the chaos shards** (those select classes by
+name through `CHAOS_SCENARIOS`, so a new chaos-tagged class would be invisible to them anyway, but a
+30-minute class in the chaos tag would still be picked up by the local `-Dincluded.groups=chaos`
+recipe above).
+
+- **Run one** - always name the class, because a lane whose members run for half an hour each is not
+  something to select by tag alone:
+  `./mvnw -Pci -pl parallel-consumer-core -am verify -DskipUTs=true -Dincluded.groups=soak -Dexcluded.groups= -Dit.test=<Name> -Dfailsafe.failIfNoSpecifiedTests=false`
+  The last flag is required rather than optional - `-am` builds the parent module first, the named
+  class is not in it, and failsafe fails the reactor there before core is reached (the trap
+  `bin/chaos-test.sh`'s header owns for the chaos lane). It buys that at the price of a run selecting
+  nothing exiting 0, so **read the scenario's own banner and summary lines out of the log before
+  recording a green** - see "A SHARD THAT RAN NOTHING MUST NOT READ AS A PASS" in
+  `bin/ci-integration-test.sh` for the same hazard in the gating lane.
+- **This lane is not `bin/soak-test.sh`.** That script is unrelated: it repeats a *short* test many
+  times under deliberate CPU contention to measure a flake **rate**. The `soak` tag is one long run
+  of one scenario. The word does two jobs in this repo; say which you mean.
+- **CI**: none, deliberately. A soak's result is a *rate under conditions*, and a lane that runs one
+  repetition per PR would report a number nobody can read.
+- **The result goes in a ledger, not in a verdict.** "Zero findings in one 30-minute run" is a
+  sighting-ledger entry: it says the shape did not reproduce once, never that it cannot. Record the
+  runs, the duration, the load shape, the seed, the broker image and the machine, in the
+  `docs/inflight/` note that owns the question - and record the same summary in the scenario's own
+  `Calibration status` javadoc block, the same convention the chaos scenarios use.
+- **Name the arms you did not run.** A soak's value is mostly in what the *next* run should vary, so
+  its javadoc lists the alternative arms in priority order, each changing one term - including the
+  control arm that removes the term under suspicion and nothing else.
+
+Members today: `CommitResponseTimeoutSoakIT` - the reproduction attempt for astubbs#175
+(`Timeout waiting for commit response`), built from the workload shape of the now-closed astubbs#177
+report; its question, candidate mechanisms and discriminator are owned by
+[`bug-177-commit-response-timeout-unreproduced.md`](inflight/bug-177-commit-response-timeout-unreproduced.md).
+
 ## Lincheck lane (`@Tag("lincheck")`) - scheduler-controlled concurrency testing, never gates
 
 Lincheck declares a class's operations and explores thread interleavings against a sequential
@@ -444,6 +484,7 @@ that work lands, and add the link then.
 - Replaying a known schedule to see a failure again -> `bin/chaos-test.sh` with `CHAOS_SEED`.
 - Running one scenario once while you change code -> the IT directly, or the chaos lane.
 - Asking how often, at what rate, or whether a number moves -> an experiment runner.
+- Hunting a field report nobody has reproduced, over tens of minutes -> the soak lane above.
 
 The distinction that matters is that a rate needs N runs. A single run gives a pass or a fail, which
 is not a rate, and no lane in this repo aggregates results across runs.