fix(core) astubbs#119: the intake load gate says so when it has latched with nothing retiring

[astubbs]

Serves #119 (confluentinc#857) - it does not close it.

Description

#487 measured the record-intake stall and left the cheap mitigation
written up but not built. Under retry-forever plus any non-zero fraction of records that never
succeed, an instance ends with every partition paused, its workers churning on records that never
retire, and nothing in the log - the state was exported only as the pc.partitions.paused
gauge. In #487's low-poison arm - that PR's measurement, not a claim about
every instance - the instance reached it with eleven of its fourteen workers idle, looking healthy
from outside. This builds that mitigation: one observation and one log line.

It takes a fraction, not one bad record. A single record that never succeeds is one held minus
one parked against a threshold of tens, so it never crosses; its offset map encodes one gap
compactly, the commit sits below it, and the instance runs indefinitely with that record retrying
underneath a healthy stream. What latches the gate is the share of held records that never leave
rising while the stream keeps arriving - healthy ones retire, these do not.

No semantic change

The gate's decision, the poller's pausing, the retry service and every counter are untouched.
WorkManager#isSufficientlyLoaded() returns exactly what it returned before, and the broker-poll
thread's route into it through shouldThrottle() observes nothing at all. The control loop's
once-per-pass call in maybeWakeupPoller becomes isSufficientlyLoadedReportingLatch(pausedPartitions),
which takes one reading of the shards and uses it for both the wakeup decision and the report -
so the report can never print an equation the decision was not made on, which is the same trap
ShardManager#getWorkableRecords() already exists to close for the DEBUG line.

The signal is consecutive passes, not elapsed time

Both operands were already there: the gate's own reading, and RecordPopulation#getRetiredTotal(),
which is monotonic and is exactly "no record left a shard, by any route". Nothing new is counted, and
no clock is read. A timing bound would be a threshold argument nobody can win - see
docs/solutions/best-practices/a-timing-bound-used-as-a-correctness-gate-manufactures-its-own-evidence.md.

Why 100 passes

Chosen from the loop's two cadences, not from a target wall-clock time, because when nothing retires
the pass rate differs by two orders of magnitude between the state being reported and the state that
must not be:

• Latched - every held record fails on every attempt, so results arrive in the mailbox
  continuously and each pass returns almost at once. As measured by
  test(soak) astubbs#119: any retry-forever instance eventually stops fetching, at a threshold you can compute #487's low-poison arm - 6,201 gate evaluations across a nine-minute
  unbroken latch, about 87ms a pass - 100 passes is roughly nine seconds, and the state is
  permanent, so being a few seconds late costs nothing.
• Healthy, inside a long user function - the mailbox is empty, so the pass blocks for
  getTimeToBlockFor(), bounded by the commit interval (five seconds by default). 100 passes is then
  over eight minutes in which not one record anywhere in the instance retired.

So the pass count gives the healthy case roughly fifty times the wall-clock grace it gives the
state it is hunting, which no single elapsed-time bound can do. PC puts no ceiling on a user
function, which is why the report is a WARN and not an exception. The full derivation lives on
WorkManager#LATCHED_PASSES_BEFORE_WARNING.

Two known false-alarm windows, both found by review and both named in that javadoc - eight minutes
is the best case, not the bound. getTimeToBlockFor() has two branches:

• The commit interval is not always five seconds. Under PERIODIC_TRANSACTIONAL_PRODUCER the
  default is DEFAULT_COMMIT_INTERVAL_FOR_TRANSACTIONS, two orders of magnitude shorter, so the
  healthy grace collapses to the same order as the latched cadence.
• The retry-delay branch needs no unusual configuration at all. When
  isWorkInFlightMeetingTarget() is false - dispatch below full concurrency, the ordinary state
  under KEY/PARTITION ordering - and any record is in retry back-off, the pass blocks for
  min(commitInterval, max(defaultMessageRetryDelay, lowestScheduled)): a one-second cadence on
  stock defaults, so about a hundred seconds of grace rather than eight minutes. Recorded as a static
  trace of the two branches, not a measured arm.

The report carries its own discriminator in both cases - it prints parkedForRetry, which this state
holds continuously and a merely-slow instance reads as zero - so narrowing the trigger on that term
is deliberately not done here: it changes what fires and wants its own measured arm. Left open in
the note, with both candidates.

Once, then quiet, then once on recovery

The WARN fires on the pass that reaches the count and not again. The clear fires when either clause
stops holding, and re-arms the WARN so a second latch is not swallowed by the first. The WARN names
inShards, parkedForRetry, workable, target * loadingFactor and pausedPartitions - scalars
only, no collection interpolated, per docs/inflight/bug-unbounded-log-lines.md.

The clear is two statements, not one, because the two ways it clears are not the same news. A
gate that has gone unloaded means intake can really resume and says so; a record leaving the shards
while the gate still reads loaded means the poller is still paused, and reporting that as "intake
has resumed" would be a false all-clear. Codex review found this through the recovery test itself,
which clears with 399 records still held against a threshold of two.

And neither line claims processing recovered. The trigger reads getRetiredTotal(), which rises
when a record leaves a shard by any route - success, revocation, or a stale container being swept.
That is right for the trigger (a revocation really does drain the shards and unlatch the gate) and
wrong for a message, because revoking a stalled instance is exactly what an operator or the group
coordinator does to one. Both lines now report what is measured and name the three routes. Pinned
by an arm that latches, revokes with zero successes, and asserts the clear does not claim recovery.

It logs on the WorkManager logger deliberately: both test logback profiles already pin that logger
name for the load gate (pc.loadgate.log.level), so the WARN reaches a soak run without a logging
change, and the recovery INFO reaches it too under the integration profile's info default.

Thread model

The three counters are written and read only by observeLoadGateLatch, whose one production caller
is maybeWakeupPoller on the control thread. They deliberately carry no @ThreadConfined and no
owning-thread assertion, which is this repo's usual pairing: the harness drives controlLoop from
more than one thread inside a single test (BlockedThreadAsserter in ProducerManagerTest), so the
assertion would fire on a legitimate caller - and a diagnostic that can kill the consumer is a worse
trade than one that can miscount. No decision reads these fields; the worst a foreign caller costs is
one emitted or missed WARN. Recorded in docs/refactoring.md's non-volatile ledger so a later "fix"
with volatile has the argument in front of it.

Red first, on both clauses

Two sabotage arms, each restored afterwards:

• Report removed entirely (master's behaviour): the two positive tests fail on their assertions
  rather than on compilation; the control test still passes, which is what a negative control should
  do.
• "Nothing retired" clause dropped: exactly the other two flip - the healthy-instance test now
  sees a WARN naming 300 held records, and the recovery INFO never arrives.

Green with both restored. Full local unit suite green, and the recovery test gained an arm that
drains below the threshold so the unloaded branch is exercised rather than assumed.

Review

Codex reviewed commit 918bc4c31 at the owner's request: five findings, four taken, one rejected
with the argument recorded as a dated cleared suspicion on observeLoadGateLatch. Every thread has
an in-thread reply and is resolved. The rejected one asked for the observation to move after the
mailbox drain; the window between two consecutive readings already spans a whole pass, drain
included, so nothing is invisible - what the position costs is one pass of latency out of a
hundred, and gating on an empty mailbox would suppress the WARN in exactly the state it exists for,
because the latched workload's failure results flow through that same mailbox continuously.

Also

BrokerPollSystem gains getPausedPartitionCountForBackPressure(), and
isSubscriptionsPausedForBackPressure() is now expressed in terms of it rather than repeating the
same cache read.

Open, for the owner to decide - not taken here

The diagnostic is silent outside RUNNING. maybeWakeupPoller's state == RUNNING guard was
right for its original job (do not wake a poller you are not running) and now also gates the
observation. A graceful shutdown that hangs in DRAINING because the buffer will not drain is a
plausible shape of this very defect, and there the WARN can never fire. Widening the guard - keeping
RUNNING for wakeupIfPaused() and observing under the existing isIdlingOrRunning() - is a
decision about what the diagnostic is for, not a mechanical fix, so it is left for you.

Checklist

• Docs updated - docs/inflight/bug-119-load-gate-counts-blocked-work-as-available.md (the
  "Make the latch loud" bullet is now done and cites this PR; the note stays open on
  confluentinc#310: Add a dead letter queue (DQL) implementation #149), and docs/refactoring.md's non-volatile ledger
• User-facing feature documentation data added under docs/features/ - not a new file. This is a
  new operator-visible surface on an existing capability, so the record that owns it,
  docs/features/backpressure-and-broker-liveness.yaml, gains a boundaries entry naming the WARN
  and the INFO. docs/features/README.md says a feature page is for a new capability, not for every
  change to one, and a second file would have split the same capability across two records.
• Tests added/updated - three in WorkManagerTest: the report fires once at WARN with the
  operands, a loaded instance whose records retire never trips it, and the recovery INFO fires once
  and re-arms the WARN
• docs/inflight/ working note (pr-/branch-) started at the PR's first commit - N/A - the
  work's note already exists and owns this item: bug-119-load-gate-counts-blocked-work-as-available.md,
  on master since test(soak) astubbs#119: any retry-forever instance eventually stops fetching, at a threshold you can compute #487. A pr- note would be a second copy of it. The
  directory's other obligation is met: docs/inflight/issue-response-119.md carries the unposted
  draft response, scoped to append to the mirror's Fork status rather than replace it, since
  confluentinc#857: Paused consumption across multiple consumers #119 mirrors a whole family and stays open on the original deadlock.
• Title & body reflect the final content of this PR
• Ran ce-simplify and ce-code-review locally. ce-simplify: one reuse finding (the operand
  tail is spelled out in each log statement rather than shared) - rejected, because collapsing it
  would trade per-field SLF4J arguments for one opaque token and make each line unsearchable in
  source; zero quality findings, zero efficiency findings (one gate read per pass confirmed
  unchanged, log arguments confirmed unevaluated when they do not fire). ce-code-review:
  correctness clean; testing found a mutant that survived all three tests (deleting the count reset
  from the recovery branch) - fixed, with the sabotage arm run; reliability found the false
  all-clear on a revocation - fixed; adversarial found the threshold's healthy bound is the
  commit interval rather than five seconds - javadoc corrected, trigger deliberately unchanged,
  and that the diagnostic is silent outside RUNNING, which is a design call left for the owner and
  described below.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Xoi3HYae8pjsEatuNFKieD

[github-actions]

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None

[github-actions]

✅ Duplicate Code Report

Two engines run in parallel for cross-validation. Each has its own thresholds tuned to its baseline - the real safety net is the per-engine "max increase vs base" check.

✅ PMD CPD

	PR	Base	Change
Clones	27	27	➖ 0
Duplicated lines	949	949	➖ 0
Duplication	0.37%	0.37%	➖ 0

Rule	Limit	Status
Max duplication	0.5%	✅ Pass (0.37%)
Max increase vs base	+0.1%	✅ Pass (+0.00%)

No new clones introduced by this PR.

✅ jscpd (language-agnostic)

	PR	Base	Change
Clones	106	106	➖ 0
Duplicated lines	1503	1503	➖ 0
Duplication	0.82%	0.82%	➖ 0

Rule	Limit	Status
Max duplication	2%	✅ Pass (0.82%)
Max increase vs base	+0.1%	✅ Pass (+0.00%)

No new clones introduced by this PR.

_{Powered by astubbs/duplicate-code-cross-check}

[astubbs]

@codex review this

[codecov]

Codecov Report

❌ Patch coverage is 94.87179% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.26%. Comparing base (e8bd2cb) to head (0bc669e).
⚠️ Report is 1 commits behind head on master.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
...va/bz/stub/parallelconsumer/state/WorkManager.java	94.28%	2 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master     #497      +/-   ##
============================================
+ Coverage     82.67%   83.26%   +0.59%     
- Complexity     1586     1604      +18     
============================================
  Files            96       96              
  Lines          5444     5475      +31     
  Branches        549      554       +5     
============================================
+ Hits           4501     4559      +58     
+ Misses          746      723      -23     
+ Partials        197      193       -4     

Flag	Coverage Δ
chaos	53.20% <56.41%> (?)
integration	62.88% <58.97%> (-0.07%)	⬇️
performance	47.54% <56.41%> (?)
unit	79.76% <92.30%> (+0.35%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

🟢 Throughput — OK

This branch measured about 1% faster than master, on the one test this measures. That is INSIDE this test's own run-to-run spread of about 17%, so read it as a reading and not as a result - re-running the same commit moves it by about as much.

What	Value	Meaning
Compared with master	1.010	above 1.00 is faster, below is slower
Subject test took	61.1s	the test under measurement
Control tests took	33.9s	the other tests in this same run
Subject ÷ controls, here	1.800	not a speed - a shape that cancels machine speed
Subject ÷ controls, master	1.817	median of recent master runs
Reported rate	63096 rec/s	this machine only; not comparable across runners

Allowable range 🟢 ≥ 0.70 · 🟡 0.50–0.70 (about a 30% loss) · 🔴 < 0.50 (about a 50% loss)

What the numbers mean, and what they cannot tell you

The one that gets misread. Subject ÷ controls is a shape, not a speed. 1.7 means the subject took 1.7 times as long as the control tests in the same run - it says nothing about master on its own, and a reviewer has already read it as "1.7x faster than master". Only Compared with master answers that question.

Why a shape and not a rate. A rate depends on which runner you drew. A shape does not: every test here processes a fixed number of records, so a runner twice as slow doubles the subject and the controls together and leaves their ratio alone. That is the whole trick, and it is why the reported rate is shown last and labelled as this machine only.

Reading the comparison. master ÷ this run. Above 1.00 the subject is proportionally quicker here than on master; below 1.00 it is slower. 0.50 means it takes twice as long relative to its controls - that is the failing bound, not a small one.

By conservation, not by correction. Every test in this lane processes a fixed number of records, so within one run the ratio of one test's time to another's is invariant under machine speed — a runner twice as slow doubles both terms and leaves the ratio alone. There is no machine-index correction to be wrong, because nothing needed correcting. share = subjectSeconds / controlSeconds, both from this same run.

Per-method times, not class times. A class time is work + setup, and container startup and @BeforeAll do not scale with work — they are the non-conserved term, and leaving them in breaks the invariant.

Reference is the median of 10 recent perf baseline (master) run(s), read from their artifacts. There is no committed baseline to go stale, and a share is dimensionless, so an old entry stays comparable to a new one without re-baselining. Shares observed: 1.613 – 1.915.

What this still cannot do. It removes machine-to-machine variance. It does not remove this test's own run-to-run variance, measured at about 30% on a single unchanged commit while its controls stayed within 5%. That is a property of the test, not of the comparison, and no arithmetic here can touch it — which is why the reference is a median and the bounds are deliberately coarse. 🟡 means look at this; only 🔴 is outside the measured spread.

Runs used: a37d148, e8bd2cb, 1743297, 4bc6e7a, b62c310, c381310, c79424a, 9c67c89, f1aa5eb, eb9fdb0

Since the previous push: ratio 1.028 -> 1.01, share 1.799 -> 1.8, rate 74775 -> 63096 (-15.6%). One push of difference sits inside this test's measured spread - read it as movement, not as a result.

_{Updated for 0bc669e · run 34315506925 · 2026-09-09 05:44 UTC}

[github-actions]

⚠️ SpotBugs Report

354 bug(s) found (rule-level exclusions only - see docs/inflight/static-spotbugs-rule-registry.md). See the annotations on the Files Changed tab for details.

_{Updated for 0bc669e · run 34315506925 · 2026-09-09 05:45 UTC}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 918bc4c313

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[astubbs]

@claude review this

[astubbs]

lgtm

[claude]

Claude finished @astubbs's task in 2m 45s —— View job

---

Review

• Gather context (PR body, prior Codex review + astubbs' responses, changed files)
• Read the current diff / files as they stand at 7700d8a
• Review WorkManager.java changes (gate reading, latch observation, thread-confinement argument)
• Review AbstractParallelEoSStreamProcessor.java / BrokerPollSystem.java changes
• Review test coverage in WorkManagerTest.java and the soak IT
• Post final review

I read this against the actual current tree (not just the diff shown to Codex against 918bc4c31), since three commits landed after that review to address its findings.

No new issues found. This is a clean, well-scoped mitigation. Specifics:

• "One reading, two consumers" claim holds up. readLoadGate() (WorkManager.java:429) is called exactly once per pass via isSufficientlyLoadedReportingLatch (WorkManager.java:416), and its LoadGateReading is passed to both the return value and observeLoadGateLatch. There's no second shard read anywhere in the new code, so the WARN really can't print an equation the wakeup decision wasn't made on.
• The two-statement clear is correctly distinct and tested. observeLoadGateLatch (WorkManager.java:526-570) branches on reading.isLoaded() inside the latchReported clear path, giving "gate no longer loaded" vs. "something retired but still loaded" separate log lines. WorkManagerTest.theIntakeLatchLogsItsRecoveryAtInfoAndReArms (WorkManagerTest.java:1121-1159) exercises both, including the case Codex's review flagged (399 held against a threshold of 2) — this is a real regression test, not just a happy-path one.
• All four prior Codex findings are genuinely resolved, not just marked so:
  • LoadGateReading is now private (WorkManager.java:452), confirmed not public.
  • The docs/refactoring.md ledger entry (line 467-473) is now a pointer to the javadoc, not a restatement of it.
  • docs/inflight/issue-response-119.md exists, scoped as an addition to the mirror's ## Fork status rather than a replacement, and left unposted per the family-mirror reasoning.
  • The "observe before drain" concern is answered with a dated cleared-suspicion block in the javadoc (WorkManager.java:502-515) that's actually correct: the observation point is fixed once per pass, so the window between two readings is one full pass including the drain — no retirement can fall in a gap. The honestly-stated residual cost (one pass of latency, self-correcting) is right, and moving the read would reintroduce the exact double-shard-read problem the whole design avoids.
• Confinement argument checks out against real test behavior. The claim that controlLoop is driven from more than one thread in ProducerManagerTest via BlockedThreadAsserter is the stated reason for skipping @ThreadConfined + assertion here; given that, plain fields for a diagnostic-only, at-most-one-WARN-or-INFO-off cost is a reasonable tradeoff, and it's now recorded in one place only (observeLoadGateLatch's javadoc) with the refactoring ledger reduced to a pointer.
• isSubscriptionsPausedForBackPressure()/getPausedPartitionCountForBackPressure() split is a genuine no-op refactor (BrokerPollSystem.java:91-102): the former is now defined in terms of the latter (> 0), and maybeWakeupPoller's pausedPartitions > 0 check is equivalent to the old boolean call, reading the same cache.
• Pass-count rationale (100) is sound and asymmetric in the right direction — tied to the two real cadences (mailbox-driven vs. commit-interval-bounded blocking) rather than a wall-clock guess, avoiding the timing-bound trap the PR cites.
• Tests are real, not just present. The suite includes a negative control (aLoadedInstanceWhoseRecordsRetireNeverTripsTheIntakeLatch, 300 passes with retirement, no WARN) alongside the positive cases, and the PR body documents red-first sabotage on both clauses (report removed, "nothing retired" clause dropped) — the kind of check docs/testing-at-write-time.md asks for.

Nothing to add beyond what Codex and astubbs already resolved in review. This looks ready from a correctness standpoint, modulo the still-open checklist item (ce-simplify/ce-code-review not run locally, explained as N/A since the PR-hosted review covered that ground).

[astubbs]

@claude review this

Context for the pass, so it does not re-derive what has already been settled.

This adds one observation and one log line to the record-intake load gate: a WARN when the gate has read loaded across LATCHED_PASSES_BEFORE_WARNING consecutive control-loop passes with no record leaving the shards, and a line when that clears. The claim is that it changes no semantics.

Already reviewed and addressed, so please spend the pass elsewhere unless you think one of these is wrong:

• Codex reviewed 918bc4c31: five findings, four taken, one rejected (moving the observation after the mailbox drain - the window between two readings already spans a whole pass, drain included). Every thread has an in-thread reply and is resolved.
• A local ce-simplify pass returned one reuse finding, rejected with reasons in the body.
• A local ce-code-review pass found three real things, all now fixed in 209316904: a clear line that would have read as an all-clear after a revocation with nothing having succeeded; a mutant that survived all three tests (deleting the count reset from the recovery branch); and a wrong bound in the threshold's own derivation - the healthy grace is the commit interval, which is 100ms by default under transactional mode, not five seconds.

Two things are deliberately open and named in the body rather than fixed, and a second opinion on either is worth more than a fresh sweep:

1. The trigger is not narrowed on parkedForRetry, which would separate a latched instance from a merely-slow one. The report prints that operand so a human can separate them; narrowing what fires wants its own measured arm, because a transient zero would suppress a real latch.
2. The observation only runs while the state is RUNNING, so a graceful shutdown that hangs in DRAINING - a plausible shape of this defect - is never reported.

Worth attacking hardest: whether the pass count can false-alarm on any healthy configuration I have not named, and whether the two clear branches are exhaustive and correctly assigned.

[claude]

Claude finished @astubbs's task in 5m 39s —— View job

---

Review

• Re-read PR body, prior Codex review, astubbs' in-thread replies, and the earlier @claude review from this morning
• Confirmed the four already-addressed Codex findings are still in the tree at 2093169 (private LoadGateReading, docs/refactoring.md reduced to a pointer, issue-response-119.md present, the two-statement clear with the "by any route" caveat)
• Traced maybeWakeupPoller → controlLoop ordering to check the "observed before mailbox drain" cleared suspicion
• Attacked the pass-count bound for an undocumented false-alarm window
• Checked the two clear branches for exhaustiveness
• Spot-checked BrokerPollSystem.java no-op refactor and WorkManagerTest.java coverage

Not re-litigating the settled items (Codex's five, ce-code-review's three, the transactional-commit-interval window already in the javadoc and the inflight note) — they check out against the current tree.

The two clear branches: exhaustive and correctly assigned

observeLoadGateLatch (WorkManager.java:568-616) enters the else whenever !(reading.isLoaded() && !somethingRetired), i.e. !isLoaded() || somethingRetired. Inside it, the split is on reading.isLoaded() alone:

• isLoaded() true → "still gated, but something retired" (case: isLoaded && somethingRetired)
• isLoaded() false → "gate no longer loaded" (case: covers both !isLoaded && somethingRetired and !isLoaded && !somethingRetired)

That's exhaustive over the boolean pair, and the assignment is right: real recovery is defined purely by !isLoaded(), so folding the !isLoaded() && !somethingRetired sub-case into the "resumed" branch is correct — retirement isn't a precondition for the gate having genuinely opened. theIntakeLatchLogsItsRecoveryAtInfoAndReArms (WorkManagerTest.java:1121) exercises both branches distinctly (399-held/threshold-2 for the first, a full drain for the second), so this isn't just true by inspection, it's pinned.

A false-alarm window the javadoc doesn't name, alongside the transactional one

LATCHED_PASSES_BEFORE_WARNING's derivation (WorkManager.java:77-88) rests on: healthy passes block for getTimeToBlockFor(), "bounded above by the commit interval." That's true along one branch of getTimeToBlockFor() (AbstractParallelEoSStreamProcessor.java:2350-2372) — the one taken when wm.isWorkInFlightMeetingTarget() is true. There's a second branch, taken whenever it's false:

if (!wm.isWorkInFlightMeetingTarget()) {
    var lowestScheduledOpt = wm.getLowestRetryTime();
    if (lowestScheduledOpt.isPresent()) {
        Duration effectiveRetryDelay = max(getDefaultMessageRetryDelay() [1s default], lowestScheduled);
        return min(getTimeBetweenCommits(), effectiveRetryDelay);
    }
}

isWorkInFlightMeetingTarget() is numberRecordsOutForProcessing() >= maxConcurrency * batchSize (WorkManager.java:657-659, ParallelConsumerOptions.java:524-526) — false whenever dispatch is below full concurrency, which is the ordinary state under KEY/PARTITION ordering whenever fewer distinct keys/partitions are active than maxConcurrency allows, not a misconfiguration. If the gate still reads loaded (a large backlog held in shards, which is exactly #487's "eleven of fourteen workers idle" shape) and any single record anywhere is in retry back-off, the block is bounded by max(1s default retry delay, lowestScheduled) instead of the commit interval — independent of PERIODIC_TRANSACTIONAL_PRODUCER.

Net effect: a healthy, ordered, backlog-holding instance with one ordinary transient retry pending can run its passes at roughly the 1-second retry-delay cadence rather than the assumed 5-second commit-interval cadence, collapsing the ~50x healthy grace to roughly ~10x — well short of "eight minutes," on stock defaults, no transactional mode required. It's a narrower and cheaper-to-trigger version of the already-documented collapse, via a mechanism the javadoc and docs/inflight/bug-119-load-gate-counts-blocked-work-as-available.md's "Open" section don't name (both attribute the only known collapse to the transactional commit interval).

I haven't run this as an experiment — it's a static trace of getTimeToBlockFor()'s two branches, not a measured arm, so treat it as worth a calibration run rather than a confirmed defect. If it holds up, it belongs in the same javadoc paragraph and the same inflight note as the transactional case, next to parkedForRetry as the discriminator (which still works here — a merely-slow instance reads parkedForRetry=0, the same escape hatch already relied on for the transactional window).

Everything else

maybeWakeupPoller's single-reading claim, the BrokerPollSystem no-op split, and the test suite's red-first coverage all hold up against the tree at 2093169 — nothing new to add there.

[astubbs]

Taken, in 531699f1d - and it is the better half of a finding I only got halfway. Thank you for going at the branch I did not read.

I had corrected the derivation for the commit-interval branch and stopped there, which left the paragraph still implying that the commit interval is the healthy bound. It is not: getTimeToBlockFor() has two, and the second needs no unusual configuration at all. Verified the trace against the source rather than taking it on the report:

• isWorkInFlightMeetingTarget() is getNumberRecordsOutForProcessing() >= getTargetAmountOfRecordsInFlight(), and the target is maxConcurrency * batchSize - so it is false whenever dispatch sits below full concurrency, which is the ordinary state under KEY/PARTITION ordering with fewer active keys than maxConcurrency allows. Not a misconfiguration, and it is exactly the shape test(soak) astubbs#119: any retry-forever instance eventually stops fetching, at a threshold you can compute #487's low-poison arm measured.
• DEFAULT_STATIC_RETRY_DELAY is one second.

So with one record in back-off the pass blocks for min(commitInterval, max(1s, lowestScheduled)) - a one-second cadence, not five - and the grace is roughly a hundred seconds rather than eight minutes, on stock defaults with no transactional mode involved. The eight minutes is the best case, not the bound, and both windows are now named on LATCHED_PASSES_BEFORE_WARNING and in docs/inflight/bug-119-load-gate-counts-blocked-work-as-available.md.

Carried over your framing that this is a static trace of the two branches and not a measured arm, in both places, because that is the bar every other claim about this mechanism was held to - a calibration run comes before anything is decided on it. The discriminator is unchanged and still applies to both windows: the report prints parkedForRetry, which this state holds continuously and a merely-slow instance reads as zero.

On the two branches being exhaustive and correctly assigned - agreed, and your reading of why !isLoaded && !somethingRetired belongs with "resumed" is the one I intended: recovery is defined by the gate having opened, and retirement is not a precondition for that.