channels_sv2: make the past-jobs cap configurable w/ a 'good' default value

[gimballock]

MAX_PAST_JOBS is really a retention window — cap ÷ template rate — and the template rate is a deployment property the library can't see. The same constant buys 50s of late-share tolerance at a 1s template cadence and over 16 minutes at 20s, so I don't think one number can be right for everyone.

Adds max_past_jobs: Option<usize> to the server and client ExtendedChannel/StandardChannel constructors. None and Some(0) both select MAX_PAST_JOBS, so nothing changes for existing callers.

Treating Some(0) as "no opinion" rather than as a literal zero keeps a zero cap unreachable: it would evict the job that just retired, so the most common late share would come back InvalidJobId with no startup error to warn you.

Group channels take no parameter — client GroupChannel has no past_jobs, and server group channels use replace_active_job.

A second commit will set the default once we've measured it; the analysis is coming in a comment.

Follow-up to #2290.

companion stratum-mining/sv2-apps#735

[gimballock]

Now that the cap (MAX_PAST_JOBS) is configurable, here's what it should default to — and why our data doesn't yet license moving it off 50.

Two losses, opposite fixes. An uncredited share is either an eviction (invalid-job-id, within a tip) or a cross-tip loss (stale-share, at a prev-hash rotation). If it's eviction, past_jobs must stay deep and the memory is unavoidable. If it's cross-tip loss, past_jobs can be tiny and the stale window sized on its own. Everything below serves one question: which one are we seeing?

Why our data can't decide

The costs are the ones already in #2290. On the memory side, at fixed connection count, cap 300 → 50 cut per-connection memory 36.5% (1.429 → 0.907 MiB, ~4.5 kB per retained job per channel). Both arms carried the same connection count within noise, so the cap moves density, not capacity — it buys headroom on the box rather than raising the connection ceiling. One caveat on that figure: per-connection memory isn't comparable across differing hardware, connection counts or trial lengths, and we published a 45.5% version of it that was wrong for exactly that reason. 36.5% is the corrected same-conditions result.

On the yield side, that same change cost 2.7 points on a channel-level metric we track — uncredited channels: those that opened successfully but never had a single share credited, 3.16% → 5.86%. Those channels stayed connected for the whole trial, so they were paying full per-connection memory while producing nothing creditable — a yield loss, not a connection loss. Both figures come from a gauge we've seen undercount under scrape pressure, so the 2.7-point delta is sturdier than either absolute.

Theory says the eviction path should be nearly empty. The required depth — how many past jobs the cap has to retain — is 1 + ceil(latency ÷ template period), which is 1 at the shipped min_interval = 5, and a code-level argument lands on the same 1–2 independently. But we can't attribute those 2.7 points of uncredited channels either way. Every simulated miner submitted one share seconds after connecting and then stopped, so no share was ever old enough to test the required depth — our invalid-job-id ≈ 0 is vacuous, not reassuring. That metric and stale-share rose together going 300 → 50, but under a one-shot workload that's correlation, not cause.

The experiment that decides

That uncredited-channel metric can't be reused: it only means anything because each channel submitted exactly once, and once miners resubmit after a rejection every channel eventually lands an accepted share, so the count collapses toward zero however much work is being lost. So this experiment needs to measure an actual stream of shares — rejected ÷ submitted, split by error code — under continuous hashing at fixed connection count. Those two counters are the discriminator, and previous runs simply never put load on them.

arm	caps	purpose
rare-tip	1, 2, 4, 50	eviction alone — locates the knee, the cap below which invalid-job-id starts climbing
rotating-tip	1, 2	adds the cross-tip path

Six runs. Without rotations, mark_past_jobs_as_stale never fires, so the rare-tip arm has no cross-tip path by construction and its losses are eviction only. The rotating arm has both. That gives two readings of the same question: the error-code split says which mechanism, and the rotating-minus-rare rejection rate at equal cap says how much is cross-tip loss — eviction appears in both arms and cancels.

That subtraction assumes eviction is equal across arms, which is close but not exact: each rotation drains past into stale, so the ring briefly can't evict while it refills, and the rotating arm suppresses eviction slightly. The bias runs against attributing loss to the past/stale coupling, so a signal that survives it is the stronger result.

The ladder is deliberately low. Eviction can only fire when a share arrives more than cap × template_period after retirement — 50 seconds of latency at cap 50 and a 1 s cadence, which is unreachable — so a ladder starting at 8 would report invalid-job-id ≈ 0 at every rung and prove nothing. Caps 1 and 2 are where the model says eviction must start biting, and they're the caps worth shipping; 50 stays as the tie-back to existing data. Low caps fill in seconds, so these are the cheap arms. Rotation rate gets reported alongside every stale-share figure, since rotating faster than mainnet inflates cross-tip loss.

Predictions, registered before the run:

if the loss is…	invalid-job-id	stale-share	rotation sensitivity	implication
eviction	rises as cap falls, knee near 1 + ceil(L/T)	flat	low	past_jobs stays deep; memory unavoidable
cross-tip loss	flat	rises as cap falls	high	past_jobs → ~2; size the stale window separately
neither	flat	flat	—	the cap isn't the cause; the default can drop to the required depth

The third row is a real outcome, named in advance so it isn't treated as a failed run: rejection rates flat across the whole ladder would mean the earlier 2.7 points came from the one-shot workload rather than from the cap, and that nothing on the yield side argues against a small default.

Recommendation: keep the default at 50 until this runs. It's the safer side of a trade-off whose downside is unquantified below it. We wouldn't argue against a smaller cap on the eviction path; we can't vouch for it on the cross-tip path.

When it does run, the knee is a floor rather than a default. It's measured against one latency distribution and one cadence, so a shipped default wants headroom above it — and since the knee moves with both, the parameter this PR adds probably matters more than whichever number we end up recommending.

Contributions:

• Measured job retention at ~4.5 kB per retained job per channel, the largest per-connection memory lever we've found.
• Derived the required depth as 1 + ceil(latency ÷ template period), showing depth is set by latency rather than share rate or connection count.
• Implicated the past/stale coupling rather than eviction, by splitting rejections per error code — a lead the experiment above is built to confirm or kill.
• Specified a six-run experiment that separates the two loss paths, with predictions registered in advance.

[gimballock]

This work is progressing, I have a solid theory of what to expect from the test. I have a physical mining test running now to validate against expectation. In short I think our setting of 50 is way too high and the better value is closer to 2 (as a conservative default value).

A lot of the analysis yesterday went into deciding if the stale share buffer and the past jobs buffer should be split into two and sized independently but eventually I realized that at realistic rates on mainnet the two buffers should be small and in approximately equal proportion.

I'll post more today once the hw results are in.

and i'm trying to get just enough of the analysis together to be informative w/o overwhelming.

[gimballock]

Hardware A/B for the default value — now running

Following up on the plan above: the run that decides the default just started on real
mining hardware. Results in a few hours.

The question

How many past jobs must a pool retain per channel? The channels_sv2 default is 50;
#2290 raised whether that's 25× larger than necessary. The cap costs ~4.5 kB per retained
job per channel, so at scale it's real memory. Retaining too few loses creditable
work: a late share whose job has been evicted is rejected invalid-job-id even though it
was valid.

The design

Two pools on one host, sharing one Bitcoin node, identical in every setting except the
cap:

	Arm A	Arm B
max_past_jobs	2	50
miner	same model, ~215 TH/s	same model, ~245 TH/s
firmware	cgminer-derived (bmminer lineage)	byte-identical build
shares/min, batch size, cadence	600, 10, 6.0 s	same

Each arm: one real SV1 miner → its own translator → its own pool. Same chain tip, same
mempool, same template timing. Both pools attest their cap in their startup log, so an arm
silently running the wrong value would be caught.

Network: testnet4, templates driven by the node over IPC with the fee threshold at 0 and a
5 s minimum interval, yielding a measured 6.0 s template cadence in both arms.

The mechanism under test

cap 2 retains:   1 active job + 2 past  =  3 creditable jobs
cgminer caches:                            3 jobs

Cap 2 exactly covers a 3-job cache, with zero margin. That's the substance of the
"floor is 2" claim. The cap is really a retention window — cap ÷ template rate — so at
6.0 s cadence arm A tolerates 12 s of staleness and arm B tolerates 300 s. Anything arm A
loses is a share delayed past its cache depth by latency.

This is why real miners matter: a simulated miner that only ever submits against the
current job can never exercise the cache, which is why earlier attempts measured nothing.

What we measure

Pool-side invalid-job-id per arm, as a rate per submitted share. At ~600 shares/min/arm:

Elapsed	Shares/arm	Detects ≥	If zero, 95% bound
2 h	72,000	0.007%	< 0.004%
4 h	144,000	0.0035%	< 0.002%

Reference point: the same units' long-run stale-share rate against their normal production
pool is ~0.145%, so 4 h resolves roughly 70× below it.

What each outcome means

• Zero in both — the expected result. Cap 2 suffices at this cadence, and the bound is
  the deliverable: "0 in 144,000 shares, loss < 0.002%." Supports lowering the default.
• Arm A > arm B — cap 2 loses creditable work; the floor is above 2, and the size of
  the gap gives the required depth. This would contradict the current recommendation.
• Both nonzero and equal — not cap-related; something else is rejecting shares, and the
  run says nothing about retention.

What it cannot tell us

One firmware build, one template cadence, testnet4, two miners. It does not establish
behaviour across other firmware, nor at production template rates — we deliberately run
~10× a typical share rate, which stresses job age harder than production would. That
asymmetry makes a null result stronger, and means a positive result would need a confirming
run at a normal share rate to rule out a queuing artifact.

Two known asymmetries, neither affecting job-cache behaviour: the miners sit in different
power modes (pre-existing, hence the hashrate difference), and one arm has a head start in
accumulated shares — so results will be reported as rates over the overlapping window, not
as raw totals.

---

🤖 Generated with Claude Code

[gimballock]

There are many details left out of that relating to why i think 2 is a good value but I can explain all of that if the test affirms that hypothesis, otherwise you probably don't care about my false logic.

But part of the estimate is based on our belief of how many jobs a miner can be working on in parallel.
This is partly answered in [SV2 spec §5.3.15] "this mining job is active and miner must start mining on it immediately", and anecdotally I was told by one of our firmware engineers that cgminer will cache the most recent 3 jobs. I think that modern firmwares prefer to stick to one job at a time but I don't really know. I think cgminer probably covers a big portion of what we will see but since spec says you should always be working on the newest job then we don't need to honor these older submissions from weird firmwares (probably old sv1)