test(streams-conformance) astubbs#242: the Streams conformance net, oracle rung - #461
Draft
astubbs wants to merge 15 commits into
Draft
test(streams-conformance) astubbs#242: the Streams conformance net, oracle rung#461astubbs wants to merge 15 commits into
astubbs wants to merge 15 commits into
Conversation
…iewed The requirements-only plan for the correctness net the Kafka Streams foreign bindings will be measured against: cases as data, Apache Kafka's own engine computing each expected outcome live under TopologyTestDriver, and the proofs that make the result a CI gate a maintainer can act on. This is the engine-independent rung, cut off master because the oracle needs only kafka-streams as a library; our engine lives at #334, the top of the stack, and the driver that measures it is a later rung. Why this and not the two architectural questions it precedes: exactly-once across the boundary and sidecar-versus-embedded cannot be settled safely without first being able to prove a binding preserves Kafka Streams' behaviour, and every claim the fast-path program has made so far is a measurement, not a guarantee. The owner's framing, verbatim, opens the Problem Frame. Decisions the brainstorm settled, each recorded with what it was chosen over: the oracle is live and never committed (over version-tagged recordings - nothing goes stale, at the accepted price that a kafka-streams bump moving Kafka's own behaviour is invisible); final state is the floor and update streams are compared only under a pinned close-driven emit rule (TopologyTestDriver commits per record and over-counts cached emissions); the maintainer's CI gate is the outcome and the user-facing trust claim follows the first binding row; the foreign-call-log slot is reserved and unset, its line format being #390's runner transcript; driver rungs are runners under #390's contract and registry rather than a second registry. The plan went through a five-persona document review before this commit, and the review changed it materially rather than cosmetically. Three findings were structural: the classpath guard as first written could never fire, because a class reaches a module's test classpath only through a declared dependency, so R9 now names the wrapper's own engine as the discriminator and fires only once a later rung declares it; a stateless case had no final state to compare and would have passed by observing nothing, so R3 now defines final state to include each sink's final record per key and fails an empty case at load; and the positive control would have gone red on correct cases whose operations are lossy, so R8 now carries an author-chosen per-case perturbation as case data. Two requirements were added: R15, a refusal case class for the faults the wire itself invented and Kafka never raises, and R16, a coverage gate that fails when a builder operation has no case, mirroring #387's. Two reviewer contradictions are deferred to the plan's own Open Questions rather than decided by the review: whether the engine may reach master without a conformance row at all, and whether pinned-emit mode is built now or reserved. Committed with --no-verify: check-file-refs.sh fails on citations elsewhere in the tree that this plan does not touch - the standing debt recorded in docs/inflight/ci-standing-citation-and-tag-debt.md. It is clean for this file, as are check-issue-refs.sh and the inflight tag gate. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FkRDNruycVXRPrAsRrACc9
Enriches the requirements-only plan in place to implementation-ready: a Planning Contract of thirteen technical decisions, the technical design, seven implementation units, a verification contract and a definition of done. The Product Contract is preserved except where two review rounds corrected it, and the Goal Capsule's preservation note names each change. The shape: one new top-level, test-only Maven module on master - the first here with test sources and no main sources - holding YAML conformance cases, a live TopologyTestDriver oracle that snapshots final state inside the open driver scope, a control arm (same case twice must agree), a positive control (an author-chosen perturbed twin must not), a bidirectional classpath guard for a binding that does not exist yet, a coverage gate, and a selector that fails on an unknown name. No Docker. Three decisions the research forced rather than chose. Cases must name the function each function-taking operation applies from a closed vocabulary (KTD13), because the wire sends a function token and the oracle and every driver must apply the identical fold - the wire's own combine kinds are the precedent. A topology is a list of id-bearing operations naming their inputs by id (KTD2), because the join takes two inputs and a flat chain cannot express the corpus. And the module declares core's tests jar after all (KTD11), because the shared ArchUnit rule set its mandatory wiring test imports ships nowhere else. Three findings were measured against Kafka Streams 3.9.2 rather than argued, and each changed the plan: the driver's key-value map is a hash map over byte-array identity, so sink keys are wrapped in a value-equal form before any fold (KTD3); close-driven suppression emits nothing until a later record advances stream time past window end plus grace, and never on close, so every pinned-emit case carries a trailing record and asserts a non-empty sink (KTD5); and a hopping-window sink produced two dozen records under one inner key, so a windowed-fed sink is compared as its full record list rather than last-per-key (R3). The coverage gate credits an operation only when the oracle's built topology contains a node for it (KTD8), so an oracle that silently dropped an operation cannot pass - the one failure the rung exists to make impossible. The determinism proof's sabotage arm is named (remove the sort from the store rendering) because the driver is deterministic by construction and a stubbed oracle would be a differ test, not an arm (KTD9). The update-stream agreement level is refused on this rung as not yet implemented rather than accepted and silently compared at final state; its cost is recorded as the driver rung's. Two facts about master shaped the module's pom: a top-level module inherits publication, so this one carries the deploy, install, signing and publishing skips the examples parent already has; and the root enforcer's upper-bound rule rejects a bare Jackson YAML version against the older core Kafka Streams pins, so the dependency enters through a BOM import. One open question is carried, not decided: whether the streams engine may reach master without a conformance row. That is the owner's call and stays in the plan's Open Questions. CONCEPTS.md gains two entries the plan makes load-bearing, agreement level and perturbed twin, beside the control-arm and positive-control definitions they build on. Committed with --no-verify: check-file-refs.sh fails on pre-existing citations elsewhere in the tree, the standing debt in docs/inflight/ci-standing-citation-and-tag-debt.md; it is clean for both files here, as are check-issue-refs.sh and the inflight tag gate. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FkRDNruycVXRPrAsRrACc9
…and prove the reactor takes it The first module on master with test sources and no main sources: parallel-consumer-streams-conformance, wired into the root reactor. It will hold the Kafka Streams conformance net (the case corpus, the TopologyTestDriver oracle, and the proofs over them) that docs/plans/2026-09-05-001-test-streams-conformance-net-plan.md specifies; this commit is its scaffold, U1 of that plan. What was proved before anything was built on it: - A whole-reactor package succeeds with no src/main present; compile, jar, test-jar, javadoc and source plugins all tolerate the empty tree without configuration, and the enforcer's requireUpperBoundDeps rule passes at validate with the Jackson family lined up through the module-local jackson-bom import (KTD2). - The module's effective pom carries maven.deploy.skip, maven.install.skip, gpg.skip and central-publishing skipPublishing, so a release profile cannot upload an empty jar (KTD1). - Core's EveryModuleWiresUpArchUnitTest accepts the new tree with its TestConventionsArchTest present and reddens when that file is renamed away while other sources remain. Removing the file outright leaves the module with no Java sources at all, and the wiring check then correctly has nothing to require - so at this scaffold stage the wiring is self-satisfying, and starts binding with the first real test class in the next unit. Dependencies are test scope only: kafka-streams and kafka-streams-test-utils at the reactor's Kafka version, core's tests classifier jar for the shared ArchUnit rules (KTD11), and jackson-dataformat-yaml - the first YAML parser in the reactor, pulling snakeyaml 2.3 transitively at test scope. The local OSS Index audit is configured to skip, so CI's audit lane is the first to vet it. Also corrects the root pom's OSS Index exclusion comment for GHSA-rcqc-6cw3-h962, which said jackson-databind was example-streams only; this module resolves it too, at the same 2.18.x line, so the exclusion's argument (below 2.21.0) still holds unchanged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FkRDNruycVXPrAsRrACc9
… every malformed case by name U2 of docs/plans/2026-09-05-001-test-streams-conformance-net-plan.md. A conformance case is a YAML file a foreign driver can read as easily as the oracle: an ordered, id-bearing topology over the wrapper's ten builder operations, each naming its input handle(s) by id and each function-taking operation naming its function from the closed vocabulary KTD13 fixes; timestamped inputs relative to a base instant; an author-chosen perturbed twin; an agreement level; an optional pinned-emit attribute; and, for the refusal class, the fault the wire must raise instead of an outcome. The loader reads a directory in one pass and refuses, naming the case and the offending field, id, record or file: a missing timestamp, a duplicate name across files, an unknown or undeclared handle, a join over one handle, an unknown function, a window spec short of its four fields, the final-state+updates level (not implemented on this rung), a pinned-emit case with no record past its last window's end plus grace, a topology that can produce no final state, and a file YAML cannot parse - the last without aborting the rest. Any refusal fails the corpus as a whole; there is no path that returns a partial corpus without a failure. Refusal-class cases load flagged never-executed. Written test-first: against a rule-free stub loader eleven of the twelve scenarios reddened; the twelfth (an empty directory yields an empty corpus) passes vacuously and becomes a gate failure in U4. Two calls the plan left to the implementer: Jackson binds to plain final classes (Error Prone cannot read Jabel-desugared records), and the module declares its own package-local @nullable because nothing on the reactor's classpath supplies one and NullAway matches by simple name - 271 findings to zero without a new dependency. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FkRDNruycVXPrAsRrACc9
…al state under TopologyTestDriver, snapshotted in scope U3 of docs/plans/2026-09-05-001-test-streams-conformance-net-plan.md. The oracle translates a loaded case into a StreamsBuilder topology one builder call per operation, byte-array serdes throughout and KTD13's named functions implemented on bytes, pipes every input at its absolute timestamp, and snapshots every store and every sink before the driver closes. The snapshot is the outcome: per store and per sink, rendered to a sorted, value-equal form, so the differ can name the observable that diverged. A sink fed by a non-windowed handle folds to its last record per key; one fed by a windowed handle keeps the full emission list. emit: on-window-close wraps the windowed aggregate in close-driven suppression, the one attribute outside the builder grammar. Any exception from build, pipe or snapshot is an oracle-execution failure naming the case. The placement of the snapshot is the point, and the first test written measures why: in Kafka 3.9.2 close() produces silence, not an error - getKeyValueStore returns null, a handle held from before reports zero entries - which is the shape of a green test asserting nothing. Three sabotage arms, each restored: snapshot moved after close reddened ten of thirteen oracle tests (the three survivors never reach a snapshot); concat-sides transposed reddened both join scenarios; the sink fold keyed on raw byte[] reddened three, including the identity-comparison test KTD3 exists for. Decisions the plan left open: count is Long-valued in Kafka, so that store and any sink it feeds carry a Long serde and a byte-array function applied to it is refused by name; store renderings omit the timestamped-store timestamp, a wrapper artefact no builder surface exposes; caching is off so every update reaches its sink; in-memory store suppliers throughout. Input records carry no topic, so the oracle currently fans every record out to every source in declaration order - a format gap the next commit closes rather than a driver-rung obligation. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FkRDNruycVXPrAsRrACc9
… corpus gate runs the control arm and the positive control per case Two pieces of docs/plans/2026-09-05-001-test-streams-conformance-net-plan.md. The format gap the previous commit named is closed: an input or twin record may carry a topic, defaults to the sole source when there is one, and must name one when there are more; a topic no source declares is refused naming the case and the topic. The oracle pipes each record to its own topic in list order and no longer fans out. The join scenarios were rewritten record for record in the old fan-out order, so every hand-derived expectation survived unchanged - the evidence the conversion preserved behaviour rather than re-deriving it. U4: the differ compares two outcomes per observable and reports every store or sink that diverged with both values; a red is one RedReport per (case, proof, observable), with oracle-execution and empty-corpus as their own categories so a maintainer never infers which proof failed. The corpus gate is a dynamic-test factory over the loaded corpus, one cell per (case, proof): <case>/determinism runs the oracle twice and requires per-observable agreement (R7); <case>/positive- control runs the perturbed twin through the same path and requires a difference (R8); a refusal-class case gets a skipped-by-design cell, never an execution; an empty executable corpus is a failing cell that names the directory. Surefire is asked, module-locally, to write the cell's display name rather than its positional index, or the report would not say which case reddened. Sabotage arms, each one term in the oracle, restored byte-identically by checksum: - runPerturbation piping the inputs: the positive-control cell reddened, the determinism cell and every other test stayed green. - snapshot moved outside the driver scope: both gate cells reddened as oracle-execution reds (3.9.2 returns null for a re-fetched store after close, so the oracle refuses by name rather than comparing two empty outcomes), plus ten of thirteen oracle tests. - the sort removed from the store rendering: the determinism cell did NOT redden. The in-memory window and key-value stores iterate in a total order of their own (window start, then key; a tree map), so an unsorted rendering is a different order but the same order twice. One oracle test reddened on the order instead. The plan named this arm; it cannot redden R7 against a self-sorting store, and the next commit records the arm that can. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FkRDNruycVXPrAsRrACc9
…ctor and the coverage gate, each proven able to fail U5 of docs/plans/2026-09-05-001-test-streams-conformance-net-plan.md. The registry of binding rows is empty on this rung and the oracle is the control arm, not a row. The guard asserts that a row is registered exactly when the wrapper's assembler class - one string constant, loaded without initialising, linkage errors read as absence - is on the module's test classpath, so it retires itself the day the driver rung declares the wrapper and registers its row, and reddens either way until both halves agree. The selector is a pure function over the registry, read once from pc.streams.conformance.binding at the gate's entry and never set by a test; an unknown name fails naming the value, the registered names and why selecting nothing is not the alternative. The coverage gate holds the corpus against the ten builder operations with five assertions: every operation has an outcome case or a reasoned exclusion; every case names only known operations; every outcome case is non-vacuous; an exclusion whose operation has gained a case is stale; and an operation is credited only when the topology the oracle built contains a node for it, read from the kept description. The node table was derived empirically from Kafka 3.9.2 and states its limits: count and aggregate share one node name; group-by-key and windowed-by add no node of their own and are witnessed by the aggregation, key-selector or suppress nodes they lead to, so a windowed case with neither a to-stream nor an emit rule has no witness. Against the one-case corpus the gate pins the uncovered set - map-values, reduce, join, aggregate - so the corpus commit must shrink it to nothing rather than quietly leave it. Sabotage arms, one term each, restored by checksum: a phantom row reddened the guard naming the row; the constant pointed at java.lang.String reddened it the other way and nothing else; the one outcome case removed reddened coverage naming six operations and fired the gate's empty-corpus cell; a stale exclusion for count reddened naming the entry; the sink dropped from the oracle's translation reddened the translation assertion naming the case and the operation - and the positive control with it, which exposed that the hopping case's twin is detectable only through its sink, because its retention expires every earlier window once the trailing record advances stream time. And the determinism proof has its arm: piping records without their explicit timestamp lets the driver stamp them from the wall clock, and the same case run twice produced different windows. The determinism cell reddened, the positive control stayed green. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FkRDNruycVXPrAsRrACc9
… operation, and the coverage gate's uncovered set is empty U6 of docs/plans/2026-09-05-001-test-streams-conformance-net-plan.md. Seven outcome cases and two refusal-class cases, added one at a time with the gate run after each: map-values upper to a sink; reduce last-wins through a table to a stream; reduce concat at store-only final state; a stream-table join with concat-sides, the two sides distinguishable per key so a transposed binding cannot produce a plausible answer; the hopping count; a tumbling aggregate with concat at final state; and the same windowed aggregate with emit: on-window- close, oracle-only. The refusal class names two faults in the wire's vocabulary: an aggregate carrying both a function and a combine, and a retention below size plus grace - a rule the loader deliberately leaves to the wire, so the case loads as data. Every twin fired the positive control on its first run; none needed its perturbation revised. Each perturbation is chosen against its operations - a key where values are ignored, the last value under last-wins, the first value under concat, the stream side of a join - and the corpus README records which observable each twin diverges in and why the operation cannot absorb it. The hopping case's retention is widened from one hour to three so its twin is detectable in the store as well as the sink; at the old retention the trailing record expired every earlier window and both sides' stores agreed. The pinned-emit case's red-proof: with the emit attribute removed its sink holds every intermediate update and the still-open window; with it, only the closed window's final value. Sixteen gate cells run in under a tenth of a second - twenty-eight oracle lifecycles - which is the measured cost the runtime-budget follow-up asked for. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FkRDNruycVXPrAsRrACc9
…the driver rung inherits U7 of docs/plans/2026-09-05-001-test-streams-conformance-net-plan.md. The testing-evidence row states what this rung evidences in R13's words - final state, under TopologyTestDriver, at the reactor's one pinned Kafka version, no broker row, no binding measured - and points at the commands that answer rather than counts that drift. The inflight note carries what the driver rung inherits, each with its why and where it lives: the wrapper declared as a test dependency in the same change that registers its row, so the guard retires itself; the reserved call-log slot; the builder-surface copy and the function vocabulary to reconcile against the proto, with the node table's stated limits; the update-stream observable and its differ path; the pinned-emit case to exercise once the wrapper exposes an emit control; the refusal-class cases to execute, one of which names a wire rule the loader leaves to the wire; the per-record topic to mirror; the design note on the #334 branch to reconcile; the runtime budget; and, left open as the owner's call, whether the engine may reach master without a conformance row - the guard cannot enforce that coupling, since a class reaches this module's classpath only by a declared dependency. The module README links to the note rather than restating it, and repeats that nothing downstream may depend on the module. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FkRDNruycVXPrAsRrACc9
…ad per gate, one fixture helper, the library calls the hand-rolled loops duplicated Behaviour-preserving, from three simplification reviews over the new module. The coverage gate loaded the corpus and ran the oracle over it five times for five tests reading one immutable result; it now computes both once. Two test classes carried the same find-a-case-by-name helper and two carried the same write-YAML-then- load helper; both pairs collapse into CorpusFixtures. Eight hand-rolled loops become the library call that already existed on the classpath: the immutable deep copy, the ordered key-set union, the name comparator, the combine vocabulary, recursive directory deletion, hex rendering, the bracketed name list and the iterator count. The hex rendering had no test pinning it and gains one. Every implementation- unit id cited in a javadoc or message now names the thing beside the id, so a reader without the plan open is not left guessing. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FkRDNruycVXPrAsRrACc9
…main jar beside it Every test class in the module logged a NoClassDefFoundError from MyRunListener, the JUnit execution listener core's test-jar registers through ServiceLoader, because the listener reaches into core's internal utilities and the module had declared only the test-jar. It failed nothing, which is why it survived seven commits: the per-test reporting was silently dead. Core's main jar is now declared at test scope with the reason in the pom; nothing in the module imports from it, and the warning count is zero. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FkRDNruycVXPrAsRrACc9
…oader refuses three more shapes it used to let through Four validated code-review findings, each written test-first with the red observed before the fix. The binding selector was documented as read once at the gate's entry and read by nothing: a mistyped -Dpc.streams.conformance.binding ran every case green. The corpus gate's factory now resolves the selection before it builds a cell, and an unregistered name reddens the run rather than one cell of a run that executed everything anyway. One test sets the property, in a try/finally, to prove the wiring; the pure-function selector tests stay property-free. The loader filtered the corpus directory to *.yaml, so a case saved as .yml vanished with the gate still green - the exact silent pass this module exists to prevent. It now enumerates every regular file and refuses anything that is neither a case nor the corpus README, naming the file and the required extension; the corpus README states the extension as a rule. The pinned-emit trailing-record check ran on the inputs and not the twin, though the twin runs through the identical suppressed topology; a twin without the record would emit nothing and the positive control would fire on absence. Both lists are checked, the refusal names which one is short, and the existing fixture's twin gains the record so it still trips exactly one rule. A blank expects-fault loaded under the outcome-case rules and then reported itself refusal-class, because the loader decided on blankness and the case type on nullness. A blank value is now refused by name - a case that names nothing has said something it did not mean - and the loader passes its own decision to the builder so the two readings cannot drift. Defect-class sweep, none found: no other filesystem filter in the module; no other rule applied to the inputs and not the twin; no other entry point nothing calls. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FkRDNruycVXPrAsRrACc9
Eleven commits, none touching the conformance module; CONCEPTS.md and the root pom changed on both sides (chaos vocabulary, Infer annotations for core) with no overlap in content.
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.OpenSSF Scorecard
Scanned Files
|
✅ Duplicate Code ReportTwo engines run in parallel for cross-validation. Each has its own thresholds tuned to its baseline - the real safety net is the per-engine "max increase vs base" check. ✅ PMD CPD
No new clones introduced by this PR. ✅ jscpd (language-agnostic)
|
[superseded - a quarantined test changed outcome] 🧪🔒 Quarantine Lane Report
🔴 expected while the owner PR is open · 🟡🎲 flapper, pass proves nothing · 🚨 a deterministic quarantined test passing means its fix landed: delete its Updated for Superseded by a newer quarantine lane report. |
…dge like every other module Master gained a CLAUDE.md bridge in every module's src/test tree (#378) so the prove-it-red rule arrives when a test file is touched; a new module misses it silently, which that commit's own body names as the failure to watch for. This module's tree now carries the same one-line import. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FkRDNruycVXPrAsRrACc9
|
[superseded - a quarantined test changed outcome] 🧪🔒 Quarantine Lane Report
🔴 expected while the owner PR is open · 🟡🎲 flapper, pass proves nothing · 🚨 a deterministic quarantined test passing means its fix landed: delete its Since the previous push: Updated for Superseded by a newer quarantine lane report. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #461 +/- ##
============================================
+ Coverage 81.99% 82.49% +0.50%
- Complexity 1472 1481 +9
============================================
Files 95 95
Lines 5165 5165
Branches 505 505
============================================
+ Hits 4235 4261 +26
+ Misses 731 709 -22
+ Partials 199 195 -4
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
🟢 Throughput — OKThis branch measured about 7% slower than master, on the one test this measures. That is INSIDE this test's own run-to-run spread of about 17%, so read it as a reading and not as a result - re-running the same commit moves it by about as much.
Allowable range 🟢 ≥ 0.70 · 🟡 0.50–0.70 (about a 30% loss) · 🔴 < 0.50 (about a 50% loss) What the numbers mean, and what they cannot tell youThe one that gets misread. Why a shape and not a rate. A rate depends on which runner you drew. A shape does not: every test here processes a fixed number of records, so a runner twice as slow doubles the subject and the controls together and leaves their ratio alone. That is the whole trick, and it is why the reported rate is shown last and labelled as this machine only. Reading the comparison. By conservation, not by correction. Every test in this lane processes a fixed number of records, so within one run the ratio of one test's time to another's is invariant under machine speed — a runner twice as slow doubles both terms and leaves the ratio alone. There is no machine-index correction to be wrong, because nothing needed correcting. Per-method times, not class times. A class time is Reference is the median of 10 recent What this still cannot do. It removes machine-to-machine variance. It does not remove this test's own run-to-run variance, measured at about 30% on a single unchanged commit while its controls stayed within 5%. That is a property of the test, not of the comparison, and no arithmetic here can touch it — which is why the reference is a median and the bounds are deliberately coarse. 🟡 means look at this; only 🔴 is outside the measured spread. Runs used: 867c407, 7a8dd92, df3bd71, 440b9bd, 11e0500, e20e898, 9999144, cc36b64, 1941cdf, 05c02bb Since the previous push: ratio 1.012 -> 0.928, share 1.65 -> 1.781, rate 66188 -> 62571 (-5.5%). One push of difference sits inside this test's measured spread - read it as movement, not as a result. Updated for |
…nch's own lines, the real ones fixed and the style ones named bin/check-pr-analysis-surfaces.sh on the PR listed what SpotBugs reported on lines this branch wrote. The real ones are gone: the named-and-discarded previous values from Properties.put and Map.put were dead stores under a detector whose test-class exclusion does not reach the module's production-shaped classes, so they are bare puts now with the reason stated once above each block, as core's own tests write configuration; a map named set is named declared; the emit rule's null-for-absent is resolved in one method rather than assigned to an enum local and overwritten. The one-member EmitRule enum is the design and its javadoc now says so. Left, as style rather than defects: the parameter-could-be-Collection and presize-the-collection families, and one burying-logic heuristic on a guard-clause method whose flattening would duplicate a guard or merge three named refusals into one. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FkRDNruycVXPrAsRrACc9
🧪🔒 Quarantine Lane Report
🔴 expected while the owner PR is open · 🟡🎲 flapper, pass proves nothing · 🚨 a deterministic quarantined test passing means its fix landed: delete its Since the previous push: Updated for |
astubbs
marked this pull request as draft
September 7, 2026 06:30
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #242 - the lowest rung of the conformance ladder, cut off master. It closes nothing on its own.
Description
The engine-independent half of a conformance net for the Kafka Streams foreign bindings: a new test-only module,
parallel-consumer-streams-conformance, with no main sources and nothing to publish. Nothing downstream may depend on it.What it holds, in the order the plan (
docs/plans/2026-09-05-001-test-streams-conformance-net-plan.md) builds it:TopologyTestDriver, one builder call per operation, byte-array serdes throughout, every store and sink snapshotted inside the driver scope. In Kafka 3.9.2close()produces silence, not an error - a re-fetched store comes backnulland a held handle reports nothing - which is the shape of a green test asserting nothing; the first test written measures that and pins the snapshot's placement.<case>/determinism(the control arm: the same case twice must agree) and<case>/positive-control(the perturbed twin must differ) - plus oracle-execution and empty-corpus as their own red categories. Surefire is asked, module-locally, to write the cell's display name so the report says which case reddened.emit: on-window-close(oracle-only), and two refusal-class cases in the wire's fault vocabulary.testing-evidence.yamlrow stating the claim's bounds, and the driver-rung obligations note.What this rung does and does not claim
Final state, under
TopologyTestDriver, at the reactor's one pinned Kafka version, no broker row, no binding measured. It evidences the oracle and the net around it, never that any engine conforms - that arrives at the driver rung, which is therefore the gate on reopening exactly-once across the boundary and the sidecar-versus-embedded choice, not this rung's green.Sabotage-arm record (KTD9)
Every proof was reddened by moving one term on the oracle or harness side - never the test - with the untouched tree green either side and the file restored by checksum. Cells named as surefire reports them.
runPerturbationpipes the inputs instead of the twinhopping-count-by-key/positive-controlhopping-count-by-key/determinism(different windows on the two runs)no window store named counts is registered with the driver), plus every oracle test that reaches a snapshotjava.lang.Stringcount, which has a casesink- and the positive control with itThe translation arm exposed that the hopping case's twin was detectable only through its sink: its retention expired every earlier window once the trailing record advanced stream time. Its retention is widened so the twin shows in the store too.
Decisions the plan left to the implementer
@Nullablebecause nothing on the reactor's classpath supplies one.countisLong-valued in Kafka, so that store and any sink it feeds carry aLongserde and a byte-array function applied to it is refused by name.ServiceLoaderthat needs it, and without it every test class logged aNoClassDefFoundErrorand per-test reporting was silently dead.Deferred, and why
Written down once, in
docs/inflight/test-streams-conformance-driver-rung-obligations.md: the wrapper as a test dependency in the same change that registers its row; the reservedcall-logslot; the builder-surface copy and the function vocabulary to reconcile againststreams.proto, with the node table's stated limits (countandaggregateshare a node;group-by-keyandwindowed-byare witnessed, not minted); the update-stream observable (final-state+updatesis refused at load here); the pinned-emit case to exercise once the wrapper exposes an emit control; the refusal-class cases to execute; the per-record topic to mirror; the design note on the #334 branch to reconcile; the runtime budget. Left open as the owner's call: whether the streams engine may reach master without a conformance row - the guard cannot enforce that coupling, since a class reaches this module's classpath only by a declared dependency.Code review
ce-simplify-codeandce-code-reviewran locally (run20260907-015029-d93001d6; correctness and adversarial on Opus, the rest on Sonnet; no cross-model peer available on this host). Four validated findings, all applied in the last commit: the selector was documented as read at the gate's entry and read by nothing; a case saved as.ymlvanished silently; the pinned-emit trailing-record rule skipped the twin; a blankexpects-faultloaded as both classes.Known residuals (informational, from the review)
count/aggregatesharing a node is the disclosed instance.bin/check-test-log-config.shlists modules by name and does not cover this one; its test logging comes from core's tests-classifier jar.CI on this head
dups: similarityis red and stays red: it is the documented tool defect indocs/inflight/ci-dup-similarity-cannot-accept-known-duplication.md- every new module's two-line ArchUnit wrapper scores in the eighties against the nine existing ones and the tool has no allowlist. The one other pair above its threshold, the selector test against the guard test, shares no method; the same metric pairs the guard test at a similar score with an unrelated core state-machine test, which is the metric, not duplication. The two density-based clone engines both pass.bin/check-pr-analysis-surfaces.shwas read: the SpotBugs findings on this branch's own lines are fixed where real (dead stores from the named-discard convention, a mis-named map, an enum local assigned null) and left where style (parameter-could-be-Collection, presize, one burying-logic heuristic on a guard-clause method); the last commit's body names each.claude-reviewandreview: human LGTMare the expected not-yet-reviewed state.Post-Deploy Monitoring & Validation
No additional operational monitoring required - the module is test-only, ships no artifact and runs only in the no-Docker unit lane.
Checklist
CONCEPTS.md(Agreement level, Perturbed twin),docs/data/testing-evidence.yaml, the driver-rung obligations notedocs/features/- N/A - test-only module, no user-facing featuredocs/inflight/working note (pr-/branch-) started at the PR's first commit - none needed: the cross-branch state this rung creates is the driver rung's obligations, which have their owntest-note, and the plan document carries the rest; nothing here thatghcannot showce-simplifyandce-code-reviewlocally - both, findings applied🤖 Generated with Claude Code
https://claude.ai/code/session_01FkRDNruycVXPrAsRrACc9