feat(clients) astubbs#242: ten foreign dispatch clients, and the runner registry that will drive them - #390
Draft
astubbs wants to merge 9 commits into
Draft
Conversation
…ndshake against the real sidecar Replaces #380's hello-world skeletons in Kotlin, Scala, Go, Python, TypeScript, Rust, Ruby, C#, C++ and Swift with the real dispatch-only client libraries, taken by path-scoped checkout from #293 rather than rewritten. Each carries a record end to end on the wire: spawn the sidecar, read its announced port, hold its stdin as the parent-death lifeline, negotiate `dispatch` and nothing else, queue to the ceiling the proxy declared, run the user's function, report a per-record outcome, half-close and reap. WHAT CHANGED FROM THE SOURCE BRANCH, AND WHY. That branch's per-language end-to-end tests spawn `TestModeMain` - the engine-backed harness in the proxy module's test jar - and assert that a record is dispatched, processed once and not redelivered. None of that exists on this stack: the sidecar here hosts no engine and answers every session UNIMPLEMENTED (#384). Bringing those tests would have meant writing the engine or writing a stand-in for it, and a stand-in would make agreement between languages a statement about the stand-in. So each language's engine-dependent test is replaced by the claim that CAN be made here, and it is the same claim the Java client already makes one rung down in `SidecarHandshakeTest`: the handshake reaches the session service, and the service's own refusal reaches the caller. That exercises the entire client-side path up to the point an engine would matter - spawn, port parse, lifeline, channel, Configure on the wire, and the mapping of the server's answer back into the language's own error idiom - across eight languages that have never spoken to this server before. The status CODE is the assertion rather than "it failed": PERMISSION_DENIED comes from the authority allowlist and RESOURCE_EXHAUSTED from the admission slot, both raised by interceptors before the service method runs, so UNIMPLEMENTED is the only answer that proves the Configure was delivered. Each language carries a permanent control arm - the same client pointed at a port nothing is listening on, which must fail differently. Everything each language could already prove without an engine is kept verbatim: the dispatch queue's FIFO hand-out and its overflow-is-a-protocol-violation rule, the in-flight ceiling counting queued-plus-executing, the session-end mapping against a scripted stream, wire and outcome conversion, credential redaction, fork safety and worker reaping. The behavioural dispatch scenarios stay with the shared conformance suite and are deferred with it. Build wiring: every module's build and test command now goes through `bin/foreign-client-step.sh` (#380's convention) so an absent toolchain is a loud skip on a developer box and red under `-Dpc.foreign.strict=1`, which every CI row sets. Each module's harness profile declares the sidecar as a test-scope dependency under `-Dpc.foreignClients` only, and writes the resolved classpath to `target/sidecar-classpath.txt` where its test reads it. THREE DEFECTS FOUND WHILE WIRING THIS UP, all of which only bite outside CI: - The Rust build script preferred the protoc copy in the local Maven repository over a real protoc on PATH, inverting the order its own header states. That copy is a bare binary with no sibling `include/`, so every build on a machine that had built the protocol module failed on `google/protobuf/duration.proto: File not found` - which reads as a broken schema. It is also written 0644 by Maven, so a second version of it failed with `Permission denied` instead. The ordering is restored and the executable bit is checked; CI never saw either, because its rows set $PROTOC explicitly. - Rust, Ruby and .NET located the sidecar by hunting the proxy module's jars on disk and resolving its dependencies with a nested `./mvnw`. In a reactor run that stops at `test` a module's output is a class DIRECTORY and no jar exists, so the harness reported "the proxy is not built" on a build that had just built it. All three now read the classpath file Maven writes, as Go, Python and TypeScript already did. - The Kotlin and Scala modules could not pass `requireUpperBoundDeps` once they depended on the gRPC transport: grpc-stub asks for error_prone_annotations 2.30.0 while the guava beside it asks 2.47.0, and Kotlin's own stdlib and coroutines split org.jetbrains:annotations the same way. Both are managed up to the highest requester in each module's own pom, which is the fix that rule prescribes and the same one parallel-consumer-proxy-protocol already carries. `bin/build-client.sh` arrives with the two container languages that need it, which is exactly what #380 said it was waiting for. Its host-run assertion gains a guard: both images build Linux binaries, so on a macOS host every extracted artifact fails to load and the portability check would report a real regression and a developer on the wrong OS as the same red. That case is now CANNOT RUN rather than a failure, and the container build - which runs each module's own tests and static analysis inside the image - is unaffected.
…hat tells a deferred cell from a typo Lands the foreign half of the conformance suite that #387 left by name - the runner registry, the repository-layout helper and the sidecar shim - and wires the prebuild that will build those runners. It is additive at the one seam that rung kept open: `ConformanceBindings.selectable()` was already a two-registry concatenation, and the foreign registry now sits beside the JVM one. WHAT IS DELIBERATELY NOT HERE, AND WHY IT IS GUARDED RATHER THAN NOTED. The driver's spawn half is not taken. It cannot compile: `ConformanceDriver.spawnAgainst` calls `harness.startEngine()`, and `ConformanceHarness` has no engine lane on this stack because the sidecar hosts no engine (#384). Writing one would mean writing the engine, and stubbing it would make agreement between ten clients a statement about the stub - which is the one thing this suite may never be. So `LanguageRunner` is not a `ConformanceBinding` here: it carries where a runner is built, what builds it and where the binary lands, and nothing that needs a running engine. `TheEngineArrivingMustBringTheForeignCellsTest` is what stops that being permanent, in the shape the java-grpc guard one registry along already uses: an equality in both directions, asserted for EVERY registered language rather than one of them. Put the engine on the classpath without registering the bindings and it goes red naming what to do; register them without an engine and it goes red the other way. It asserts per language because the realistic rot is not "the engine landed and nobody noticed" but "six languages were wired up and four were left", and four silent languages read exactly like four that passed. A DEFERRED CELL IS NOW NAMED BY THE FAILURE RATHER THAN ABSENT FROM IT. Selecting `go` used to be indistinguishable from a misspelling - the same "names bindings this suite does not register" - and the reader's next move differs: `go` is not a typo, its client and its runner exist, and what is missing is an engine. The selector now says so, and names the two guard tests that go red the day it stops being true. The deferred set is derived from the runner registry rather than listed, so a language added to one cannot be forgotten in the other. THREE THINGS THAT COULD NOT BE CHECKED AT ALL BEFORE, AND CAN BE NOW: - `LanguageRunnerRegistryTest` reads the client modules off disk and asserts the registry is the same set, both directions. Nothing else on this rung can exercise an entry - the suite cannot spawn a runner - so an entry naming a renamed module, or a binary path the language's build stopped writing to, would have sat correct-looking until the engine landed and ten cells needed debugging at once. It also pins that only the two reactor-built languages carry no build command, and keeps the absent-runner negative control: a missing binary FAILS naming the command that builds it, never skips. - `SidecarShimTest` RUNS the shim rather than reading it: it announces its port, stays alive, and exits by itself when its stdin closes, which is the whole of the spawning contract as a client observes it. The lifeline half is the half worth testing - a shim that exited after announcing would look right in every transcript and would break every spawned run, and it is exactly what an innocent edit to a `cat` would produce. - The prebuild is wired at `process-test-classes` and builds nothing today, because no foreign binding is registered. Wiring it with the registry rather than after it means the day one is, its runner is already built before the matrix asks for it, instead of the four-at-a-time build race its own header records being rediscovered.
… now have something to put in, and correct what they claimed The release-documentation records, the CI matrix and every client README were written when these modules printed one line or, on the source branch, ran against an engine. Neither is true here, and the difference is exactly the thing a reader must not have to work out for themselves. RELEASE DATA. Ten module-maturity rows and ten testing-evidence rows, one per language. Each says the same two things in its own terms: what IS evidenced - the handshake against a real sidecar process for the eight languages whose toolchain can reach one, plus each library's own logic over fixtures - and what is not, which is everything past the handshake, because no record has crossed the wire. C++ and Swift say something narrower still: their toolchains live in containers with no JVM, so they have not spoken to a sidecar at all, and what is evidenced is their in-image suites and the portability of what those images produced. That retires `docs/inflight/docs-proxy-client-modules-carry-no-maturity-or-evidence-rows.md`, whose "what lifts it" was precisely this. What did not retire with it is split into a new note: the per-module fragment mechanism on #293 is still not here, and the reasons that rung gave for declining it are now spent - the modules have content and they have rows - so the remaining question is the merge path itself, which is a trade rather than an oversight. READMES. Every one of the ten gains a second warning block under its EXPERIMENTAL banner, saying that no record has crossed the wire on this branch and why. Several also had claims that were simply wrong here - "proven end to end against the real sidecar", "carries records end to end over the real protocol", a table row naming a test file this rung deleted - and those are corrected rather than left for a reader to discover by running them. CI. The `clients` matrix rows now build and test real clients, so three things move. C++ and Swift become container rows: each module's Dockerfile is its toolchain pin, the image runs the module's own tests and static analysis, and `docker` rather than a language compiler is what the probe checks. Swift moves off `macos-latest` with it - the macOS image's bundled compiler answered the whole question for a program that printed one line, and answers none of it for a client needing grpc-swift, swift-protobuf and two protoc plugins built from source. And the Rust row installs protoc, because its `build.rs` generates from the frozen schema on every `cargo build` - the one row of the ten that runs protoc on the runner at all. The Go module's test command gains `-race`, which closes the item `docs/inflight/static-polyglot-client-analysers.md` ranked first of five: this is the client with the executor fan-out, the detector is in the toolchain, and a per-language note already claimed a coverage the gating recipe did not have. It also gains `-count=1`, which is not from that survey and is the more urgent of the two - Go caches a package's test result on its own inputs, and the sidecar a Go test spawns is not one of them, so breaking the sidecar's refusal on purpose returned `ok (cached)` and the sabotage arm proving the handshake test could fail read as green.
…s a branch boundary Every client README was written on `feats/proxy-requirements` and cites that branch's tree. Most of those paths do not exist here, and a citation that looks right reads as right - so `bin/check-file-refs.sh` fails, which is the gate doing its job on an extraction. Three kinds, handled three ways rather than silenced as one: - **The frozen documents moved and the citations are simply repointed.** The protocol specification and the client-authoring guide live in `parallel-consumer-proxy-protocol` here, not in `parallel-consumer-proxy`; ten READMEs pointed at the old home. This is a real repair, not an exemption. - **The per-language wave notes, the two bug notes, the parked containerisation note and the plan stay on the source branch, and each citation now says so** - with the `git show` command that reaches it, which is what `docs/citations.md` prescribes for a path this tree does not hold. They are not copied here: they are the record of how those waves went, and this rung is not their owner. - **`target/sidecar-classpath.txt` is build output**, named in two READMEs because a reader has to know where the file comes from. Marked as output rather than repointed at nothing. The Kotlin module's `src/test/CLAUDE.md` write-time bridge is deleted, for the same reason both rungs below deleted theirs: the document it imports arrives with #378.
… race, on a differently loaded box #380 recorded this flake with its control arms and asked for sightings rather than a rewrite. This is one: a full reactor `test` run with `-Dpc.foreignClients` on the dispatch-clients branch went red on `processInKeyOrder(CommitMode)[1]` with the same sanity-check assertion and the same shape - nine seeded records, none polled - and a parameterisation neither earlier run produced. It is recorded rather than acted on because it says nothing new about the cause and everything about the condition: a second agent was running `parallel-consumer-core`'s suite in another worktree on the same machine at the time, which is the load variable the original note already named. The diff on this branch touches no file under `parallel-consumer-core`. Still not a quarantine candidate on this evidence - quarantine is master-state, and every sighting so far is one machine under load it inflicted on itself. What would change that is CI producing one.
Dependency ReviewThe following issues were found:
|
❌ Duplicate Code ReportTwo engines run in parallel for cross-validation. Each has its own thresholds tuned to its baseline - the real safety net is the per-engine "max increase vs base" check. ✅ PMD CPD
No new clones introduced by this PR. ❌ jscpd (language-agnostic)
|
🧪🔒 Quarantine Lane Report
🔴 expected while the owner PR is open · 🟡🎲 flapper, pass proves nothing · 🚨 a deterministic quarantined test passing means its fix landed: delete its |
6 tasks
…x structurally could not Every one of these was invisible locally for a reason worth naming, because the reason is the same in four of the five: this machine has a warm `~/.m2`, a full rustup profile, no Linux, and a duplicate-code baseline it cannot compute. **A CMake file got an HTML comment, and the C++ image stopped configuring.** The citation-repair pass inserted `<!-- file-refs: N/A -->` markers to close paragraphs that cite the source branch, and `bin/check-file-refs.sh` scans more than markdown - `CMakeLists.txt` was one of the files it flagged, so it got a marker in a syntax CMake reads as an unquoted argument. The gate then went green and the image went red. Rewritten in CMake's own `#` form; every other non-markdown file the pass touched was swept for the same shape and none had it. Not caught locally because the container build that would have caught it is exactly the one the low-disk guard said not to start again. **The Rust CI row installed a minimal toolchain and the module's build command is `cargo clippy`.** `--profile minimal` ships rustc and cargo and nothing else, so the row died with "'cargo-clippy' is not installed" before compiling anything. `--component clippy`, and the row now prints its version so a silent no-op setup cannot look like a working one. **The Scala cache warm stopped one phase short of what its own dependency graph needs.** It runs `-pl :parallel-consumer-proxy-client-scala -am compile`, and the comment beside it said the Scala module "declares no Parallel Consumer dependency" - true of the skeleton, false now that it depends on the gRPC transport, which reaches the protocol module, which test-depends on core's test-jar. Maven resolves an inter-module test-jar from the reactor only once the producing module has reached `test-compile`; stopped at `compile` it falls back to the repository, which on a runner with an empty `~/.m2` is `Could not find artifact ...core:jar:tests`. That failed the cache job, and with it the cache every other lane restores from. Diagnosed with a control arm rather than a guess: in a clean local repository (`-Dmaven.repo.local=/tmp/pc-cleanroom`), `compile` fails with the runner's exact error and `test-compile` succeeds - same command otherwise, one term changed. A developer's warm `~/.m2` holds the artifact from any earlier build, which is why this failure exists only on a cold machine. **gson reached the Kotlin and Scala modules at 2.11.0**, which is the version OSS Index matches for CVE-2025-53864, so the whole-tree CVE scan failed. It arrives through `grpc-core`, and the Java client aggregator already pins it forward for its own children - dependencyManagement does not flow between siblings, so these two modules had inherited none of it. Both pins now match that aggregator's, because their graph IS its graph with a Kotlin or Scala surface on top. The `error_prone_annotations` pin these modules gained earlier moves to the same number for the same reason; two differently-motivated numbers in one graph is how they come to be "fixed" into agreement later. **Six copies of the sidecar-classpath wiring, which the duplicate-code check was right about.** Making eight languages spawn the same sidecar the same way left eight near-identical `maven-dependency-plugin` executions, and jscpd's max-increase rule failed on them. The execution is now managed ONCE in the clients aggregator and a module opts in by declaring the plugin - which is safe to hoist precisely because pluginManagement contributes nothing to a module that does not declare it, so the conformance module, which deliberately depends on no sidecar at all, is untouched. The per-module dependency stays where it is: it is the thing that differs, and it is what makes `-am` order the build. The copies had already drifted, which is the argument for collapsing them rather than tolerating them: two of the six wrote a differently-named file, and their harnesses hunted jars on disk instead of reading it.
…ness wiring that is actually its own The hoist put the `sidecar-classpath` execution in the clients aggregator and left six copies of the prose explaining it in the modules, which is the duplication moved rather than removed - jscpd counts comments, and the six blocks were still the biggest clones in the report. The reasoning now lives once, where the mechanism does. What each module keeps is the only thing that cannot be hoisted: a test-scoped dependency on `parallel-consumer-proxy`, inside the opt-in profile, which is what makes `-am` build the sidecar before the module's suite tries to spawn it. It cannot go to the aggregator because a profile's `<dependencies>` reach every child, and one of those children is the conformance module - whose pom records, as a reactor measurement rather than an assertion, that it depends on no sidecar at all. What remains identical between the six is a dependency declaration and a plugin declaration. That is not duplication with a refactor available; it is what "these eight modules all spawn the same process" looks like once the shared part has been shared.
|
…imports `protobuf-compiler` ships the binary and NOT the well-known types, so protoc ran and then could not resolve `google/protobuf/duration.proto` and `timestamp.proto` - which the frozen schema imports. The failure reads as a broken .proto (twenty-six "is not defined" lines about the schema) rather than as a half-installed toolchain, which is the whole reason it is worth a commit message. `libprotobuf-dev` is what carries them, and where it puts them is what the crate's build script looks for: `../include` relative to the resolved protoc, which for `/usr/bin/protoc` is `/usr/include`. apt for both rather than a setup action, so the binary and the types come from one distribution and land where each other expects. THIS WAS ALREADY WRITTEN DOWN IN THIS REPOSITORY AND THE ROW DID NOT READ IT. The Swift module's Dockerfile installs exactly this pair and says why in a sentence beginning "libprotobuf-dev is not optional despite nothing here linking C++". Swept for the same shape: every place in the tree that installs `protobuf-compiler` - this row, and the C++ and Swift Dockerfiles - now takes `libprotobuf-dev` beside it, and the two Dockerfiles already did. The row also now asserts the types are on disk before the crate needs them, so a half-installed toolchain fails at the provisioning step naming the package rather than three minutes later naming the schema. That is the same rule the row's toolchain probe already follows: a setup step that silently did not do what it claimed must not be discovered by its victim. Only this row is affected: Go, Ruby and TypeScript commit their generated stubs, .NET's Grpc.Tools package ships its own protoc, and C++ and Swift generate inside their images.
…s its third parameter `Integration Tests` on this branch failed `PartitionStateCommittedOffsetIT.committedOffsetRemoved` on the `[1] latest` arm, in `checkHowManyRecordsWithKeyPresent` - expected 2, got 1. The ledger already carries that exact assertion on `[2] earliest` and a different one on `[3] none`, so with this sighting every parameter of the method has been seen failing. WHAT MAKES IT WORTH RECORDING RATHER THAN SHRUGGING AT. `[1] latest` is the arm that `latest-reset-nudge-race-committedoffsetremoved-2026-07-30.md` was written for and merged a fix for. That fix solved an unwinnable AWAIT; what failed here is the ASSERTION three statements later - and `docs/plans/2026-08-05-001-investigate-committedoffset-latest-reflake.md` predicted exactly this in as many words: moving the nudge inside the await made its count unbounded while call sites went on assuming exactly one. So this is that plan's hypothesis arriving on the arm it was written about, which is a great deal more useful to the next reader than "flaked again". The ambient probe's autopsy is quoted rather than re-derived, per the rule that says to read it before diagnosing by hand: probe clean, no rebalance dwell, no lag stagnation, no frozen partitions - the fault is test-side rather than consumer-group progress. Recorded, not acted on, and not quarantined. This branch's diff touches no file under parallel-consumer-core and the modules it adds are client wrappers no core test resolves, so the sighting is master state; and the quarantine registry wants an owning fix PR, which the diagnosis above says would have to be written first. The rule that this exists to satisfy is that a flake observed on a PR's CI gets its ledger entry before that PR merges - the logs expire long before anybody picks the work up.
astubbs
added a commit
that referenced
this pull request
Aug 31, 2026
…s engine back Merges `feats/proxy-dispatch-clients` (#390, the top of the Wagon A stack) into this branch, so that #293 can be retargeted onto that rung and its displayed diff collapse to the engine residue. This is the plan's "retarget move" - `docs/plans/2026-08-31-001-process-god-branch-decomposition-plan.md` - and it is an ordinary merge: no history was rewritten and nothing was force-pushed. The pre-merge tip is preserved as `origin/backup/pre-stack-merge-293`. The merge also brings `master` forward, because the stack is current with it and this branch was not. That is most of the volume and most of the conflicts. ## Which side won, and why The rule was: the god branch's SEMANTIC content, expressed on the stack's refined STRUCTURE. - **Engine-only files stayed** - `ProxyProcessor`, `ConfigureHandler`, the dispatch waves, epochs, leases, reconnect and the produce path. They are the residue this campaign exists to expose. - **Stack-only files were taken** - the ten foreign clients, the runner registry, the sidecar shim, the self-retiring guards, `Main` and `ParentDeathWatchdog`. - **Records lost, everywhere.** The stack converted five value types from Java records to plain final classes because Jabel rewrites a record into a class with no source positions and Error Prone 2.42.0 crashes rather than reporting on it. This branch never met that only because its root pom had no Error Prone; master's does now, so the conversion is load-bearing here. - **Dependency pins are the stack's**: protobuf 3.25.8 (`requireUpperBoundDeps` refuses 3.25.5 against grpc-protobuf's transitive ask), gson 2.14.0, `error_prone_annotations` 2.48.0 in the proxy and the Java client aggregator and 2.47.0 in the protocol module - deliberately unequal, because their highest requesters differ. - **`clients.yml` is this branch's**, not the stack's. The stack's copy is a smaller earlier file whose own header says it has no conformance lane, no per-language static analysis and no dependency audit; only its `--component clippy` fix and its workflow-level `PC_FOREIGN_CLIENTS_STRICT` were folded in. - **Master's copy won for everything master owns** - the gate scripts, the copyright checker and its self-test, `repo-hygiene.yml`, `docs/ci.md`, `docs/copyright.md`. This branch's only delta in most of those was a copyright header in a pre-canonical form, written before #338 landed the canonical one on master. - **The two ledgers were merged as unions**, not resolved to a side. `docs/inflight/bug-857-family.md` keeps both sides' sightings, this branch's renumbered per the file's own stated convention; `docs/quarantined-tests.md` drops three entries whose owning PRs retired the annotation on master, because a registry entry without a `@Quarantined` test is a hard gate failure. ## The verdict-free work return, re-expressed on master's shape Unit U4 is still on this branch as `WorkManager.onAbandonedResult` and its call sites, and `WorkManager.java` auto-merged keeping them - so resolving `WorkContainer`, `ProcessingShard` and `ShardManager` to master alone would have compiled to nothing. Master moved underneath it: #335 collapsed `inFlight` and `maybeUserFunctionSucceeded` into one atomic `ExecutionState`, and #336 and #373 replaced the shard's raw `availableWorkContainerCnt` with a compare-and-set claim. `ProcessingShard.onAbandoned` and `ShardManager.onAbandoned` are therefore taken verbatim from `feats/proxy-verdict-free-return` (#295), which is cut on current master and is the authoritative re-expression - it routes the restore through `includeInSelection` rather than incrementing a counter that no longer exists. `WorkContainer` is NOT #295's, and the difference is deliberate: that branch marks abandonment with an `AtomicBoolean` cleared by the claim winner, while this branch keys it by delivery (`markAbandoned(long)`, `isAbandonedForCurrentDelivery`, `isReturnForSupersededDelivery`). The delivery-keyed form is strictly more: it tells a late return for delivery n from a live return for delivery n+1, which the boolean cannot, and acting on that confusion ends a running flight and decrements `numberRecordsOutForProcessing` twice. It also needs no clearing on redelivery, so it does not race the claim. Master's own `deliveryCount` - already an `AtomicLong` incremented only by a won claim - supplies the identity, so the field is the one thing added. ## The reconciliation duties #387 and #390 recorded Both are discharged, and the guards they left behind now pass because the gap closed rather than because anything was edited. **`ProxyHarness` and `ConformanceHarness` are one class again.** #387 cut the latter out of the former with the engine lane removed, and wrote into the class header, the module pom and a deferred inflight note that whoever landed the engine must merge them - leaving open which module it ends up in, on the grounds that "the engine lane decides". It decides the sidecar module: the lane needs `ConfigureHandler` and `ProxyProcessor`, and `TestModeMain` - the process a spawned foreign runner actually starts - is in that same test tree and would otherwise have to import the conformance module and close a cycle. So the NAME is the extracted rung's and the MODULE is this branch's, and `parallel-consumer-proxy-conformance` reaches it through the proxy test-jar its pom already declared. The stack's refinements to the class survive: `Delivery` stays a plain final class, and the `startEmbeddedClient` lane stays beside the restored `startEngine`. **`ConformanceDriver.spawnAgainst` is back**, with `LanguageRunner` a `ConformanceBinding` again - it stopped being one on #390 for exactly as long as there was no engine to spawn against - and the ten foreign bindings are registered in `ConformanceBindings.selectable()`. `ConformanceBindings.deferredUntilTheEngineArrives()` is kept and now returns an empty list, which is the honest reading: nothing is deferred. It is not deleted because `TheEngineArrivingMustBringTheForeignCellsTest` reads it, and because the next capability to arrive without a cell needs that list rather than a new one. ## Two duplications the merge surfaced, resolved rather than left - **The protocol documents existed twice.** #383 moved `protocol-specification.md` and `client-authoring-guide.md` into `parallel-consumer-proxy-protocol/docs/` and predicted the rename conflict here. The protocol module wins on the merits - all three artifacts are the contract rather than the server - but the CONTENT is this branch's, because the stack's copies carry "no client generates from this schema yet" and a `git show` fallback for a Go generator script that exists here. `SpecificationCoverageTest` moved with them; every citation was re-pointed. - **`HarnessScenario` existed twice**, once per harness. The stack's plain-class version won, in the sidecar module beside the class it serves. ## Five Java records had to become plain final classes, or nothing compiled `master` gained Error Prone after this branch forked, and Error Prone 2.42.0 does not report on a Jabel-desugared record - it crashes, failing the whole compilation with a message attributed to line 1 of an unrelated file. #387 hit this one rung down and converted its five value types; the same thing was waiting here for `ProxyProcessor.ManifestOutcome`, `ManifestReconciler.Reconciliation`, `InFlightRegistry.InFlight`, `LivenessSettings` and `TestModeMainTest.Run`. Neither term can move - the root pom pins Error Prone at 2.42.0 because 2.43.0 needs a JVM this build cannot use, and Jabel serves the release 8 target - so the records lost, which is what the rest of the repository already does. `grep -rn "record "` now finds none. ## Two defects this merge introduced and then fixed - **The default lane ran a test that needs a profile.** Keeping BOTH sidecar-spawning tests - this branch's engine-backed `OneRecordThroughTheSidecarTest` and the stack's engine-less `SidecarHandshakeTest` - left the Kotlin and Scala exclusion property naming only the first, so the second ran in an ordinary build with no `sidecar-classpath.txt` and failed. Both are now behind one regex, which is also the arrangement that makes the pairing legible. - **Every proxy module's data record existed twice.** This branch keeps module-maturity and testing-evidence records in per-module `docs/data/*.d/` fragments; the stack, which never imported that mechanism, wrote them into the monolithic files. The merge kept both and `bin/check-docs-data.sh` went from green to 37 structural problems. The fragments win, and deleting the monolithic copies also deleted eleven rows asserting "NO RECORD HAS CROSSED THE WIRE" - true on a stack whose sidecar hosts no engine, false here. ## How it was verified JDK 17, macOS. Full default reactor `./mvnw --fail-at-end test`: **BUILD SUCCESS**, every module green including the conformance suite - so the `java-grpc` cell runs against a live engine for the first time, and `GrpcSpikeConformanceTest` answers all five scenarios over a real gRPC stream. The four self-retiring guards #387 and #390 left behind all pass, and **none of them was edited**: `TheEngineArrivingMustBringTheGrpcBindingTest`, `TheEngineArrivingMustBringTheGrpcCellTest`, `TheEngineArrivingMustBringTheForeignCellsTest` (which asserts about every registered language, not one) and `SelectorMatchingNothingFailsTest`. `bin/check-all.sh` real exit code 1, with three gates failing and all three inherited, established against the pre-merge tip rather than assumed: `check-inflight-tags` reports the same 53 problems before and after; `check-file-refs` was already red for the `@`-prefixed bridge imports that #378 fixes; and `check-branch-self-reference` did not exist on this branch before the merge - its findings are on notes inherited from `master` byte-identical. `check-proto-lint` and `check-proto-breaking` report CANNOT RUN for want of `buf`. **Not run locally: `-Dpc.foreignClients`.** It builds C++ and Swift inside containers, and this box is under a low-disk warning that names a container build as what would tip it over. CI's per-language rows are where that lane runs. ## Two findings recorded rather than fixed - `ParallelEoSStreamProcessorTest.processInKeyOrder` failed its own preamble sanity check once here - and the control arm reddened `master` HARDER: unmodified `master`, uncontended, failed all three parameterisations where this branch under concurrent load failed one. That refutes the existing ledger's "the base branch was green" row, and is the first evidence the flake is master-state. Recorded in `docs/inflight/test-processinkeyorder-sanity-check-races-the-first-poll.md`. - One `java-grpc` conformance run left its last record unsettled, 1 of 2 full-reactor runs and green 3 of 3 in isolation. `core` and `java-direct` were green in the same run, so the suspect is the engine's settle path under a full ceiling rather than the scenario. The scenario was NOT weakened - `docs/inflight/proxy-a-java-grpc-ceiling-run-left-its-last-record-unsettled.md`. ## What is deliberately NOT done here `Main#sessionServiceFactory` still returns `NoEngineSessionService`, so the production entry point hosts no engine. That is unit U10's, which #384 named as still having no PR, and it owns the drain that must land with the engine - plus eight cross-language handshake tests assert the `UNIMPLEMENTED` refusal by status code specifically. Recorded rather than faked, in `docs/inflight/proxy-the-production-entry-point-still-hosts-no-engine.md`. The engine is exercised end to end through `TestModeMain` regardless.
astubbs
marked this pull request as draft
August 31, 2026 21:21
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #242.
depends on #387
Description
Ten foreign-language client libraries, each implementing negotiated dispatch and nothing else, and
each proving as much of that as a sidecar with no engine can be made to prove. Kotlin, Scala, Go,
Python, TypeScript, Rust, Ruby, C#, C++ and Swift. They replace the hello-world skeletons
#380 landed, which that PR described as scaffolding waiting for exactly
this rung.
Every client declares
capabilities: ["dispatch"]and no more, so an engine that already supportsleases, heartbeats, reconnect and a shutdown drain will not use any of them here. That is what makes
ten clients cuttable as one PR: dispatch is a complete first feature slice, and the wire carries the
rest whether a client asks for it or not.
Where this came from
Extracted from #293, the language-proxy god PR, as the top rung of the
Wagon A stack in
docs/plans/2026-08-31-001-process-god-branch-decomposition-plan.md(extractionA9). Taken by path-scoped checkout rather than rewritten; every place it differs is named below.
The honesty constraint, and the split it forces
The sidecar on this stack hosts no Parallel Consumer engine. It binds, announces its port,
admits one connection under the transport's rules and answers every session
UNIMPLEMENTED(#384). So no record can cross the wire here, and the source branch's
per-language end-to-end tests - which spawn the engine-backed
TestModeMainand assert that a recordis dispatched, processed once and not redelivered - cannot be brought. Bringing them would have meant
writing the engine or writing a stand-in for it, and a stand-in makes agreement between ten clients a
statement about the stand-in.
Each of those tests is therefore replaced by the claim that can be made, which is the same one
the Java client already makes one rung down in
SidecarHandshakeTest: the handshake reaches thesession service, and the service's own refusal reaches the caller. That is the whole client-side path
up to the point an engine would matter - spawn the child directly, read
port:off its stdout, holdits stdin as the parent-death lifeline, open the channel, put
Configureon the wire, and map whatcame back into the language's own error idiom - exercised in eight languages that have never spoken
to this server before.
The status code is the assertion, not "it failed". A refusal from the authority allowlist is
PERMISSION_DENIEDand one from the admission slot isRESOURCE_EXHAUSTED, both raised byinterceptors before the service method runs; only
UNIMPLEMENTEDcan have come from the serviceitself. Every language also carries a permanent control arm - the same client pointed at a port
nothing is listening on, which must fail differently - so the assertion cannot pass on any failure at
all.
What runs live, per language
Debugnever prints a Kafka property; the port line among log chatterswift format lint --strictinside the image, plus the portability pairNothing deferred is stubbed, and nothing deferred is left to a comment - see the guards below.
The foreign runner machinery, and the one piece that could not come
#387 left the runner registry, the driver's spawn half, the sidecar shim and
the repo-layout helper for this rung by name, and kept
ConformanceBindingsas a two-registryconcatenation so the addition would be purely additive. It is.
The spawn half is the piece that could not come, for the reason that rung gave about
java-grpc.ConformanceDriver.spawnAgainstcallsharness.startEngine(), andConformanceHarnesshas noengine lane here - so it cannot compile, any more than a
@DisabledgRPC test could.LanguageRunneris therefore not a
ConformanceBindingon this rung: it carries where a runner is built, thecommand that builds it and where the binary lands, and nothing that needs a running engine.
What that leaves is guarded rather than described:
TheEngineArrivingMustBringTheForeignCellsTestasserts, for every registered language, thatthe binding exists exactly when the engine is on this module's classpath. Both halves are false
today; it goes red whichever way that changes, and retires itself. It is per language deliberately,
because the realistic rot is not "the engine landed and nobody noticed" but "six languages were
wired up and four were left", and four silent languages read exactly like four that passed.
-Dpc.conformance.language=goused to be indistinguishable from a misspelling.gois not a typo -its client and its runner exist, and what is missing is an engine - so the failure says that, and
names the guard tests that go red the day it stops being true. The deferred set is derived from the
runner registry, so a language added to one cannot be forgotten in the other.
LanguageRunnerRegistryTestreads the client modules off disk and asserts the registry is thesame set, both directions, plus that each runner builds inside its own module and lands inside it,
that only the two reactor-built languages carry no build command, and that an absent runner FAILS
naming the command that builds it. Nothing else on this rung can exercise a registry entry, so
without this an entry naming a renamed module would sit correct-looking until the engine landed.
SidecarShimTestruns the shim rather than reading it: it announces its port, stays alive, andexits by itself when its stdin closes. The lifeline half is the half worth testing - a shim that
exited after announcing would look right in every transcript and break every spawned run.
ConformanceRunnerPrebuildis wired atprocess-test-classesand builds nothing today, becauseno foreign binding is registered. Wiring it now means the day one is, its runner is already built
before the matrix asks for it, rather than the four-at-a-time build race its header records being
rediscovered.
Sabotage: one arm across nine languages, and four more
Every arm changes one term and is restored byte-identically; the control arm was run before and after.
Arm 1 is the one that matters, and the green half is the interesting half. Nine languages
independently agree about which gRPC status the server sent, and each one's control arm - pointed at
a dead port - stays green through it, which is what says the assertion is about the refusal rather
than about failing at all.
Arm 4 was run twice, and the first run is worth reporting because it was wrong. Pointed at a
proxy class, the guard stayed green - because the conformance module deliberately has no dependency
on the sidecar at all, so no proxy class is ever on its classpath. That is a property of the
sabotage, not of the test: the day a foreign binding is registered, the engine is on that classpath
by construction, because registering one requires the spawn half, which requires the harness's engine
lane. Re-run against a class that IS on the classpath, the guard goes red naming what to do. The
sibling guard one registry along has the same shape and the same answer.
What CI found that this box structurally could not
Reported rather than quietly fixed, because the pattern is the point: most of these were invisible
here for the same reason - a warm
~/.m2, a full rustup profile, a non-Linux host, and aduplicate-code baseline a local run cannot compute.
CMakeLists.txtgot an HTML comment and the C++ image stopped configuring. Thecitation-repair pass closed cited paragraphs with
<!-- file-refs: N/A -->, andbin/check-file-refs.shscans more than markdown. The gate went green and the image went red;every other non-markdown file that pass touched was swept for the same shape and none had it. It
was not caught locally because the container build that would have caught it is the one the
low-disk guard said not to start again. The row is green now, host-run portability assertion
included.
--profile minimalships no clippy, and this module's build command iscargo clippy; fixedwith
--component clippy. Then, with the crate compiling for the first time,protobuf-compilerturned out to ship the binary and not the well-known types, so protoc ran and could not resolve
the schema's own
duration.proto/timestamp.protoimports - a failure that reads as a broken.protorather than a half-installed toolchain.libprotobuf-devcarries them, and puts them inthe
/usr/includethe crate's build script derives from the resolved protoc.That pairing was already written down in this repository and this row did not read it: the
Swift module's Dockerfile installs exactly the same two packages and explains why. Swept: every
place in the tree that installs
protobuf-compilernow takeslibprotobuf-devbeside it, and thetwo Dockerfiles already did. Both halves now assert what they installed - the row prints
cargo clippy --versionand fails on a missingduration.protoat the provisioning step - so asetup that silently did nothing cannot be discovered three minutes later by its victim.
module reaches the protocol module, which test-depends on core's test-jar,
compilecannot resolveit from the reactor and falls back to a repository a runner does not have. Diagnosed with a control
arm in a clean local repository:
compilereproduces the runner's exact error,test-compilesucceeds, one term changed.
grpc-core- the version OSS Index matches forCVE-2025-53864. The Java client aggregator already pins it forward for its own children, and
dependencyManagement does not flow between siblings; both modules now carry the same two pins as
that aggregator, because their graph is its graph with another surface on top.
near-identical
maven-dependency-pluginexecutions. The execution is now managed once in theclients aggregator, and a module opts in by declaring the plugin - safe to hoist precisely because
pluginManagement contributes nothing to a module that does not declare it, so the conformance
module (which depends on no sidecar at all) is untouched. The copies had already drifted: two wrote
a differently-named file, which is why two harnesses hunted jars instead of reading it.
Four defects found while wiring it up, all of which only bite outside CI
protocover one onPATH,inverting the order its own header states. That copy is a bare binary with no sibling
include/,so every build on a machine that had built the protocol module died on
google/protobuf/duration.proto: File not found- which reads as a broken schema. Maven also writesit
0644, so a second version of it failed withPermission deniedinstead. Ordering restored, andthe executable bit checked. CI never saw either: its rows set
$PROTOC.go testcaches a package's result on its owninputs, and the sidecar a Go test spawns is not one of them - so arm 1 returned
ok (cached)andread as green.
-count=1fixes it, and the same commit adds-race, whichdocs/inflight/static-polyglot-client-analysers.mdranked as the cheapest real win on the wholepolyglot surface and recorded as claimed-but-not-gated.
nested
./mvnwfor its dependencies. In a reactor run that stops attesta module's output is aclass DIRECTORY and no jar exists, so all three reported "the proxy is not built" on a build that
had just built it. All three now read the classpath file Maven writes, as Go, Python and TypeScript
already did - which also deletes three copies of the jar-hunting code.
bin/build-client.sh --testcould not tell a portability regression from a developer on the wrongOS. Both container images build Linux binaries, so on macOS the statically linked artifact and its
dynamic control both fail to load, and the script reported "the extracted build is not portable".
That case is now CANNOT RUN (exit 2), which is what the rest of that script's exit codes already
mean. The container build itself is unaffected and still runs each module's own tests inside the
image, which is why it remains a real check on any host.
Five places this differs from #293 on purpose
bin/foreign-client-step.sh, which isbuild(polyglot) astubbs#242: eleven language client modules, each proving its toolchain reaches a running program #380's convention and not that branch's: an absent toolchain is a loud
skip on a developer box and red under
-Dpc.foreign.strict=1, which every CI row sets.--hellofixture assertion in that wrapper is now used by no module, because these clientsreplace the ten programs that used it. It is left in place with its self-test rather than removed:
the wrapper belongs to build(polyglot) astubbs#242: eleven language client modules, each proving its toolchain reaches a running program #380, and deleting a self-tested capability is that
rung's call, not this one's. Named here so it is not mistaken for an oversight.
macos-latest. The macOS image's bundled compiler answered the whole questionfor a program that printed one line, and answers none of it for a client needing grpc-swift,
swift-protobuf and two protoc plugins built from source - which the module's image already
assembles and caches. Both container rows now probe for Docker rather than for a language compiler.
poc/probe scripts are not taken. They are exploratory scripts drivenagainst the engine-backed harness, which does not exist here.
carries its own recipe. That is its own extraction, and
docs/inflight/static-polyglot-client-analysers.mdalready owns the survey and the ranked list -this PR closes its first item and records the Go cache finding beside it.
How it was verified
JDK 17, macOS (Apple Silicon), on this branch:
TypeScript, Rust, Ruby, .NET, Kotlin and Scala all built and ran their own suites green, and all
went red under arm 1 and green again after it. Ruby and Go ran on the versions this machine has
from
miserather than the versions the matrix pins.ctest/swift testand its static-analysis or formatter step ran green inside the image. Their host-run portability
assertion was not exercised locally, because this host is macOS - which is the guard described
above, and which CI's C++ row has since executed for real.
bin/check-all.shon its real exit code: 0. Two gates report CANNOT RUN -check-proto-breaking.shandcheck-proto-lint.sh, both wanting abufthis machine does nothave. Both belong to a base rung and neither is affected by this diff.
COPYRIGHT_CHECK_REQUIRE_FORK_POINT=1 bin/check-copyright-headers.sh- 0 violations against areal fork point.
bin/check-docs-data.shgreen with the twenty new records present.What is NOT locally verified and rides on CI: the pinned toolchain versions in the matrix, which
this machine does not have; the two container rows on a Linux runner, where the host-run portability
assertion actually executes; and the Rust row's
protocprovisioning. CI has since proved most ofit - Go, Python, TypeScript, Ruby, .NET and C++ all pass on their pinned toolchains - and the Rust
row is what the section above is about.
One core test went red locally and is reported as a rate rather than a verdict.
ParallelEoSStreamProcessorTest.processInKeyOrderfailed its own preamble sanity check on 2 of 2 fullreactor runs on this branch, and the identical command on the base branch, same box, same window, was
green; the method alone is green 3 of 3,
parallel-consumer-coreis byte-identical between the twobranches, and CI's
testsjob - the same suite on a Linux runner - is green on this PR. It is theflake #380 recorded with the same assertion and the same shape, and this run
is added to its sighting ledger with the control arm, because the base-green/branch-red split is not
explained by anything this diff touches and saying so is more use than a verdict.
Commits were made with
--no-verify, and the reason is recorded rather than assumed. Thepre-commit hook resolves the wrong project directory and gates this branch against a different
worktree's tree - the same failure #387 recorded, whose fix is in flight on
#382.
bin/check-all.shrun in this worktree exits 0, which is the check thehook was standing in for.
The duplication report, answered
PMD CPD is clean - no new clones, no change in duplicated lines. jscpd's max-increase rule is
red, and this is what was done about it and what is left.
Removed, because a refactor existed: the
sidecar-classpathexecution and the prose explaining itwere six near-identical blocks across the foreign modules, and both are now in the clients aggregator
once. That was worth doing on its own merits - the copies had already drifted, and two of them wrote
a differently-named file, which is why two harnesses hunted jars on disk instead of reading it.
Not removed, each for a reason rather than by omission:
duration.tsandtimestamp.tsare protoc's outputfor two structurally identical well-known types. There is nothing to refactor and nothing that may
be hand-edited.
to read identically in every language's document, and a reader of one README must not have to
follow a link to learn that no record has crossed the wire. The base branch already carries the
EXPERIMENTAL half of it eleven times.
thing that cannot be hoisted - a profile's
<dependencies>reach every child of the aggregator, andone of those children is the conformance module, whose pom records as a reactor measurement that it
depends on no sidecar at all. What is identical between the eight is "these modules all spawn the
same process", which is a fact about them rather than a copy.
Same shape as #380's answer for the ArchUnit wrappers it flagged: the rule
asks that duplication this PR introduced and can remove be removed, and the residue is named rather
than left for a reviewer to re-derive.
Known-red, and not a fault
Check PR Dependenciesis red until test(conformance) astubbs#242: one definition of correct for every client, anchored by Parallel Consumer itself #387 and its own parents merge.That is the dependency gate doing its job on a stack rung.
claude-reviewis red until somebody asks for a review; every rung below is in the same state.dups: clonesis red on jscpd's max-increase rule, for the residue described above. PMD CPD,the other engine, is clean.
Unit Testsfailed on a Maven Central read timeout fetchingfb-contrib, with no testhaving run - infrastructure, and the class this repository already has a solutions write-up for.
Integration Testsfailed onPartitionStateCommittedOffsetIT.committedOffsetRemoved[1] latestin
checkHowManyRecordsWithKeyPresent. Master state, not this PR's: the diff touches no fileunder
parallel-consumer-core, and the ambient probe's own autopsy reports "probe clean ... thefault is likely in the test itself, not consumer-group progress". It is a known member of the
load-tightness family, and this sighting is the interesting one -
[1] latestis the arm a merged2026-07-30 fix was written for, and it now fails at the assertion rather than the await it
fixed, which is exactly what
docs/plans/2026-08-05-001-investigate-committedoffset-latest-reflake.mdpredicted. Added to the ledger with that attribution rather than re-diagnosed or quarantined; the
registry wants an owning fix PR, which that diagnosis says has to be written first.
roadmap-stage: N/A - the
language-proxy-sidecarentry names #293 as its carrier and sits atlimited-pocon the strength of that PR's content. Ten client libraries that cannot yet reach an engine make no more advanced artifact exist, so its stage is unchanged.Checklist
"no record has crossed the wire" block and the overclaims it replaced,
AGENTS.md's module table,the conformance deferral note, the static-analyser survey's first item marked done, and a split-out
note for the per-module fragment mechanism this rung still does not import.
docs/features/- N/A - nothing here is auser-facing feature yet. These modules publish to no registry and cannot be depended on; the
release-documentation records they earn are the maturity and evidence rows, which are in this PR and
state in both directions what is and is not evidenced.
new conformance-module tests, and five sabotage arms with control arms either side.
ce-simplifyandce-code-reviewlocally - N/A - asked for@claude review thison the PRinstead. Most of the diff is code taken from feat(proxy) astubbs#242: a sidecar giving non-JVM runtimes key-ordered concurrency, in eleven languages #293 where it was already
reviewed, and the changes this rung made to it are named one by one above.