Skip to content

Commit ec432b4

Browse files
committed
ci(streams) #255: warm the Kafka sources jars the streams module unpacks
The streams module gets Kafka's `sources` and `test-sources` classifier jars through two `dependency:unpack` executions that name them as `<artifactItems>`. `prepare-deps` warms the cache with `dependency:go-offline`, which resolves the declared dependency graph - and an artifactItem is not in that graph, nor is either jar declared anywhere else. So both stayed cold, and every lane that built this module fetched them from Maven Central inside `generate-sources`. That put them on the CDN route lottery recorded in docs/solutions/build-errors/maven-central-timeout-azure-west-regions-2026-04-21.md: an exactly-240s read timeout, and re-running does not reliably help because the runner is often reassigned to the same region. It struck #394 and #395 - Unit and Integration both red at `unpack (unpack-kafka-streams-sources)` with zero tests run, so the lane named "Unit Tests" pointed at a subsystem that never reached compilation. Same plugin-resolves-for-itself class the proxy stack already warms three times over (protoc, scala-maven-plugin, the SpotBugs detector plugins), reached from a new direction: here the plugin resolving for itself is maven-dependency-plugin. The step follows that established pattern - version derived from the pom property rather than a literal, a non-version help:evaluate answer is a hard error naming the property, and the jars are asserted on disk afterwards. Rejected alternative: warming by building the module, which is what the Connect spike branch does. It works, but costs a full build of core inside this job, and that branch records why it cannot be shortened to a cheap phase-only walk - the module's test-scope resolution wants core's tests-classifier jar, which does not exist until core reaches `package`. Two jars are the whole gap and `kafka.version` is a real root-pom property, so naming them is cheaper and no more prone to rot. Verified by extracting the step's script from the workflow and running it: - Empty local repository, script otherwise verbatim: exit 0, and both jars arrive at exactly the paths the assertions name, byte-identical to the populated repository's copies. - Property renamed to `kafka.verzion`, one term changed and nothing else: exit 1, `::error::kafka.verzion did not resolve to a version from the root pom (got 'null object or invalid expression')`, and nothing fetched. The guard fires and names the property. - On-disk assertion is independently load-bearing: a run where the `get` goal succeeded but the asserted path was elsewhere exited 1 on the `test -s`, which is the case the exit code alone would have called a pass. - Sufficiency: after the warm, `-pl parallel-consumer-streams -am generate-test-sources` runs OFFLINE from a cleaned target and both unpack executions succeed, producing the unpacked sources. Offline makes any Central fetch a hard failure, so this shows the warmed jars satisfy the artifactItems resolve rather than merely sitting near it. `kafka.version` is the default every live lane builds at. The one job that overrides it, `test-kafka-compat`, is `if: false`; the comment says so, so re-enabling it comes with the note that its Kafka version falls outside this warm.
1 parent 3fa6b6f commit ec432b4

1 file changed

Lines changed: 58 additions & 0 deletions

File tree

.github/workflows/maven.yml

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -55,6 +55,64 @@ jobs:
5555
setup-java-Linux-x64-maven-
5656
- name: Download all dependencies
5757
run: ./mvnw --batch-mode -Pci dependency:go-offline -DincludeScope=test -U
58+
# The plugin-resolves-for-itself class, reached from a new direction: here the plugin doing its
59+
# own resolving is maven-dependency-plugin. `parallel-consumer-streams` unpacks Kafka's
60+
# `sources` and `test-sources` CLASSIFIER jars through two `dependency:unpack` executions that
61+
# name them as <artifactItems>, and go-offline resolves the DECLARED dependency graph - an
62+
# artifactItem is not in it, and neither jar is declared anywhere else. So both stay cold, and
63+
# every lane that builds this module fetches them from Central inside generate-sources, on the
64+
# CDN route lottery in
65+
# docs/solutions/build-errors/maven-central-timeout-azure-west-regions-2026-04-21.md: an
66+
# exactly-240s read timeout, and re-running does not reliably help because the runner is often
67+
# reassigned to the same region. Seen on astubbs/parallel-consumer#394 and
68+
# astubbs/parallel-consumer#395 - Unit AND Integration both red at
69+
# `unpack (unpack-kafka-streams-sources)` with ZERO tests run, so the lane called "Unit Tests"
70+
# names a subsystem that never got as far as compiling.
71+
#
72+
# Warmed by COORDINATE rather than by building the module. Building it warms the jars too, and
73+
# is what the Connect spike branch does (`-pl parallel-consumer-connect -am -DskipTests
74+
# package`) - but it costs a full build of core inside this job, and that branch records why it
75+
# cannot be shortened to a cheap phase-only walk: the module's test-scope resolution wants
76+
# core's tests-classifier jar, which does not exist until core reaches `package`. Two jars are
77+
# the whole gap here and `kafka.version` is a real root-pom property, so naming them is both
78+
# cheaper and no more prone to rot.
79+
#
80+
# Verified locally against an empty local repository: the two `get`s fetch exactly the two jars
81+
# the assertions name, after which `-pl parallel-consumer-streams -am generate-test-sources`
82+
# runs OFFLINE from a cleaned target and both unpack executions succeed - so what this warms is
83+
# sufficient for the artifactItems resolve, not merely adjacent to it.
84+
#
85+
# `kafka.version` is the default every live lane builds at. The one job that overrides it,
86+
# `test-kafka-compat`, is `if: false`; re-enabling it puts its Kafka version back outside this
87+
# warm. (astubbs#255)
88+
- name: Warm the Kafka sources jars the streams module unpacks
89+
run: |
90+
set -euo pipefail
91+
evaluate() {
92+
./mvnw -q --batch-mode -N \
93+
-Dexpression="$1" -DforceStdout \
94+
org.apache.maven.plugins:maven-help-plugin:3.5.2:evaluate 2>/dev/null | tail -1
95+
}
96+
# A renamed property makes help:evaluate print "null object or invalid expression", which
97+
# dependency:get would happily turn into a warm of the wrong coordinate - and warming
98+
# nothing is indistinguishable from a cache hit until it fails a job downstream. Require a
99+
# version-shaped answer rather than merely a non-empty one.
100+
require_version() {
101+
case "$2" in
102+
[0-9]*) : ;;
103+
*) echo "::error::$1 did not resolve to a version from the root pom (got '$2') - the Kafka sources cache warm would be a silent no-op"; exit 1 ;;
104+
esac
105+
}
106+
kafka_version=$(evaluate kafka.version)
107+
require_version kafka.version "$kafka_version"
108+
for classifier in sources test-sources; do
109+
./mvnw --batch-mode -ntp \
110+
org.apache.maven.plugins:maven-dependency-plugin:3.11.0:get \
111+
-Dartifact="org.apache.kafka:kafka-streams:${kafka_version}:jar:${classifier}"
112+
# Assert the jar is on disk: the exit code says the goal ran, only the file says the
113+
# cache this job exists to populate actually got populated.
114+
test -s ~/.m2/repository/org/apache/kafka/kafka-streams/"${kafka_version}"/kafka-streams-"${kafka_version}"-"${classifier}".jar
115+
done
58116
- name: Save Maven cache (rotating key)
59117
if: success()
60118
uses: actions/cache/save@v4

0 commit comments

Comments
 (0)