Skip to content

Commit 32576e5

Browse files
jason810496jason810496
committed
Add Java SDK capability manifest and compatibility matrix
The Language SDK conformance spec defines which TaskInstance states and capabilities a Language SDK may declare, but nothing lets an SDK say what it actually supports. Readers of the Java SDK docs cannot tell which parts of the spec the runtime implements today — and the gaps are real and moving (native Dag authoring, deferral, and the state stores are not there yet), so an unqualified "see the conformance spec" overstates what a Java task can do. The manifest is hand-authored YAML rather than something derived from the SDK's own sources: it describes the SDK instead of being part of it, so it has no business shipping in a user's runtime dependencies, and deriving it would mean a JDK and a Gradle run in a hook that only ever renders a table. It therefore sits above every subproject in settings.gradle.kts, where no source set can pick it up. Validation carries the weight a compiler would have: unknown keys are rejected along with missing ones, since it is the only thing standing between a typo and a wrong published table. A prek hook regenerates the contributor-facing README table and the Dokka module doc from the manifest, so a capability landing in the runtime is a one-line edit rather than three tables to update by hand. The shared schema, SDK registry, and renderer let the Go and TypeScript SDKs declare theirs the same way.
1 parent f9b72d8 commit 32576e5

11 files changed

Lines changed: 1024 additions & 0 deletions

‎.pre-commit-config.yaml‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -280,6 +280,21 @@ repos:
280280
(?x)
281281
^java-sdk/gradle\.properties$|
282282
^java-sdk/sdk/schema/schema\.json$
283+
- id: update-java-sdk-readme-matrix
284+
name: Update the Java SDK compatibility matrix in java-sdk/README.md and Dokka module doc
285+
entry: ./scripts/ci/prek/update_java_sdk_readme_matrix.py
286+
language: python
287+
files: >
288+
(?x)
289+
^java-sdk/capabilities\.yaml$|
290+
^java-sdk/gradle\.properties$|
291+
^java-sdk/README\.md$|
292+
^java-sdk/sdk/module\.md$|
293+
^scripts/ci/prek/lang_sdk_compat_matrix\.py$|
294+
^scripts/ci/prek/update_java_sdk_readme_matrix\.py$
295+
additional_dependencies: ['PyYAML>=6.0', 'rich>=13.6.0']
296+
pass_filenames: false
297+
require_serial: true
283298
- id: check-go-version-in-sync
284299
name: Check Go toolchain version is consistent across build files
285300
entry: ./scripts/ci/prek/check_go_version_in_sync.py

‎contributing-docs/30_new_language_sdk.rst‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -491,6 +491,42 @@ native Dags they are *not applicable* (``n/a``) rather than unsupported.
491491
The SDK exposes an object-storage API (an ``ObjectStoragePath`` equivalent) usable from
492492
native Dag code.
493493

494+
Compatibility matrix
495+
~~~~~~~~~~~~~~~~~~~~~~
496+
497+
The dimensions above are prose; each SDK also declares them *machine-readably* so its
498+
documentation can state what it actually supports without anyone hand-editing a table.
499+
An SDK owns one hand-authored manifest at the root of its own tree, and a prek hook
500+
propagates it outwards — nothing downstream is edited by hand:
501+
502+
.. code-block:: text
503+
504+
java-sdk/capabilities.yaml <- the only file you edit
505+
|
506+
| render: scripts/ci/prek/lang_sdk_compat_matrix.py
507+
| hook: update-java-sdk-readme-matrix
508+
|
509+
+--> java-sdk/README.md (contributor-facing)
510+
|
511+
+--> java-sdk/sdk/module.md (Dokka module doc -> the published API reference)
512+
|
513+
+--> airflow-core/docs/authoring-and-scheduling/language-sdks/index.rst
514+
(consolidated cross-SDK matrix — TODO, not generated yet)
515+
516+
The hook is a *regenerating* check: it rewrites its targets and exits non-zero when either was
517+
stale, so a drifted table fails the build and the fix is to re-stage the regenerated file.
518+
519+
Keep the manifest out of anything the SDK publishes — it describes the SDK rather than being part
520+
of it, and a capability declaration has no business in a user's runtime dependencies. For the Java
521+
SDK that means the file sits above every subproject in ``settings.gradle.kts``, so no source set or
522+
resource directory can pick it up.
523+
524+
When you add a Language SDK, register it in ``LANG_SDKS`` in
525+
``scripts/ci/prek/lang_sdk_compat_matrix.py``, write a ``capabilities.yaml`` in the same schema,
526+
and add the equivalent hook for your SDK. Adding or renaming a dimension means editing
527+
``STATE_DIMENSIONS`` / ``CAPABILITY_DIMENSIONS`` there **and** the prose above in the same PR — the
528+
renderer validates every manifest against that list, so the two cannot drift silently.
529+
494530

495531
Testing
496532
-------

‎java-sdk/README.md‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -558,6 +558,57 @@ Close the vote, **drop** the staging repository in Nexus, remove the `dist/dev`
558558
candidate, fix the issue, and cut the next RC (`...-rc2`). The released version
559559
stays the same (e.g. `<VERSION>`); only the RC counter in the tag increments.
560560

561+
## Compatibility matrix
562+
563+
Which Airflow TaskInstance states and capabilities this SDK supports. This table is generated from
564+
[`capabilities.yaml`](capabilities.yaml); the conformance dimensions are defined in the
565+
[Language SDK conformance spec](https://github.com/apache/airflow/blob/main/contributing-docs/30_new_language_sdk.rst).
566+
Do not edit the table by hand — edit `capabilities.yaml` and let the `update-java-sdk-readme-matrix`
567+
prek hook regenerate it.
568+
569+
<!-- BEGIN AUTO-GENERATED LANG-SDK COMPAT MATRIX -->
570+
571+
*Min. Airflow version: 3.3 · supervisor schema: 2026-06-16*
572+
573+
| Dimension | Tier | Supported | Since | Notes |
574+
|---|---|---|---|---|
575+
| **TaskInstance states** | | | | |
576+
| state: `success` | MUST | ✓ | 3.3 | |
577+
| state: `failed` | MUST | ✓ | 3.3 | |
578+
| state: `up_for_retry` | MUST | ✓ | 3.3 | RetryTask |
579+
| state: `skipped` | SHOULD | ✗ | – | runtime does not emit TaskState skipped yet |
580+
| state: `deferred` | MAY | ✗ | – | runtime does not emit DeferTask yet |
581+
| state: `up_for_reschedule` | MAY | ✗ | – | runtime does not emit RescheduleTask yet |
582+
| state: `awaiting_input` | MAY | ✗ | – | runtime does not emit AwaitInputTask yet |
583+
| state: `removed` | MAY | ✓ | 3.3 | |
584+
| **Runtime capabilities** | | | | |
585+
| capability: `mixed-lang-stub-target` | MUST | ✓ | 3.3 | @task.stub |
586+
| capability: `task-logging` | MUST | ✓ | 3.3 | SLF4J + JPL bridged to the task log |
587+
| capability: `xcom-read-write` | MUST | ✓ | 3.3 | |
588+
| capability: `connection-read` | MUST | ✓ | 3.3 | |
589+
| capability: `variable-read-write` | MUST | ✗ | – | getVariable only; no write over the comm socket yet |
590+
| capability: `self-contained-bundle` | MUST | ✓ | 3.3 | Airflow metadata embedded in the jar artifact |
591+
| capability: `task-state-store` | MAY | ✗ | – | no task-facing state-store API yet |
592+
| capability: `asset-state-store` | MAY | ✗ | – | no task-facing state-store API yet |
593+
| capability: `asset-event-emit` | MAY | ✗ | – | runtime does not emit asset events yet |
594+
| capability: `asset-event-read` | MAY | ✗ | – | no task-facing asset-event API yet |
595+
| **Native-Dag authoring** | | | | |
596+
| capability: `native-dag-authoring` | SHOULD | ✗ | – | native Dag authoring not implemented yet |
597+
| capability: `task-args` | MUST † | n/a | – | |
598+
| capability: `dag-params` | MUST † | n/a | – | |
599+
| capability: `taskflow-dependencies` | MUST † | n/a | – | |
600+
| capability: `branching` | SHOULD † | n/a | – | |
601+
| capability: `dag-test` | SHOULD † | n/a | – | |
602+
| capability: `task-group` | MAY † | n/a | – | |
603+
| capability: `dynamic-task-mapping` | MAY † | n/a | – | |
604+
| capability: `asset-inlets-outlets` | MAY † | n/a | – | |
605+
| capability: `asset-scheduling` | MAY † | n/a | – | |
606+
| capability: `object-store` | MAY † | n/a | – | |
607+
608+
*Marks: ✓ supported · ✗ not supported · n/a not applicable. A tier marked † applies only when `native-dag-authoring` is supported.*
609+
610+
<!-- END AUTO-GENERATED LANG-SDK COMPAT MATRIX -->
611+
561612
## Contributing
562613

563614
The user implements a Java application containing task methods annotated (or

‎java-sdk/capabilities.yaml‎

Lines changed: 125 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,125 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.
17+
---
18+
# Single source of truth for what the Java SDK supports. Update it when the runtime gains or loses a
19+
# conformance dimension; the compatibility tables in README.md and sdk/module.md are regenerated
20+
# from it by the update-java-sdk-readme-matrix prek hook. The normative meaning of every dimension
21+
# is defined in contributing-docs/30_new_language_sdk.rst, and the set of legal keys is enforced by
22+
# scripts/ci/prek/lang_sdk_compat_matrix.py.
23+
#
24+
# This lives outside every subproject in settings.gradle.kts so it is never packed into a published
25+
# artifact: it describes the SDK, it is not part of it.
26+
#
27+
# Each dimension carries `supported` and optionally `since` and `note`. Omit `since` when the
28+
# dimension is unsupported (declaring one there is an error) or when the release that introduced it
29+
# is unknown; omit `note` when there is nothing to add.
30+
sdk: java
31+
32+
min_airflow_version: "3.3"
33+
34+
# Keep in sync with airflowSupervisorSchemaVersion in gradle.properties, which is what stamps the
35+
# JAR manifest. The render hook fails if the two disagree.
36+
supervisor_schema_version: "2026-06-16"
37+
38+
# The runtime terminates a task with SucceedTask, RetryTask, or TaskState (failed/removed); it does
39+
# not yet emit skipped, DeferTask, RescheduleTask, or AwaitInputTask.
40+
states:
41+
success:
42+
supported: true
43+
since: "3.3"
44+
failed:
45+
supported: true
46+
since: "3.3"
47+
up_for_retry:
48+
supported: true
49+
since: "3.3"
50+
note: "RetryTask"
51+
skipped:
52+
supported: false
53+
note: "runtime does not emit TaskState skipped yet"
54+
deferred:
55+
supported: false
56+
note: "runtime does not emit DeferTask yet"
57+
up_for_reschedule:
58+
supported: false
59+
note: "runtime does not emit RescheduleTask yet"
60+
awaiting_input:
61+
supported: false
62+
note: "runtime does not emit AwaitInputTask yet"
63+
removed:
64+
supported: true
65+
since: "3.3"
66+
67+
# Runtime capabilities reflect the task-facing Client surface; native-Dag authoring is not
68+
# implemented yet, so every native capability is unsupported.
69+
capabilities:
70+
mixed-lang-stub-target:
71+
supported: true
72+
since: "3.3"
73+
note: "@task.stub"
74+
task-logging:
75+
supported: true
76+
since: "3.3"
77+
note: "SLF4J + JPL bridged to the task log"
78+
xcom-read-write:
79+
supported: true
80+
since: "3.3"
81+
connection-read:
82+
supported: true
83+
since: "3.3"
84+
variable-read-write:
85+
supported: false
86+
note: "getVariable only; no write over the comm socket yet"
87+
self-contained-bundle:
88+
supported: true
89+
since: "3.3"
90+
note: "Airflow metadata embedded in the jar artifact"
91+
task-state-store:
92+
supported: false
93+
note: "no task-facing state-store API yet"
94+
asset-state-store:
95+
supported: false
96+
note: "no task-facing state-store API yet"
97+
asset-event-emit:
98+
supported: false
99+
note: "runtime does not emit asset events yet"
100+
asset-event-read:
101+
supported: false
102+
note: "no task-facing asset-event API yet"
103+
native-dag-authoring:
104+
supported: false
105+
note: "native Dag authoring not implemented yet"
106+
task-args:
107+
supported: false
108+
dag-params:
109+
supported: false
110+
taskflow-dependencies:
111+
supported: false
112+
branching:
113+
supported: false
114+
dag-test:
115+
supported: false
116+
task-group:
117+
supported: false
118+
dynamic-task-mapping:
119+
supported: false
120+
asset-inlets-outlets:
121+
supported: false
122+
asset-scheduling:
123+
supported: false
124+
object-store:
125+
supported: false

‎java-sdk/sdk/build.gradle.kts‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -265,7 +265,15 @@ sourceSets {
265265

266266
dokka {
267267
moduleVersion.set(project.version.toString())
268+
pluginsConfiguration.html {
269+
// Widens the narrow compatibility-matrix columns; see the comments in the file.
270+
customStyleSheets.from(layout.projectDirectory.file("dokka/matrix.css"))
271+
}
268272
dokkaSourceSets.configureEach {
273+
// Module-level documentation, including the generated Language SDK compatibility matrix.
274+
// Dokka rejects the file unless "# Module sdk" is its very first line, so module.md carries
275+
// the ASF license header just below the heading instead of above it.
276+
includes.from("module.md")
269277
// Suppress everything in 'execution' since it's implementation detail.
270278
perPackageOption {
271279
matchingRegex = """org\.apache\.airflow\.sdk\.execution.*"""

‎java-sdk/sdk/dokka/matrix.css‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
/*!
2+
* Licensed to the Apache Software Foundation (ASF) under one
3+
* or more contributor license agreements. See the NOTICE file
4+
* distributed with this work for additional information
5+
* regarding copyright ownership. The ASF licenses this file
6+
* to you under the Apache License, Version 2.0 (the
7+
* "License"); you may not use this file except in compliance
8+
* with the License. You may obtain a copy of the License at
9+
*
10+
* http://www.apache.org/licenses/LICENSE-2.0
11+
*
12+
* Unless required by applicable law or agreed to in writing,
13+
* software distributed under the License is distributed on an
14+
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
* KIND, either express or implied. See the License for the
16+
* specific language governing permissions and limitations
17+
* under the License.
18+
*/
19+
20+
/* Dokka sizes content-table cells with `min-width: 40px` against ~31px of horizontal padding, so the
21+
narrow columns of the Language SDK compatibility matrix collapse until their headers and values
22+
break mid-word ("Supported", "MUST †").
23+
24+
These selectors necessarily apply to every content table in the API reference, because a table
25+
generated from Markdown carries no class of its own. They are therefore deliberately written to
26+
only *raise a floor*, never to forbid wrapping: a header stays on one line (headers are short in
27+
any table), and the narrow value columns get a minimum width instead of `nowrap`. An unrelated
28+
table with long prose in those columns still wraps as before rather than overflowing. */
29+
30+
.table--container th {
31+
white-space: nowrap;
32+
}
33+
34+
/* Tier ("SHOULD †"), Supported ("n/a") and Since ("3.3") in the compatibility matrix. */
35+
.table--container td:nth-child(2),
36+
.table--container td:nth-child(3) {
37+
min-width: 9ch;
38+
}
39+
40+
.table--container td:nth-child(4) {
41+
min-width: 6ch;
42+
}

‎java-sdk/sdk/module.md‎

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
# Module sdk
2+
3+
<!--
4+
Licensed to the Apache Software Foundation (ASF) under one
5+
or more contributor license agreements. See the NOTICE file
6+
distributed with this work for additional information
7+
regarding copyright ownership. The ASF licenses this file
8+
to you under the Apache License, Version 2.0 (the
9+
"License"); you may not use this file except in compliance
10+
with the License. You may obtain a copy of the License at
11+
12+
http://www.apache.org/licenses/LICENSE-2.0
13+
14+
Unless required by applicable law or agreed to in writing,
15+
software distributed under the License is distributed on an
16+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
17+
KIND, either express or implied. See the License for the
18+
specific language governing permissions and limitations
19+
under the License.
20+
-->
21+
22+
The Apache Airflow Java SDK — author and run Airflow task implementations in JVM languages.
23+
24+
## Language SDK compatibility matrix
25+
26+
The table below is generated from `java-sdk/capabilities.yaml` and shows which Airflow Language SDK
27+
conformance dimensions the Java SDK currently supports. The normative meaning of each dimension is
28+
defined in the Language SDK conformance specification
29+
(`contributing-docs/30_new_language_sdk.rst`).
30+
31+
<!-- BEGIN AUTO-GENERATED LANG-SDK COMPAT MATRIX -->
32+
33+
*Min. Airflow version: 3.3 · supervisor schema: 2026-06-16*
34+
35+
| Dimension | Tier | Supported | Since | Notes |
36+
|---|---|---|---|---|
37+
| **TaskInstance states** | | | | |
38+
| state: `success` | MUST | ✓ | 3.3 | |
39+
| state: `failed` | MUST | ✓ | 3.3 | |
40+
| state: `up_for_retry` | MUST | ✓ | 3.3 | RetryTask |
41+
| state: `skipped` | SHOULD | ✗ | – | runtime does not emit TaskState skipped yet |
42+
| state: `deferred` | MAY | ✗ | – | runtime does not emit DeferTask yet |
43+
| state: `up_for_reschedule` | MAY | ✗ | – | runtime does not emit RescheduleTask yet |
44+
| state: `awaiting_input` | MAY | ✗ | – | runtime does not emit AwaitInputTask yet |
45+
| state: `removed` | MAY | ✓ | 3.3 | |
46+
| **Runtime capabilities** | | | | |
47+
| capability: `mixed-lang-stub-target` | MUST | ✓ | 3.3 | @task.stub |
48+
| capability: `task-logging` | MUST | ✓ | 3.3 | SLF4J + JPL bridged to the task log |
49+
| capability: `xcom-read-write` | MUST | ✓ | 3.3 | |
50+
| capability: `connection-read` | MUST | ✓ | 3.3 | |
51+
| capability: `variable-read-write` | MUST | ✗ | – | getVariable only; no write over the comm socket yet |
52+
| capability: `self-contained-bundle` | MUST | ✓ | 3.3 | Airflow metadata embedded in the jar artifact |
53+
| capability: `task-state-store` | MAY | ✗ | – | no task-facing state-store API yet |
54+
| capability: `asset-state-store` | MAY | ✗ | – | no task-facing state-store API yet |
55+
| capability: `asset-event-emit` | MAY | ✗ | – | runtime does not emit asset events yet |
56+
| capability: `asset-event-read` | MAY | ✗ | – | no task-facing asset-event API yet |
57+
| **Native-Dag authoring** | | | | |
58+
| capability: `native-dag-authoring` | SHOULD | ✗ | – | native Dag authoring not implemented yet |
59+
| capability: `task-args` | MUST † | n/a | – | |
60+
| capability: `dag-params` | MUST † | n/a | – | |
61+
| capability: `taskflow-dependencies` | MUST † | n/a | – | |
62+
| capability: `branching` | SHOULD † | n/a | – | |
63+
| capability: `dag-test` | SHOULD † | n/a | – | |
64+
| capability: `task-group` | MAY † | n/a | – | |
65+
| capability: `dynamic-task-mapping` | MAY † | n/a | – | |
66+
| capability: `asset-inlets-outlets` | MAY † | n/a | – | |
67+
| capability: `asset-scheduling` | MAY † | n/a | – | |
68+
| capability: `object-store` | MAY † | n/a | – | |
69+
70+
*Marks: ✓ supported · ✗ not supported · n/a not applicable. A tier marked † applies only when `native-dag-authoring` is supported.*
71+
72+
<!-- END AUTO-GENERATED LANG-SDK COMPAT MATRIX -->

0 commit comments

Comments
 (0)