Skip to content

Commit a8e0e13

Browse files
jason810496jason810496
committed
Add Java SDK capability manifest and compatibility matrix
The Language SDK conformance spec defines which TaskInstance states and capabilities a Language SDK may declare, but nothing lets an SDK say what it actually supports. Readers of the Java SDK docs cannot tell which parts of the spec the runtime implements today — and the gaps are real and moving (native Dag authoring, deferral, and the state stores are not there yet), so an unqualified "see the conformance spec" overstates what a Java task can do. The manifest is hand-authored YAML rather than something derived from the SDK's own sources: it describes the SDK instead of being part of it, so it has no business shipping in a user's runtime dependencies, and deriving it would mean a JDK and a Gradle run in a hook that only ever renders a table. It therefore sits above every subproject in settings.gradle.kts, where no source set can pick it up. Validation carries the weight a compiler would have: unknown keys are rejected along with missing ones, since it is the only thing standing between a typo and a wrong published table. A prek hook regenerates the contributor-facing README table and the Dokka module doc from the manifest, so a capability landing in the runtime is a one-line edit rather than three tables to update by hand. The shared schema, SDK registry, and renderer let the Go and TypeScript SDKs declare theirs the same way.
1 parent f9b72d8 commit a8e0e13

11 files changed

Lines changed: 1001 additions & 0 deletions

‎.pre-commit-config.yaml‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -280,6 +280,21 @@ repos:
280280
(?x)
281281
^java-sdk/gradle\.properties$|
282282
^java-sdk/sdk/schema/schema\.json$
283+
- id: update-java-sdk-readme-matrix
284+
name: Update the Java SDK compatibility matrix in java-sdk/README.md and Dokka module doc
285+
entry: ./scripts/ci/prek/update_java_sdk_readme_matrix.py
286+
language: python
287+
files: >
288+
(?x)
289+
^java-sdk/capabilities\.yaml$|
290+
^java-sdk/gradle\.properties$|
291+
^java-sdk/README\.md$|
292+
^java-sdk/sdk/module\.md$|
293+
^scripts/ci/prek/lang_sdk_compat_matrix\.py$|
294+
^scripts/ci/prek/update_java_sdk_readme_matrix\.py$
295+
additional_dependencies: ['PyYAML>=6.0', 'rich>=13.6.0']
296+
pass_filenames: false
297+
require_serial: true
283298
- id: check-go-version-in-sync
284299
name: Check Go toolchain version is consistent across build files
285300
entry: ./scripts/ci/prek/check_go_version_in_sync.py

‎contributing-docs/30_new_language_sdk.rst‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -491,6 +491,32 @@ native Dags they are *not applicable* (``n/a``) rather than unsupported.
491491
The SDK exposes an object-storage API (an ``ObjectStoragePath`` equivalent) usable from
492492
native Dag code.
493493

494+
Compatibility matrix
495+
~~~~~~~~~~~~~~~~~~~~
496+
497+
The dimensions above are prose; each SDK also declares them *machine-readably* in one
498+
hand-authored ``<sdk>/capabilities.yaml``, from which a prek hook generates every table:
499+
500+
.. code-block:: text
501+
502+
java-sdk/capabilities.yaml <- the only file you edit
503+
|
504+
| hook: update-java-sdk-readme-matrix
505+
|
506+
+--> java-sdk/README.md (contributor-facing)
507+
+--> java-sdk/sdk/module.md (Dokka -> the published API reference)
508+
+--> airflow-core/docs/authoring-and-scheduling/language-sdks/index.rst
509+
(consolidated cross-SDK matrix — TODO, not generated yet)
510+
511+
The hook rewrites its targets and exits non-zero when either was stale, so a drifted table fails
512+
the build. Keep the manifest out of whatever the SDK publishes — it describes the SDK rather than
513+
being part of it; for Java that means sitting above every subproject in ``settings.gradle.kts``.
514+
515+
To add an SDK, register it in ``LANG_SDKS`` in ``scripts/ci/prek/lang_sdk_compat_matrix.py``, write
516+
a ``capabilities.yaml`` in the same schema, and add the equivalent hook. Adding or renaming a
517+
dimension means editing ``STATE_DIMENSIONS`` / ``CAPABILITY_DIMENSIONS`` there **and** the prose
518+
above in the same PR — the renderer validates every manifest against that list.
519+
494520

495521
Testing
496522
-------

‎java-sdk/README.md‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -558,6 +558,57 @@ Close the vote, **drop** the staging repository in Nexus, remove the `dist/dev`
558558
candidate, fix the issue, and cut the next RC (`...-rc2`). The released version
559559
stays the same (e.g. `<VERSION>`); only the RC counter in the tag increments.
560560

561+
## Compatibility matrix
562+
563+
Which Airflow TaskInstance states and capabilities this SDK supports. This table is generated from
564+
[`capabilities.yaml`](capabilities.yaml); the conformance dimensions are defined in the
565+
[Language SDK conformance spec](https://github.com/apache/airflow/blob/main/contributing-docs/30_new_language_sdk.rst).
566+
Do not edit the table by hand — edit `capabilities.yaml` and let the `update-java-sdk-readme-matrix`
567+
prek hook regenerate it.
568+
569+
<!-- BEGIN AUTO-GENERATED LANG-SDK COMPAT MATRIX -->
570+
571+
*Min. Airflow version: 3.3 · supervisor schema: 2026-06-16*
572+
573+
| Dimension | Tier | Supported | Since | Notes |
574+
|---|---|---|---|---|
575+
| **TaskInstance states** | | | | |
576+
| state: `success` | MUST | ✓ | 3.3 | |
577+
| state: `failed` | MUST | ✓ | 3.3 | |
578+
| state: `up_for_retry` | MUST | ✓ | 3.3 | RetryTask |
579+
| state: `skipped` | SHOULD | ✗ | – | runtime does not emit TaskState skipped yet |
580+
| state: `deferred` | MAY | ✗ | – | runtime does not emit DeferTask yet |
581+
| state: `up_for_reschedule` | MAY | ✗ | – | runtime does not emit RescheduleTask yet |
582+
| state: `awaiting_input` | MAY | ✗ | – | runtime does not emit AwaitInputTask yet |
583+
| state: `removed` | MAY | ✓ | 3.3 | |
584+
| **Runtime capabilities** | | | | |
585+
| capability: `mixed-lang-stub-target` | MUST | ✓ | 3.3 | @task.stub |
586+
| capability: `task-logging` | MUST | ✓ | 3.3 | SLF4J + JPL bridged to the task log |
587+
| capability: `xcom-read-write` | MUST | ✓ | 3.3 | |
588+
| capability: `connection-read` | MUST | ✓ | 3.3 | |
589+
| capability: `variable-read-write` | MUST | ✗ | – | getVariable only; no write over the comm socket yet |
590+
| capability: `self-contained-bundle` | MUST | ✓ | 3.3 | Airflow metadata embedded in the jar artifact |
591+
| capability: `task-state-store` | MAY | ✗ | – | no task-facing state-store API yet |
592+
| capability: `asset-state-store` | MAY | ✗ | – | no task-facing state-store API yet |
593+
| capability: `asset-event-emit` | MAY | ✗ | – | runtime does not emit asset events yet |
594+
| capability: `asset-event-read` | MAY | ✗ | – | no task-facing asset-event API yet |
595+
| **Native-Dag authoring** | | | | |
596+
| capability: `native-dag-authoring` | SHOULD | ✗ | – | native Dag authoring not implemented yet |
597+
| capability: `task-args` | MUST † | n/a | – | |
598+
| capability: `dag-params` | MUST † | n/a | – | |
599+
| capability: `taskflow-dependencies` | MUST † | n/a | – | |
600+
| capability: `branching` | SHOULD † | n/a | – | |
601+
| capability: `dag-test` | SHOULD † | n/a | – | |
602+
| capability: `task-group` | MAY † | n/a | – | |
603+
| capability: `dynamic-task-mapping` | MAY † | n/a | – | |
604+
| capability: `asset-inlets-outlets` | MAY † | n/a | – | |
605+
| capability: `asset-scheduling` | MAY † | n/a | – | |
606+
| capability: `object-store` | MAY † | n/a | – | |
607+
608+
*Marks: ✓ supported · ✗ not supported · n/a not applicable. A tier marked † applies only when `native-dag-authoring` is supported.*
609+
610+
<!-- END AUTO-GENERATED LANG-SDK COMPAT MATRIX -->
611+
561612
## Contributing
562613

563614
The user implements a Java application containing task methods annotated (or

‎java-sdk/capabilities.yaml‎

Lines changed: 113 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,113 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.
17+
---
18+
sdk: java
19+
20+
min_airflow_version: "3.3"
21+
22+
# Keep in sync with airflowSupervisorSchemaVersion in gradle.properties, which is what stamps the
23+
# JAR manifest. The render hook fails if the two disagree.
24+
supervisor_schema_version: "2026-06-16"
25+
26+
# The runtime terminates a task with SucceedTask, RetryTask, or TaskState (failed/removed); it does
27+
# not yet emit skipped, DeferTask, RescheduleTask, or AwaitInputTask.
28+
states:
29+
success:
30+
supported: true
31+
since: "3.3"
32+
failed:
33+
supported: true
34+
since: "3.3"
35+
up_for_retry:
36+
supported: true
37+
since: "3.3"
38+
note: "RetryTask"
39+
skipped:
40+
supported: false
41+
note: "runtime does not emit TaskState skipped yet"
42+
deferred:
43+
supported: false
44+
note: "runtime does not emit DeferTask yet"
45+
up_for_reschedule:
46+
supported: false
47+
note: "runtime does not emit RescheduleTask yet"
48+
awaiting_input:
49+
supported: false
50+
note: "runtime does not emit AwaitInputTask yet"
51+
removed:
52+
supported: true
53+
since: "3.3"
54+
55+
# Runtime capabilities reflect the task-facing Client surface; native-Dag authoring is not
56+
# implemented yet, so every native capability is unsupported.
57+
capabilities:
58+
mixed-lang-stub-target:
59+
supported: true
60+
since: "3.3"
61+
note: "@task.stub"
62+
task-logging:
63+
supported: true
64+
since: "3.3"
65+
note: "SLF4J + JPL bridged to the task log"
66+
xcom-read-write:
67+
supported: true
68+
since: "3.3"
69+
connection-read:
70+
supported: true
71+
since: "3.3"
72+
variable-read-write:
73+
supported: false
74+
note: "getVariable only; no write over the comm socket yet"
75+
self-contained-bundle:
76+
supported: true
77+
since: "3.3"
78+
note: "Airflow metadata embedded in the jar artifact"
79+
task-state-store:
80+
supported: false
81+
note: "no task-facing state-store API yet"
82+
asset-state-store:
83+
supported: false
84+
note: "no task-facing state-store API yet"
85+
asset-event-emit:
86+
supported: false
87+
note: "runtime does not emit asset events yet"
88+
asset-event-read:
89+
supported: false
90+
note: "no task-facing asset-event API yet"
91+
native-dag-authoring:
92+
supported: false
93+
note: "native Dag authoring not implemented yet"
94+
task-args:
95+
supported: false
96+
dag-params:
97+
supported: false
98+
taskflow-dependencies:
99+
supported: false
100+
branching:
101+
supported: false
102+
dag-test:
103+
supported: false
104+
task-group:
105+
supported: false
106+
dynamic-task-mapping:
107+
supported: false
108+
asset-inlets-outlets:
109+
supported: false
110+
asset-scheduling:
111+
supported: false
112+
object-store:
113+
supported: false

‎java-sdk/sdk/build.gradle.kts‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -265,7 +265,15 @@ sourceSets {
265265

266266
dokka {
267267
moduleVersion.set(project.version.toString())
268+
pluginsConfiguration.html {
269+
// Widens the narrow compatibility-matrix columns; see the comments in the file.
270+
customStyleSheets.from(layout.projectDirectory.file("dokka/matrix.css"))
271+
}
268272
dokkaSourceSets.configureEach {
273+
// Module-level documentation, including the generated Language SDK compatibility matrix.
274+
// Dokka rejects the file unless "# Module sdk" is its very first line, so module.md carries
275+
// the ASF license header just below the heading instead of above it.
276+
includes.from("module.md")
269277
// Suppress everything in 'execution' since it's implementation detail.
270278
perPackageOption {
271279
matchingRegex = """org\.apache\.airflow\.sdk\.execution.*"""

‎java-sdk/sdk/dokka/matrix.css‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
/*!
2+
* Licensed to the Apache Software Foundation (ASF) under one
3+
* or more contributor license agreements. See the NOTICE file
4+
* distributed with this work for additional information
5+
* regarding copyright ownership. The ASF licenses this file
6+
* to you under the Apache License, Version 2.0 (the
7+
* "License"); you may not use this file except in compliance
8+
* with the License. You may obtain a copy of the License at
9+
*
10+
* http://www.apache.org/licenses/LICENSE-2.0
11+
*
12+
* Unless required by applicable law or agreed to in writing,
13+
* software distributed under the License is distributed on an
14+
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
* KIND, either express or implied. See the License for the
16+
* specific language governing permissions and limitations
17+
* under the License.
18+
*/
19+
20+
/* Dokka sizes content-table cells with `min-width: 40px` against ~31px of horizontal padding, so the
21+
narrow columns of the Language SDK compatibility matrix collapse until their headers and values
22+
break mid-word ("Supported", "MUST †").
23+
24+
These selectors necessarily apply to every content table in the API reference, because a table
25+
generated from Markdown carries no class of its own. They are therefore deliberately written to
26+
only *raise a floor*, never to forbid wrapping: a header stays on one line (headers are short in
27+
any table), and the narrow value columns get a minimum width instead of `nowrap`. An unrelated
28+
table with long prose in those columns still wraps as before rather than overflowing. */
29+
30+
.table--container th {
31+
white-space: nowrap;
32+
}
33+
34+
/* Tier ("SHOULD †"), Supported ("n/a") and Since ("3.3") in the compatibility matrix. */
35+
.table--container td:nth-child(2),
36+
.table--container td:nth-child(3) {
37+
min-width: 9ch;
38+
}
39+
40+
.table--container td:nth-child(4) {
41+
min-width: 6ch;
42+
}

‎java-sdk/sdk/module.md‎

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
# Module sdk
2+
3+
<!--
4+
Licensed to the Apache Software Foundation (ASF) under one
5+
or more contributor license agreements. See the NOTICE file
6+
distributed with this work for additional information
7+
regarding copyright ownership. The ASF licenses this file
8+
to you under the Apache License, Version 2.0 (the
9+
"License"); you may not use this file except in compliance
10+
with the License. You may obtain a copy of the License at
11+
12+
http://www.apache.org/licenses/LICENSE-2.0
13+
14+
Unless required by applicable law or agreed to in writing,
15+
software distributed under the License is distributed on an
16+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
17+
KIND, either express or implied. See the License for the
18+
specific language governing permissions and limitations
19+
under the License.
20+
-->
21+
22+
The Apache Airflow Java SDK — author and run Airflow task implementations in JVM languages.
23+
24+
## Language SDK compatibility matrix
25+
26+
Which Airflow TaskInstance states and capabilities the Java SDK currently supports. The normative
27+
meaning of each dimension is defined in the
28+
[Language SDK conformance specification](https://github.com/apache/airflow/blob/main/contributing-docs/30_new_language_sdk.rst).
29+
30+
<!-- BEGIN AUTO-GENERATED LANG-SDK COMPAT MATRIX -->
31+
32+
*Min. Airflow version: 3.3 · supervisor schema: 2026-06-16*
33+
34+
| Dimension | Tier | Supported | Since | Notes |
35+
|---|---|---|---|---|
36+
| **TaskInstance states** | | | | |
37+
| state: `success` | MUST | ✓ | 3.3 | |
38+
| state: `failed` | MUST | ✓ | 3.3 | |
39+
| state: `up_for_retry` | MUST | ✓ | 3.3 | RetryTask |
40+
| state: `skipped` | SHOULD | ✗ | – | runtime does not emit TaskState skipped yet |
41+
| state: `deferred` | MAY | ✗ | – | runtime does not emit DeferTask yet |
42+
| state: `up_for_reschedule` | MAY | ✗ | – | runtime does not emit RescheduleTask yet |
43+
| state: `awaiting_input` | MAY | ✗ | – | runtime does not emit AwaitInputTask yet |
44+
| state: `removed` | MAY | ✓ | 3.3 | |
45+
| **Runtime capabilities** | | | | |
46+
| capability: `mixed-lang-stub-target` | MUST | ✓ | 3.3 | @task.stub |
47+
| capability: `task-logging` | MUST | ✓ | 3.3 | SLF4J + JPL bridged to the task log |
48+
| capability: `xcom-read-write` | MUST | ✓ | 3.3 | |
49+
| capability: `connection-read` | MUST | ✓ | 3.3 | |
50+
| capability: `variable-read-write` | MUST | ✗ | – | getVariable only; no write over the comm socket yet |
51+
| capability: `self-contained-bundle` | MUST | ✓ | 3.3 | Airflow metadata embedded in the jar artifact |
52+
| capability: `task-state-store` | MAY | ✗ | – | no task-facing state-store API yet |
53+
| capability: `asset-state-store` | MAY | ✗ | – | no task-facing state-store API yet |
54+
| capability: `asset-event-emit` | MAY | ✗ | – | runtime does not emit asset events yet |
55+
| capability: `asset-event-read` | MAY | ✗ | – | no task-facing asset-event API yet |
56+
| **Native-Dag authoring** | | | | |
57+
| capability: `native-dag-authoring` | SHOULD | ✗ | – | native Dag authoring not implemented yet |
58+
| capability: `task-args` | MUST † | n/a | – | |
59+
| capability: `dag-params` | MUST † | n/a | – | |
60+
| capability: `taskflow-dependencies` | MUST † | n/a | – | |
61+
| capability: `branching` | SHOULD † | n/a | – | |
62+
| capability: `dag-test` | SHOULD † | n/a | – | |
63+
| capability: `task-group` | MAY † | n/a | – | |
64+
| capability: `dynamic-task-mapping` | MAY † | n/a | – | |
65+
| capability: `asset-inlets-outlets` | MAY † | n/a | – | |
66+
| capability: `asset-scheduling` | MAY † | n/a | – | |
67+
| capability: `object-store` | MAY † | n/a | – | |
68+
69+
*Marks: ✓ supported · ✗ not supported · n/a not applicable. A tier marked † applies only when `native-dag-authoring` is supported.*
70+
71+
<!-- END AUTO-GENERATED LANG-SDK COMPAT MATRIX -->

0 commit comments

Comments
 (0)