Skip to content

Commit 1b902cc

Browse files
jason810496jason810496
authored
Add TypeScript SDK API reference docs with TypeDoc (#70812)
* Add TypeScript SDK API reference docs via sphinx-js * Render flattened TSDoc link tags as inline code Wrapping the referenced identifier in backticks makes it render as an inline literal and keeps sphinxcontrib-spelling from treating the identifier as prose, so the build no longer fails on names like getVariableOrThrow and the spelling wordlist stays clean. * Build priority docs packages fully even in spellcheck-only mode Packages whose inventory cannot be fetched (typically brand-new docs packages not yet published) are built first precisely so that their objects.inv exists for every other package's intersphinx mapping. The Sphinx spelling builder never writes objects.inv, so a spellcheck-only pre-build left the inventory missing, every other package failed inventory resolution, and the retry logic rebuilt the whole docs set in up to three additional full passes - the --spellcheck-only CI job ran 2h27m while the equivalent --docs-only job finished in 32m. * Reword TaskContext.signal TSDoc to avoid non-dictionary word "abortable" is not in the spellcheck dictionary and docs prose should not need a wordlist entry for a phrasing that can simply be avoided. * Resolve the ts-sdk version in publish-docs when no stable.txt is staged publish-docs resolves every package's version up front via get_package_version_possibly_from_stable_txt, which knew nothing about ts-sdk, so the "Publish documentation and validate versions" CI job failed with "Unsupported package: ts-sdk". Fall back to the version in ts-sdk/package.json the same way task-sdk falls back to its source version, matching what DocsPublisher already does. * Build the TypeScript SDK API reference with TypeDoc instead of sphinx-js sphinx-js held the docs toolchain back to TypeDoc 0.25 and TypeScript 5.4 — several majors behind the TypeScript 6 the SDK itself is built with — and its comment renderer could not represent TSDoc inline tags, so a bundled TypeDoc plugin had to flatten every `{@link}` into literal text just to keep the build from crashing. Both the version cap and the workaround existed only to bridge TypeScript into Sphinx. Rendering with TypeDoc directly removes that bridge. The toolchain tracks the same TypeScript the SDK compiles with, the reference is no longer squeezed through a hand-maintained RST mirror of the public API that had to be updated whenever an export changed, and cross-references resolve natively. This follows the precedent already set by the Java SDK, whose Javadoc is built with Dokka rather than Sphinx and staged into the same publish pipeline. The package root is the only entry point given to TypeDoc. It already re-exports everything `./coordinator` exposes, so listing both produced two navigation sections that documented one API between them, with no symbol unique to either. Dropping the Python dependency also releases the constraint it dragged in: sphinx-js required cattrs < 25, which had pinned the whole workspace back from cattrs 26.1.0 to 24.1.3. The lock file returns to what main resolves. * Publish the TypeScript SDK API reference automatically The reference could be built but never published on its own. Three pieces were missing relative to the Java SDK, whose Javadoc already runs this path end to end. `ts-sdk` was absent from NON_SHORT_NAME_PACKAGES, so the S3 publisher treated it as a provider short name and expanded it to a provider distribution that does not exist, rather than publishing it as a top-level package. Nothing built the reference on a pull request. A break in the docs toolchain or the documented sources would therefore only surface during a release, which is the worst moment to discover it. The new job gates on its own selective-check output rather than reusing the ts-sdk test trigger: that group deliberately ignores Markdown, because a README edit should not run the SDK test suite, but the landing page is Markdown and the reference is generated from the sources, so the docs build has to react to both. Finally, the release procedure was undocumented, leaving no way for a release manager to know the docs need a separate, deliberate publish — providers-only publish waves intentionally skip the SDKs. * Fix TypeDoc build failing to resolve esbuild and @msgpack/msgpack The docs tsconfig included every file under ts-sdk/src/ as a program root, which pulled in cli/pack.ts — a CLI-only module unreachable from the public API that needs the optional esbuild peer dependency the isolated docs toolchain never installs. Rooting the program at the entry point instead lets TypeScript's own import graph decide what belongs in the type-checked program, so unreachable internal modules are no longer swept in. @msgpack/msgpack is different: coordinator/frames.ts is a real transitive dependency of the public startCoordinator export (via comm-channel.ts), so type-checking it needs the package installed, not excluded. * Fix @msgpack/msgpack still unresolved in a genuinely clean docs install The previous fix added @msgpack/msgpack to ts-sdk/docs/package.json, but that alone can't work: Node's module resolution walks node_modules ancestors of the *importing file* (ts-sdk/src/coordinator/frames.ts), and ts-sdk/docs/ is a sibling of ts-sdk/src/, never an ancestor. The fix only appeared to succeed locally because ts-sdk/node_modules already existed from an unrelated pnpm install and was picked up by that ancestor walk — a genuinely fresh checkout (matching CI, and reported here) has no such directory. Redirect the bare specifier with a `paths` entry instead, pointing at where npm actually installs the package for this toolchain. A path substitution to the package's directory doesn't trigger its package.json main/types resolution under node16/nodenext module resolution (verified with --traceResolution), so the path names the concrete declaration file the package's own `types` field already points to. * Restore import statements in TypeScript SDK doc examples * Address TypeScript SDK docs review feedback Keeps selective-checks accurate for the new ts-sdk docs build (tsconfig.json and package.json changes now trigger it, and docs-only package.json/lock changes no longer trigger the unrelated supervisor-schema check), fixes the alphabetical ordering of the new run-ts-sdk-docs row, simplifies the @msgpack/msgpack path mapping to resolve through its package.json instead of a hardcoded internal path, wraps the long tsconfig.json comment, and aligns the README's cancellation section wording with task.ts's TSDoc.
1 parent b099814 commit 1b902cc

31 files changed

Lines changed: 4519 additions & 40 deletions

‎.github/workflows/ci-amd.yml‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -138,6 +138,7 @@ jobs:
138138
run-coverage: ${{ steps.source-run-info.outputs.run-coverage }}
139139
run-go-sdk-tests: ${{ steps.selective-checks.outputs.run-go-sdk-tests }}
140140
run-java-sdk-tests: ${{ steps.selective-checks.outputs.run-java-sdk-tests }}
141+
run-ts-sdk-docs: ${{ steps.selective-checks.outputs.run-ts-sdk-docs }}
141142
run-helm-tests: ${{ steps.selective-checks.outputs.run-helm-tests }}
142143
run-kubernetes-tests: ${{ steps.selective-checks.outputs.run-kubernetes-tests }}
143144
run-mypy-providers: ${{ steps.selective-checks.outputs.run-mypy-providers }}
@@ -1082,6 +1083,38 @@ jobs:
10821083
if-no-files-found: error
10831084
overwrite: 'true'
10841085

1086+
build-ts-sdk-docs:
1087+
name: "TypeScript SDK docs"
1088+
needs: [build-info]
1089+
runs-on: ${{ fromJSON(needs.build-info.outputs.runner-type) }}
1090+
timeout-minutes: 30
1091+
permissions:
1092+
contents: read
1093+
packages: read
1094+
if: needs.build-info.outputs.run-ts-sdk-docs == 'true'
1095+
env:
1096+
GITHUB_REPOSITORY: ${{ github.repository }}
1097+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
1098+
GITHUB_USERNAME: ${{ github.actor }}
1099+
VERBOSE: "true"
1100+
steps:
1101+
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
1102+
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
1103+
with:
1104+
persist-credentials: false
1105+
- name: "Install Breeze"
1106+
uses: ./.github/actions/breeze
1107+
- name: "Build TypeScript SDK API reference"
1108+
run: breeze build-docs --sdk-docs-only --sdk=typescript
1109+
- name: "Upload TypeScript SDK docs artifact"
1110+
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
1111+
with:
1112+
name: ts-sdk-docs
1113+
path: generated/_build/docs/ts-sdk/
1114+
retention-days: 7
1115+
if-no-files-found: error
1116+
overwrite: 'true'
1117+
10851118
tests-airflow-ctl:
10861119
name: "Airflow CTL tests"
10871120
uses: ./.github/workflows/airflow-distributions-tests.yml
@@ -1137,6 +1170,7 @@ jobs:
11371170
- tests-go-sdk
11381171
- tests-java-sdk
11391172
- build-java-sdk-docs
1173+
- build-ts-sdk-docs
11401174
- tests-with-lowest-direct-resolution-core
11411175
- tests-with-lowest-direct-resolution-providers
11421176
uses: ./.github/workflows/finalize-tests.yml

‎.github/workflows/ci-arm.yml‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -127,6 +127,7 @@ jobs:
127127
run-coverage: ${{ steps.source-run-info.outputs.run-coverage }}
128128
run-go-sdk-tests: ${{ steps.selective-checks.outputs.run-go-sdk-tests }}
129129
run-java-sdk-tests: ${{ steps.selective-checks.outputs.run-java-sdk-tests }}
130+
run-ts-sdk-docs: ${{ steps.selective-checks.outputs.run-ts-sdk-docs }}
130131
run-helm-tests: ${{ steps.selective-checks.outputs.run-helm-tests }}
131132
run-kubernetes-tests: ${{ steps.selective-checks.outputs.run-kubernetes-tests }}
132133
run-mypy-providers: ${{ steps.selective-checks.outputs.run-mypy-providers }}
@@ -1071,6 +1072,38 @@ jobs:
10711072
if-no-files-found: error
10721073
overwrite: 'true'
10731074

1075+
build-ts-sdk-docs:
1076+
name: "TypeScript SDK docs"
1077+
needs: [build-info]
1078+
runs-on: ${{ fromJSON(needs.build-info.outputs.runner-type) }}
1079+
timeout-minutes: 30
1080+
permissions:
1081+
contents: read
1082+
packages: read
1083+
if: needs.build-info.outputs.run-ts-sdk-docs == 'true'
1084+
env:
1085+
GITHUB_REPOSITORY: ${{ github.repository }}
1086+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
1087+
GITHUB_USERNAME: ${{ github.actor }}
1088+
VERBOSE: "true"
1089+
steps:
1090+
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
1091+
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
1092+
with:
1093+
persist-credentials: false
1094+
- name: "Install Breeze"
1095+
uses: ./.github/actions/breeze
1096+
- name: "Build TypeScript SDK API reference"
1097+
run: breeze build-docs --sdk-docs-only --sdk=typescript
1098+
- name: "Upload TypeScript SDK docs artifact"
1099+
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
1100+
with:
1101+
name: ts-sdk-docs
1102+
path: generated/_build/docs/ts-sdk/
1103+
retention-days: 7
1104+
if-no-files-found: error
1105+
overwrite: 'true'
1106+
10741107
tests-airflow-ctl:
10751108
name: "Airflow CTL tests"
10761109
uses: ./.github/workflows/airflow-distributions-tests.yml
@@ -1126,6 +1159,7 @@ jobs:
11261159
- tests-go-sdk
11271160
- tests-java-sdk
11281161
- build-java-sdk-docs
1162+
- build-ts-sdk-docs
11291163
- tests-with-lowest-direct-resolution-core
11301164
- tests-with-lowest-direct-resolution-providers
11311165
uses: ./.github/workflows/finalize-tests.yml

‎.github/workflows/publish-docs-to-s3.yml‎

Lines changed: 79 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -356,18 +356,19 @@ jobs:
356356
INCLUDE_COMMITS: ${{ startsWith(inputs.ref, 'providers') && 'true' || 'false' }}
357357
FAIL_ON_INVENTORIES: ${{ inputs.ignore-missing-inventories != true && '--fail-on-missing-third-party-inventories' || '' }}
358358
run: |
359-
# The Java SDK docs are built with Dokka by the separate build-java-sdk-docs job;
360-
# java-sdk is not a Sphinx package, so it must not be passed to breeze build-docs.
359+
# The language SDK docs are built by their own jobs with their native toolchains
360+
# (Dokka for java-sdk, TypeDoc for ts-sdk). Neither is a Sphinx package, so they
361+
# must not be passed to breeze build-docs.
361362
SPHINX_INCLUDE_DOCS=""
362363
for package in ${INCLUDE_DOCS}; do
363-
if [[ "${package}" != "java-sdk" ]]; then
364+
if [[ "${package}" != "java-sdk" && "${package}" != "ts-sdk" ]]; then
364365
SPHINX_INCLUDE_DOCS="${SPHINX_INCLUDE_DOCS} ${package}"
365366
fi
366367
done
367368
if [[ -n "${INCLUDE_DOCS}" && -z "${SPHINX_INCLUDE_DOCS// /}" ]]; then
368-
echo "Only java-sdk docs were requested - skipping the Sphinx docs build."
369+
echo "Only language SDK docs were requested - skipping the Sphinx docs build."
369370
mkdir -p generated/_build/docs
370-
echo "The Sphinx docs build was skipped: only java-sdk docs were requested." \
371+
echo "The Sphinx docs build was skipped: only language SDK docs were requested." \
371372
> generated/_build/docs/sphinx-build-skipped.txt
372373
echo "sphinx-build-skipped=true" >> "${GITHUB_OUTPUT}"
373374
else
@@ -489,14 +490,80 @@ jobs:
489490
if-no-files-found: 'error'
490491
overwrite: 'true'
491492

493+
build-ts-sdk-docs:
494+
needs: [build-info]
495+
timeout-minutes: 30
496+
name: "Build TypeScript SDK API reference"
497+
runs-on: ubuntu-latest
498+
permissions:
499+
contents: read
500+
packages: read
501+
env:
502+
GITHUB_REPOSITORY: ${{ github.repository }}
503+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
504+
GITHUB_USERNAME: ${{ github.actor }}
505+
VERBOSE: "true"
506+
outputs:
507+
built: ${{ steps.sdk-present.outputs.exists }}
508+
steps:
509+
- name: "Checkout ${{ inputs.ref }}"
510+
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
511+
with:
512+
persist-credentials: false
513+
ref: ${{ inputs.ref }}
514+
- name: "Decide whether to build the TypeScript SDK API reference"
515+
id: sdk-present
516+
env:
517+
REF: ${{ inputs.ref }}
518+
INCLUDE_DOCS: ${{ inputs.include-docs }}
519+
run: |
520+
# Mirrors the java-sdk rule: only build for full doc builds (include-docs == 'all')
521+
# or when 'ts-sdk' is explicitly listed. A providers-only publish wave must not
522+
# refresh the ts-sdk docs as a side effect.
523+
if [[ "${INCLUDE_DOCS}" != "all" && " ${INCLUDE_DOCS} " != *" ts-sdk "* ]]; then
524+
echo "exists=false" >> "${GITHUB_OUTPUT}"
525+
echo "include-docs='${INCLUDE_DOCS}' is a specific package list without 'ts-sdk' — skipping TypeDoc build."
526+
elif [[ -f "ts-sdk/docs/typedoc.json" ]]; then
527+
echo "exists=true" >> "${GITHUB_OUTPUT}"
528+
else
529+
echo "exists=false" >> "${GITHUB_OUTPUT}"
530+
echo "ts-sdk/docs/ not found at ref ${REF} — skipping TypeDoc build."
531+
fi
532+
- name: "Install Breeze"
533+
if: steps.sdk-present.outputs.exists == 'true'
534+
uses: ./.github/actions/breeze
535+
# The TypeDoc toolchain is installed with `npm ci` inside the build container, which
536+
# reads its download cache from HOME. Without this the ~250 pinned packages are
537+
# fetched from the registry on every run.
538+
- name: "Cache TypeDoc toolchain downloads"
539+
if: steps.sdk-present.outputs.exists == 'true'
540+
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
541+
with:
542+
path: ts-sdk/docs/.npm-home/.npm
543+
key: ts-sdk-docs-npm-${{ hashFiles('ts-sdk/docs/package-lock.json') }}
544+
restore-keys: ts-sdk-docs-npm-
545+
- name: "Build TypeScript SDK API reference"
546+
if: steps.sdk-present.outputs.exists == 'true'
547+
run: breeze build-docs --sdk-docs-only --sdk=typescript
548+
- name: "Upload TypeScript SDK docs"
549+
if: steps.sdk-present.outputs.exists == 'true'
550+
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
551+
with:
552+
name: ts-sdk-docs
553+
path: generated/_build/docs/ts-sdk/
554+
retention-days: '7'
555+
if-no-files-found: 'error'
556+
overwrite: 'true'
557+
492558
publish-docs-to-s3:
493-
needs: [build-docs, build-java-sdk-docs, build-info, update-registry]
559+
needs: [build-docs, build-java-sdk-docs, build-ts-sdk-docs, build-info, update-registry]
494560
# `update-registry` is skipped for non-provider distributions, and a job whose dependency was
495561
# skipped is skipped in turn, so its result is checked rather than implied - which means the
496562
# two that are not conditional have to be asserted here as well.
497563
if: >
498564
!cancelled() && needs.build-docs.result == 'success' &&
499565
needs.build-java-sdk-docs.result == 'success' &&
566+
needs.build-ts-sdk-docs.result == 'success' &&
500567
needs.update-registry.result != 'failure'
501568
name: "Publish documentation to S3"
502569
permissions:
@@ -542,6 +609,12 @@ jobs:
542609
with:
543610
name: java-sdk-docs
544611
path: generated/_build/docs/java-sdk
612+
- name: "Download TypeScript SDK docs artifact"
613+
if: needs.build-ts-sdk-docs.outputs.built == 'true'
614+
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
615+
with:
616+
name: ts-sdk-docs
617+
path: generated/_build/docs/ts-sdk
545618
- name: "Make sure SBOM dir exists and has the right permissions"
546619
run: |
547620
sudo mkdir -vp ./files/sbom

‎airflow-core/docs/authoring-and-scheduling/language-sdks/typescript.rst‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,14 +26,19 @@ The TypeScript SDK lets you implement Airflow task logic in TypeScript (or plain
2626
Node.js. The Dag and its scheduling remain in Python; individual tasks delegate to a Node.js subprocess that
2727
is spawned by :class:`~airflow.sdk.coordinators.node.NodeCoordinator` for each task instance.
2828

29-
The SDK is the ``@apache-airflow/ts-sdk`` package (ESM-only). It is currently in **alpha** and its API may change.
29+
The SDK is an ESM-only package that ships from the ``ts-sdk/`` directory of the Airflow repository. It is currently in **alpha** and its API may change.
3030

3131
.. warning::
3232

3333
The SDK is not yet published to npm. To try it today, build it from source in the
3434
`ts-sdk/ <https://github.com/apache/airflow/tree/main/ts-sdk>`__ directory of the Airflow repository and
3535
depend on it locally (see ``ts-sdk/example/`` for a working setup).
3636

37+
.. seealso::
38+
39+
For the full TypeScript API reference (task handlers, ``TaskClient``, and the coordinator runtime),
40+
see the `TypeScript SDK API reference <https://airflow.apache.org/docs/ts-sdk/stable/>`__.
41+
3742
.. contents:: Contents
3843
:local:
3944
:depth: 2

‎dev/breeze/doc/ci/04_selective_checks.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -595,6 +595,7 @@ GitHub Actions to pass the list of parameters to a command to execute
595595
| run-mypy | Whether mypy check is supposed to run in this build | true | |
596596
| run-system-tests | Whether system tests should be run ("true"/"false") | true | |
597597
| run-task-sdk-tests | Whether Task SDK tests should be run ("true"/"false") | true | |
598+
| run-ts-sdk-docs | Whether the TypeScript SDK API reference should be built — on `ts-sdk/docs/` or `ts-sdk/src/` changes, including Markdown ("true"/"false") | true | |
598599
| run-ts-sdk-e2e-tests | Whether TypeScript SDK e2e tests should be run — on `ts-sdk/`, TS e2e test, or Node coordinator changes ("true"/"false") | true | |
599600
| run-ui-tests | Whether UI tests should be run ("true"/"false") | true | |
600601
| run-unit-tests | Whether unit tests should be run ("true"/"false") | true | |

0 commit comments

Comments
 (0)