Add TypeScript SDK API reference docs with TypeDoc by jason810496 · Pull Request #70812 · apache/airflow

jason810496 · 2026-07-31T07:16:48Z

Why

The @apache-airflow/ts-sdk package has a public API annotated with JSDoc/TSDoc but no rendered API reference. This adds one, built with TypeDoc and the @clean-jsdoc-theme/typedoc theme, and wires it into the existing build-docs and publish-docs pipelines.

The reference is built with TypeScript's own documentation tool rather than through Sphinx. That follows the precedent already set by the Java SDK.

Note for reviewers: earlier revisions of this PR built the reference through sphinx-js. That approach is no longer present — see Why not sphinx-js below for what it cost and why it was dropped.

How

Renders src/index.ts to HTML under _build/html; the toolchain is pinned in package.json + package-lock.json (TypeDoc 0.28, @clean-jsdoc-theme/typedoc 5.1). It is kept as its own npm package so docs-only dependencies stay out of the SDK's runtime and build toolchain.
breeze build-docs --sdk-docs-only --sdk=typescript runs the toolchain in a node:22 container (no local Node required, mirroring how the Java SDK's Dokka build runs in eclipse-temurin), stages the HTML to generated/_build/docs/ts-sdk/stable/.
build-ts-sdk-docs job in publish-docs-to-s3.yml mirrors build-java-sdk-docs
The landing page is ts-sdk/docs/index.md. The theme renders page bodies through MDX, where <!-- is a syntax error, so scripts/prepare-readme.mjs strips the ASF header into a gitignored scratch copy before TypeDoc reads it. The committed file keeps the standard header the insert-license hook enforces on every Markdown file.
Publishing is wired up too, not just building: ts-sdk is added to NON_SHORT_NAME_PACKAGES in publish_docs_to_s3.py (without it the S3 publisher would expand ts-sdk as a provider short name into a non-existent apache-airflow-providers-ts-sdk instead of publishing it as a top-level package), a build-ts-sdk-docs job now runs in ci-amd.yml / ci-arm.yml too (mirroring java-sdk's), gated on a new run-ts-sdk-docs selective-check output, and the release procedure is documented in ts-sdk/README.md.

Why not sphinx-js

Recorded because the PR carried this approach for several revisions:

sphinx-js 5.0.3 requires TypeDoc 0.25 (its plugin targets internals that changed in 0.28), and TypeDoc 0.25 caps TypeScript at 5.4 — several majors behind the TypeScript 6 the SDK is built with. That forced a standalone tsconfig.json downgrading the SDK's es2023 target to es2022 for the parse pass.
sphinx-js's comment renderer only understands text and code parts and raised Not implemented on TSDoc {@link} tags, so a bundled TypeDoc plugin (sphinxJsConfig.mjs) had to flatten every inline tag to literal text.
The reference itself was a hand-maintained api.rst listing each export with js:autoclass / js:autofunction, which had to be edited whenever the public surface changed.
The Python dependency dragged the workspace backwards: sphinx-js requires cattrs < 25, pinning uv.lock from cattrs 26.1.0 down to 24.1.3.

Was generative AI tooling used to co-author this PR?

Yes, with help of Claude Code Opus 4.8 following the guidelines

github-actions · 2026-08-04T02:42:58Z

uv.lock on main just moved via #71039 ("Record the relaxed types-paramiko bound in uv.lock"), commit 381339d and this PR currently conflicts.

Quickest fix:

git fetch upstream main && git rebase upstream/main
rm uv.lock && uv lock
git add uv.lock && git rebase --continue
git push --force-with-lease

Automated nudge — ignore if you're not ready to rebase. This comment is updated in place on future uv.lock bumps.

uranusjr · 2026-08-07T01:24:27Z

Does TypeScript/JavaScript/Node not have its native tooling for this? I’m not thrilled with using sphinx-js (particularly with it pinning cattrs, but also that there’s one more moving block). Java SDK does not tie its API docs into Sphinx.

If we’re committed to using sphinx-js, I think this still misses some mechanisms to correctly hook into CI build and publish jobs.

jason810496 · 2026-08-07T06:23:59Z

Does TypeScript/JavaScript/Node not have its native tooling for this? I’m not thrilled with using sphinx-js (particularly with it pinning cattrs, but also that there’s one more moving block). Java SDK does not tie its API docs into Sphinx.

If we’re committed to using sphinx-js, I think this still misses some mechanisms to correctly hook into CI build and publish jobs.

I use sphinx-js in the first place as I prefer to able to navigate back to the airflow core site from the user point of view. However, if we considering whether it's able to bump the native TypeDoc framwork itself, then I think we should use just TypeDoc itself. (It shouldn't be difficult to add a tab on the navbar to navigate back to the core docs site).

jason810496 · 2026-08-10T12:06:26Z

Let’s wire this up for PRs. Otherwise I think this is mostly good.

Sure! I just wired them up, the fail k8s test is not related.

guan404ming

Overall looks great. Just found some small issue:

TS_SDK_DOCS_FILES misses ts-sdk/tsconfig.json and ts-sdk/package.json (both feed the TypeDoc build), so changes to them skip the docs build until release publish.
Non-Markdown ts-sdk/docs/ files still match TS_SDK_FILES, so docs-toolchain-only changes trigger the full e2e suite.

Suggest adding those two files to TS_SDK_DOCS_FILES and excluding ^ts-sdk/docs/ from TS_SDK_FILES.

Wrapping the referenced identifier in backticks makes it render as an inline literal and keeps sphinxcontrib-spelling from treating the identifier as prose, so the build no longer fails on names like getVariableOrThrow and the spelling wordlist stays clean.

Packages whose inventory cannot be fetched (typically brand-new docs packages not yet published) are built first precisely so that their objects.inv exists for every other package's intersphinx mapping. The Sphinx spelling builder never writes objects.inv, so a spellcheck-only pre-build left the inventory missing, every other package failed inventory resolution, and the retry logic rebuilt the whole docs set in up to three additional full passes - the --spellcheck-only CI job ran 2h27m while the equivalent --docs-only job finished in 32m.

"abortable" is not in the spellcheck dictionary and docs prose should not need a wordlist entry for a phrasing that can simply be avoided.

publish-docs resolves every package's version up front via get_package_version_possibly_from_stable_txt, which knew nothing about ts-sdk, so the "Publish documentation and validate versions" CI job failed with "Unsupported package: ts-sdk". Fall back to the version in ts-sdk/package.json the same way task-sdk falls back to its source version, matching what DocsPublisher already does.

sphinx-js held the docs toolchain back to TypeDoc 0.25 and TypeScript 5.4 — several majors behind the TypeScript 6 the SDK itself is built with — and its comment renderer could not represent TSDoc inline tags, so a bundled TypeDoc plugin had to flatten every `{@link}` into literal text just to keep the build from crashing. Both the version cap and the workaround existed only to bridge TypeScript into Sphinx. Rendering with TypeDoc directly removes that bridge. The toolchain tracks the same TypeScript the SDK compiles with, the reference is no longer squeezed through a hand-maintained RST mirror of the public API that had to be updated whenever an export changed, and cross-references resolve natively. This follows the precedent already set by the Java SDK, whose Javadoc is built with Dokka rather than Sphinx and staged into the same publish pipeline. The package root is the only entry point given to TypeDoc. It already re-exports everything `./coordinator` exposes, so listing both produced two navigation sections that documented one API between them, with no symbol unique to either. Dropping the Python dependency also releases the constraint it dragged in: sphinx-js required cattrs < 25, which had pinned the whole workspace back from cattrs 26.1.0 to 24.1.3. The lock file returns to what main resolves.

The reference could be built but never published on its own. Three pieces were missing relative to the Java SDK, whose Javadoc already runs this path end to end. `ts-sdk` was absent from NON_SHORT_NAME_PACKAGES, so the S3 publisher treated it as a provider short name and expanded it to a provider distribution that does not exist, rather than publishing it as a top-level package. Nothing built the reference on a pull request. A break in the docs toolchain or the documented sources would therefore only surface during a release, which is the worst moment to discover it. The new job gates on its own selective-check output rather than reusing the ts-sdk test trigger: that group deliberately ignores Markdown, because a README edit should not run the SDK test suite, but the landing page is Markdown and the reference is generated from the sources, so the docs build has to react to both. Finally, the release procedure was undocumented, leaving no way for a release manager to know the docs need a separate, deliberate publish — providers-only publish waves intentionally skip the SDKs.

The docs tsconfig included every file under ts-sdk/src/ as a program root, which pulled in cli/pack.ts — a CLI-only module unreachable from the public API that needs the optional esbuild peer dependency the isolated docs toolchain never installs. Rooting the program at the entry point instead lets TypeScript's own import graph decide what belongs in the type-checked program, so unreachable internal modules are no longer swept in. @msgpack/msgpack is different: coordinator/frames.ts is a real transitive dependency of the public startCoordinator export (via comm-channel.ts), so type-checking it needs the package installed, not excluded.

The previous fix added @msgpack/msgpack to ts-sdk/docs/package.json, but that alone can't work: Node's module resolution walks node_modules ancestors of the *importing file* (ts-sdk/src/coordinator/frames.ts), and ts-sdk/docs/ is a sibling of ts-sdk/src/, never an ancestor. The fix only appeared to succeed locally because ts-sdk/node_modules already existed from an unrelated pnpm install and was picked up by that ancestor walk — a genuinely fresh checkout (matching CI, and reported here) has no such directory. Redirect the bare specifier with a `paths` entry instead, pointing at where npm actually installs the package for this toolchain. A path substitution to the package's directory doesn't trigger its package.json main/types resolution under node16/nodenext module resolution (verified with --traceResolution), so the path names the concrete declaration file the package's own `types` field already points to.

uranusjr · 2026-08-12T00:01:05Z

Some general comments

We should cache npm in CI.
_stage_sdk_docs changes java-sdk staging from a merge to a destructive rmtree + copytree. Probably fine? Just confirming it’s intended.
breeze build-docs --sdk-docs-only with no --sdk exits 0 having built nothing. Since the help text says “Requires at least one”, maybe we can raise an error when none is provided?

uranusjr · 2026-08-12T00:03:37Z

Do we need a pre-commit hook to keep versions here and ts-sdk/package.json in sync?

uranusjr

This is generally fine. All comments made above can be follow-up PRs.

Keeps selective-checks accurate for the new ts-sdk docs build (tsconfig.json and package.json changes now trigger it, and docs-only package.json/lock changes no longer trigger the unrelated supervisor-schema check), fixes the alphabetical ordering of the new run-ts-sdk-docs row, simplifies the @msgpack/msgpack path mapping to resolve through its package.json instead of a hardcoded internal path, wraps the long tsconfig.json comment, and aligns the README's cancellation section wording with task.ts's TSDoc.

guan404ming

Looks great! Thanks for the update.

* Add TypeScript SDK API reference docs via sphinx-js * Render flattened TSDoc link tags as inline code Wrapping the referenced identifier in backticks makes it render as an inline literal and keeps sphinxcontrib-spelling from treating the identifier as prose, so the build no longer fails on names like getVariableOrThrow and the spelling wordlist stays clean. * Build priority docs packages fully even in spellcheck-only mode Packages whose inventory cannot be fetched (typically brand-new docs packages not yet published) are built first precisely so that their objects.inv exists for every other package's intersphinx mapping. The Sphinx spelling builder never writes objects.inv, so a spellcheck-only pre-build left the inventory missing, every other package failed inventory resolution, and the retry logic rebuilt the whole docs set in up to three additional full passes - the --spellcheck-only CI job ran 2h27m while the equivalent --docs-only job finished in 32m. * Reword TaskContext.signal TSDoc to avoid non-dictionary word "abortable" is not in the spellcheck dictionary and docs prose should not need a wordlist entry for a phrasing that can simply be avoided. * Resolve the ts-sdk version in publish-docs when no stable.txt is staged publish-docs resolves every package's version up front via get_package_version_possibly_from_stable_txt, which knew nothing about ts-sdk, so the "Publish documentation and validate versions" CI job failed with "Unsupported package: ts-sdk". Fall back to the version in ts-sdk/package.json the same way task-sdk falls back to its source version, matching what DocsPublisher already does. * Build the TypeScript SDK API reference with TypeDoc instead of sphinx-js sphinx-js held the docs toolchain back to TypeDoc 0.25 and TypeScript 5.4 — several majors behind the TypeScript 6 the SDK itself is built with — and its comment renderer could not represent TSDoc inline tags, so a bundled TypeDoc plugin had to flatten every `{@link}` into literal text just to keep the build from crashing. Both the version cap and the workaround existed only to bridge TypeScript into Sphinx. Rendering with TypeDoc directly removes that bridge. The toolchain tracks the same TypeScript the SDK compiles with, the reference is no longer squeezed through a hand-maintained RST mirror of the public API that had to be updated whenever an export changed, and cross-references resolve natively. This follows the precedent already set by the Java SDK, whose Javadoc is built with Dokka rather than Sphinx and staged into the same publish pipeline. The package root is the only entry point given to TypeDoc. It already re-exports everything `./coordinator` exposes, so listing both produced two navigation sections that documented one API between them, with no symbol unique to either. Dropping the Python dependency also releases the constraint it dragged in: sphinx-js required cattrs < 25, which had pinned the whole workspace back from cattrs 26.1.0 to 24.1.3. The lock file returns to what main resolves. * Publish the TypeScript SDK API reference automatically The reference could be built but never published on its own. Three pieces were missing relative to the Java SDK, whose Javadoc already runs this path end to end. `ts-sdk` was absent from NON_SHORT_NAME_PACKAGES, so the S3 publisher treated it as a provider short name and expanded it to a provider distribution that does not exist, rather than publishing it as a top-level package. Nothing built the reference on a pull request. A break in the docs toolchain or the documented sources would therefore only surface during a release, which is the worst moment to discover it. The new job gates on its own selective-check output rather than reusing the ts-sdk test trigger: that group deliberately ignores Markdown, because a README edit should not run the SDK test suite, but the landing page is Markdown and the reference is generated from the sources, so the docs build has to react to both. Finally, the release procedure was undocumented, leaving no way for a release manager to know the docs need a separate, deliberate publish — providers-only publish waves intentionally skip the SDKs. * Fix TypeDoc build failing to resolve esbuild and @msgpack/msgpack The docs tsconfig included every file under ts-sdk/src/ as a program root, which pulled in cli/pack.ts — a CLI-only module unreachable from the public API that needs the optional esbuild peer dependency the isolated docs toolchain never installs. Rooting the program at the entry point instead lets TypeScript's own import graph decide what belongs in the type-checked program, so unreachable internal modules are no longer swept in. @msgpack/msgpack is different: coordinator/frames.ts is a real transitive dependency of the public startCoordinator export (via comm-channel.ts), so type-checking it needs the package installed, not excluded. * Fix @msgpack/msgpack still unresolved in a genuinely clean docs install The previous fix added @msgpack/msgpack to ts-sdk/docs/package.json, but that alone can't work: Node's module resolution walks node_modules ancestors of the *importing file* (ts-sdk/src/coordinator/frames.ts), and ts-sdk/docs/ is a sibling of ts-sdk/src/, never an ancestor. The fix only appeared to succeed locally because ts-sdk/node_modules already existed from an unrelated pnpm install and was picked up by that ancestor walk — a genuinely fresh checkout (matching CI, and reported here) has no such directory. Redirect the bare specifier with a `paths` entry instead, pointing at where npm actually installs the package for this toolchain. A path substitution to the package's directory doesn't trigger its package.json main/types resolution under node16/nodenext module resolution (verified with --traceResolution), so the path names the concrete declaration file the package's own `types` field already points to. * Restore import statements in TypeScript SDK doc examples * Address TypeScript SDK docs review feedback Keeps selective-checks accurate for the new ts-sdk docs build (tsconfig.json and package.json changes now trigger it, and docs-only package.json/lock changes no longer trigger the unrelated supervisor-schema check), fixes the alphabetical ordering of the new run-ts-sdk-docs row, simplifies the @msgpack/msgpack path mapping to resolve through its package.json instead of a hardcoded internal path, wraps the long tsconfig.json comment, and aligns the README's cancellation section wording with task.ts's TSDoc.

boring-cyborg Bot added area:dev-tools area:ts-sdk backport-to-v3-3-test Backport to v3-3-test kind:documentation labels Jul 31, 2026

jason810496 self-assigned this Jul 31, 2026

phanikumv reviewed Jul 31, 2026

View reviewed changes

Comment thread ts-sdk/docs/sphinxJsConfig.mjs Outdated

jason810496 removed the backport-to-v3-3-test Backport to v3-3-test label Jul 31, 2026

jason810496 force-pushed the feature/ts-sdk/typescript-api-docs branch from 29a9a15 to bc59b65 Compare August 3, 2026 03:48

jason810496 marked this pull request as ready for review August 3, 2026 08:29

jason810496 requested review from amoghrajesh, ashb, bugraoz93, choo121600, ephraimbuddy, gopidesupavan, jedcunningham, jscheffl, potiuk and vatsrahul1001 as code owners August 3, 2026 08:29

jason810496 requested a review from uranusjr August 3, 2026 08:29

jason810496 marked this pull request as draft August 4, 2026 10:14

jason810496 force-pushed the feature/ts-sdk/typescript-api-docs branch from 0c39519 to 8c185dc Compare August 4, 2026 12:42

jason810496 marked this pull request as ready for review August 5, 2026 02:05

jason810496 force-pushed the feature/ts-sdk/typescript-api-docs branch 2 times, most recently from de67e1a to bfebf21 Compare August 5, 2026 06:40

jason810496 requested a review from guan404ming August 6, 2026 03:25

jason810496 force-pushed the feature/ts-sdk/typescript-api-docs branch from bfebf21 to c42a3ef Compare August 6, 2026 03:26

jason810496 marked this pull request as ready for review August 10, 2026 10:00

guan404ming reviewed Aug 11, 2026

View reviewed changes

Comment thread airflow-core/docs/authoring-and-scheduling/language-sdks/typescript.rst Outdated

jason810496 and others added 10 commits August 11, 2026 15:00

Add TypeScript SDK API reference docs via sphinx-js

d060b86

Reword TaskContext.signal TSDoc to avoid non-dictionary word

5d0131b

"abortable" is not in the spellcheck dictionary and docs prose should not need a wordlist entry for a phrasing that can simply be avoided.

Restore import statements in TypeScript SDK doc examples

d189da7

jason810496 force-pushed the feature/ts-sdk/typescript-api-docs branch from 8578ec0 to d189da7 Compare August 11, 2026 15:01

uranusjr reviewed Aug 11, 2026

View reviewed changes

Comment thread dev/breeze/src/airflow_breeze/utils/selective_checks.py

uranusjr reviewed Aug 11, 2026

View reviewed changes

Comment thread dev/breeze/src/airflow_breeze/utils/selective_checks.py

uranusjr reviewed Aug 11, 2026

View reviewed changes

Comment thread ts-sdk/docs/tsconfig.json

uranusjr reviewed Aug 11, 2026

View reviewed changes

Comment thread ts-sdk/README.md Outdated

uranusjr reviewed Aug 12, 2026

View reviewed changes

Comment thread dev/breeze/doc/ci/04_selective_checks.md

uranusjr reviewed Aug 12, 2026

View reviewed changes

Comment thread ts-sdk/docs/tsconfig.json Outdated

uranusjr reviewed Aug 12, 2026

View reviewed changes

uranusjr approved these changes Aug 12, 2026

View reviewed changes

jason810496 force-pushed the feature/ts-sdk/typescript-api-docs branch from 9176f4e to 078a53e Compare August 12, 2026 03:03

guan404ming approved these changes Aug 12, 2026

View reviewed changes

jason810496 merged commit 1b902cc into apache:main Aug 12, 2026
157 of 158 checks passed

jason810496 deleted the feature/ts-sdk/typescript-api-docs branch August 12, 2026 04:47

jason810496 mentioned this pull request Aug 12, 2026

Add a prek hook to keep ts-sdk/docs/package.json versions in sync with ts-sdk/package.json #71479

Merged

1 task

Uh oh!

Conversation

jason810496 commented Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

How

Why not sphinx-js

Was generative AI tooling used to co-author this PR?

Uh oh!

Uh oh!

github-actions Bot commented Aug 4, 2026

Uh oh!

uranusjr commented Aug 7, 2026

Uh oh!

jason810496 commented Aug 7, 2026

Uh oh!

jason810496 commented Aug 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

guan404ming left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

uranusjr commented Aug 12, 2026

Uh oh!

Uh oh!

Uh oh!

uranusjr Aug 12, 2026

Choose a reason for hiding this comment

Uh oh!

uranusjr left a comment

Choose a reason for hiding this comment

Uh oh!

guan404ming left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jason810496 commented Jul 31, 2026 •

edited

Loading

jason810496 commented Aug 10, 2026 •

edited

Loading

guan404ming left a comment •

edited

Loading