Skip to content

Generate a JSON Schema for datasetConfig from DatasetConfig #890

Description

@vinishjail97

Feature Request / Improvement

Publish a JSON Schema for datasetConfig.yaml, generated from the DatasetConfig class rather than written by hand.

Why

Writing a valid datasetConfig is the first thing a new user does and the first place they get stuck: sourceFormat, targetFormats, tableBasePath, tableName, namespace, partitionSpec, and the right conversionSourceProviderClass. Today the only description of that shape is prose in the docs, so it drifts from the code silently.

A generated schema gives three things at once:

  • Editors and IDEs validate and complete the file.
  • Any agent or script can check a config before running a sync, instead of discovering problems from a stack trace.
  • The documentation stops being a second source of truth that can go stale.

PR #829 includes a hand-written config-schema.md for exactly this purpose. That is the thing to replace with something generated.

Scope

  • Generate a JSON Schema from DatasetConfig and its nested types at build time.
  • Publish it as a release artifact and at a stable URL, so $schema references keep working.
  • Fail the build when the schema is out of date with the model, so it cannot silently drift.
  • Reference it from the docs in place of any hand-maintained field list.

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions