Skip to content

Define when the serialized Dag SERIALIZER_VERSION must be bumped #71364

Description

@jason810496

Context

That PR adds an optional per-task _arg_bindings property to the serialized Dag. Review split on whether an additive field warrants a SERIALIZER_VERSION bump, which surfaced the real gap: there is no written rule for when the version changes, so every PR touching serialization re-litigates it.

Problem

SerializedDAG.SERIALIZER_VERSION is 3 today, and from_dict chains conversion_v1_to_v2 / conversion_v2_to_v3 for older blobs. But:

  • No documented rule says when a bump is required, so "is this additive enough to skip it?" is argued case by case.
  • conversion_v2_to_v3 is a comment plus ser_obj["__version"] = 3 — it carries no conversion, so the number does not yet signal anything a reader can act on.
  • In Airflow 2 the version stayed at 1 for the whole major, so it never carried information. Repeating that in Airflow 3 wastes the mechanism.
  • schema.json is also weaker than it looks on the write path: definitions.tasks is {"type": "array", "additionalProperties": {...}}, and additionalProperties is inert for arrays, so validate_schema never checks task objects against definitions.operator. Any policy that leans on "the schema catches it" needs this fixed first.

Scope

Decide and document:

  1. What a bump means — a format-compat marker, or a promise that a conversion_vN_to_vN+1 exists and does real work.
  2. Whether additive optional fields bump, or only removals / renames / semantic changes.
  3. What a bump obligates: conversion function, from_dict accept-list, schema.json, tests.
  4. How far back from_dict supports old versions, and when old conversions can be dropped.
  5. Whether to repair the inert definitions.tasks validation so schema.json is genuinely enforced.
  6. Release policy.
  7. Date based versioning? Semantic versioning?

Record the outcome in-repo (contributing docs or a serialized_objects.py module docstring) so it stops being re-decided per PR.

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions