Add on_skipped_intervals_callback for catchup=False DAGs

[Tegh25]

---

• Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
• For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
• When adding dependency, check compliance with the ASF 3rd Party License Policy.
• For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

---

closes: #66791

When a Dag is configured with catchup=False and the scheduler advances past missed intervals (for example after a restart or re-enable), the skip happens silently: no DagRun rows are created and no hook fires. Operators who need observability for skipped partitions (SLA accounting, compliance audit trails, dashboard counters) had to compare last_automated_data_interval against wall clock externally.

This PR adds two symmetric extension points:

1. on_skipped_intervals_callback — a Dag-level callback on the SDK
   DAG constructor, using the same context-based signature as
   on_success_callback / on_failure_callback. The context includes
   dag, reason ("skipped_intervals"), and skipped_range (a
   DataInterval from the previous automated run's data_interval_end to
   the new run's data_interval_start).

2. on_intervals_skipped — an AIP-61 listener hookspec for plugins
   that need in-process observability without reloading the Dag file.
   The listener receives a SkippedIntervalsSummary with the same
   skipped_range.

How it works

Skipped intervals are detected at scheduled DagRun creation time (timetable-agnostic):

• When the scheduler creates a new SCHEDULED run for a catchup=False
  Dag, it compares the new run's data_interval_start against the
  data_interval_end of the most recent prior scheduled run.
• If there is a gap, it builds a SkippedIntervalsSummary with
  skipped_range spanning those two boundaries. The scheduler does not
  enumerate every skipped interval on the hot path.
• Detection is gated: this runs only when not dag.catchup AND
  (has_on_skipped_intervals_callback is set OR an on_intervals_skipped
  listener implementation is registered).

Listener fires synchronously in the scheduler (same pattern as on_dag_run_running for Dag start events).

Callback follows the existing Dag callback pipeline:
DagSkippedIntervalsCallbackRequest → DatabaseCallbackSink → Dag File Processor reloads the Dag file and invokes the callable. The skipped gap is serialized as a (start, end) datetime pair (skipped_range) in the request because no DagRun row exists for skipped intervals and the processor cannot reliably reconstruct the gap later.

To enumerate individual skipped intervals (for example to count them or log each one), call context["dag"].iter_dagrun_infos_between(skipped_range.start, skipped_range.end) inside the callback, or the equivalent on the Dag in a listener implementation.

Files changed

Area	Files
SDK Dag definition	task-sdk/src/airflow/sdk/definitions/dag.py, task-sdk/src/airflow/sdk/definitions/context.py
Timetable types	airflow-core/src/airflow/timetables/base.py
Serialization	serialization/definitions/dag.py, serialized_objects.py
Callback transport	callbacks/callback_requests.py, dag_processing/processor.py
Supervisor schema	task-sdk/src/airflow/sdk/execution_time/schema/versions/v2026_06_23.py, schema.json
Listener	listeners/spec/dag.py (new), listeners/listener.py
Scheduler	jobs/scheduler_job_runner.py

Usage (preview)

from airflow.sdk import DAG


def notify_skipped(context):
    skipped_range = context["skipped_range"]
    print(f"Gap: {skipped_range.start} -> {skipped_range.end}")
    for info in context["dag"].iter_dagrun_infos_between(
        skipped_range.start, skipped_range.end
    ):
        if info.data_interval is None:
            continue
        interval = info.data_interval
        print(f"Skipped: {interval.start} -> {interval.end}")


with DAG(
    "my_dag",
    schedule="@daily",
    catchup=False,
    on_skipped_intervals_callback=notify_skipped,
) as dag:
    ...

Follow-up (Will be added to this PR)

• Unit and integration tests (scheduler skip detection, callback execution, listener hook, serialization roundtrip)

• User documentation (on_skipped_intervals_callback in Dag reference, on_intervals_skipped in listeners docs)

• Newsfragment in airflow-core/newsfragments/ (user-facing feature)

Test plan

To be completed before marking Ready for review:

• Scheduler creates skip callback request when catchup=False Dag jumps past intervals and callback is configured

• Listener on_intervals_skipped fires with correct intervals

• Dag File Processor executes callback with skipped_intervals in context

• No overhead when callback and listener are both unset

• Serialization roundtrip preserves has_on_skipped_intervals_callback

• prek run --from-ref main --stage pre-commit passes

---

Was generative AI tooling used to co-author this PR?

• Yes - Cursor (Auto)
  Generated-by: Cursor (Auto) following the guidelines

[boring-cyborg]

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
Here are some useful points:

• Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
• In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example Dag that shows how users should use it.
• Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
• Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
• Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
• Be sure to read the Airflow Coding style.
• Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
  Apache Airflow is a community-driven project and together we are making it better 🚀.
  In case of doubts contact the developers at:
  Mailing List: dev@airflow.apache.org
  Slack: https://s.apache.org/airflow-slack

[Tegh25]

Hi @ferruzzi, please review when you get the chance. Thanks

[ferruzzi]

Sorry for the delay, I'll look tomorrow. The failing CI is going to have to be addressed. Run your static-checks locally for a start, then rebase main and do a force-push. We'll see what's left failing and can look at them together if you need a hand.

[ferruzzi]

Good start, left some comments and I can make another pass when you get those addresseed

[ferruzzi]

Left a question for discussion but otherwise LGTM.

[ferruzzi]

Please leave this comment open as a reminder to verify this file before we merge.

[uranusjr]

I feel we lack a discussion on this feature. Can someone raise a thread on the dev list so maintainers are aware of this?

[Tegh25]

Thanks for working on this @Tegh25, could you resolve conflicts and move forward with the PR?

Hi @1fanwang, I will try to get the merge conflicts resolved today.

[uranusjr]

We should have some test cases on skipped_range since it’s a half-open-ish envelope built from the previous run's data_interval_end and the new run's data_interval_start, both boundaries are off-by-one candidates. Otherwise I think this is generally fine.

[uranusjr]

Based on existing naming patterns between callbacks and listeners, this probably should be on_dag_skipped_intervals?

[Tegh25]

From what I've observed, listener names are expressed in terms of the event being emitted. Here the event is specifically “intervals skipped” when catchup=False, and the fact that it’s Dag-scoped is already implied by it living in listeners/spec/dag.py and by the scheduler code calling hook.on_intervals_skipped.

I'm happy to change the name if you feel strongly about it. However, given the maturity of this PR with the implementation, tests, docs, newsfragment, etc., it may be more effort than its worth to change the hook name for a small improvement. The original issue also mentioned adding a listener hook with the specific name of on_intervals_skipped.