Airflow 3: Scheduler liveness probe failure & performance drop with thousands of dynamic tasks (KubernetesExecutor)

[sdg9670f]

TL;DR:
Running Airflow 3 on K8s with KubernetesExecutor (2 scheduler replicas). Processing thousands of dynamic tasks causes _do_scheduling() execution time (~25s+) to exceed the heartbeat interval (5s). Because heartbeat updates are synchronous in the main loop, delayed heartbeats trigger false-positive K8s liveness probe failures and pod restarts. We want to solve this without capping dynamic task counts or throttling task concurrency. Looking for tuning strategies for maximum parallel throughput.

Hi everyone,

We are currently running Airflow 3 deployed via the official Helm chart in a Kubernetes environment using KubernetesExecutor with 2 scheduler replicas.

Recently, we encountered severe scheduler instability when running DAGs with large-scale dynamic task mapping (generating several thousand tasks per run). The main symptom is that during heavy task expansion, the scheduler pod's liveness probe repeatedly fails, causing Kubernetes to restart the scheduler pods while they are actively processing tasks.

---

⚙️ Environment & Configuration

• Deployment: Airflow 3 via official Helm Chart
• Executor: KubernetesExecutor (workers spawned as individual K8s pods)
• Scheduler Replicas: 2
• Resource Usage: CPU and Memory utilization on scheduler pods look healthy with plenty of headroom (no OOM or CPU starvation observed).
• Key Configuration Snippet:

scheduler:
  enable_health_check: 'True'
  max_tis_per_query: 128
  task_queued_timeout: 1200

---

🔍 Deep Dive: Problem Mechanics & Root Cause

After analyzing the Airflow execution loop (job.py & scheduler_job_runner.py), we identified the exact flow leading to these false-positive restarts:

Loop Duration (25s+) > heartbeat_sec (5s) x threshold (30s window)
  │
  ├─► _do_scheduling() takes 20-30s+ to expand/queue thousands of dynamic tasks
  ├─► Synchronous heartbeat skips sleeping (sleep_for = 0) & loops immediately
  ├─► Time since latest_heartbeat exceeds health check window
  │
  ├─► RESULT A: K8s Probe Check = False FAIL ──► Unnecessary Pod Restart
  └─► RESULT B: Actual issues (e.g., DB deadlocks) masked or missed

Because the main loop executes back-to-back without breathing room, there is almost no window for the health check to pass. The pod gets killed by Kubernetes even though it is performing heavy, valid scheduling work.

---

❓ Key Question: How to Scale Without Throttling?

Our primary goal is to maintain high parallelism and scale. We do NOT want to reduce the number of dynamic tasks or lower the task concurrency limits—we want Airflow to execute as many tasks simultaneously as possible.

With that in mind, how can we resolve this bottleneck and probe failure while maximizing throughput?

Any insights, recommended config settings, or architectural advice from high-volume production setups would be greatly appreciated!

Thanks in advance!

[amerob]

Your loop analysis is right, but the arithmetic on the probe is off in a way that changes the fix.

is_alive() for a SchedulerJob doesn't use the heartrate at all. From airflow/jobs/job.py:

def health_check_threshold(job_type: str, heartrate: int) -> int | float:
    grace_multiplier = 2.1
    if job_type == "SchedulerJob":
        health_check_threshold_value = conf.getint("scheduler", "scheduler_health_check_threshold")
    ...
    else:
        health_check_threshold_value = heartrate * grace_multiplier

The 2.1x grace multiplier other job types get is skipped for the scheduler. It compares latest_heartbeat against scheduler_health_check_threshold flat, default 30s. So scheduler_heartbeat_sec (5s) never enters the liveness calculation — it only controls how often the loop is willing to write a heartbeat (perform_heartbeat(..., only_if_necessary=True)). Your 25s+ _do_scheduling() is running straight into that 30s wall, and that's the only number that matters here.

Second: a single slow loop can't restart your pod. The chart's default is

livenessProbe:
  initialDelaySeconds: 10
  timeoutSeconds: 20
  failureThreshold: 5
  periodSeconds: 60

That's 5 consecutive failures 60s apart — five straight minutes in which every probe sees a >30s-stale heartbeat. That isn't one big expansion, it's sustained back-to-back long loops. Worth confirming before you tune anything: if pods are dying more often than once per ~5 min, you're chasing the wrong thing.

One more that bites people — enable_health_check: 'True' is not what your probe is using. That flag starts a small HTTP server on port 8974 serving /health. The chart's scheduler liveness probe is an exec:

airflow jobs check --job-type SchedulerJob --local

Unless you've set scheduler.livenessProbe.command yourself, toggling enable_health_check changes nothing about the restarts. Both paths land on the same 30s threshold so it isn't the cause, but it's a dead end to tune.

For the fix, you don't have to throttle anything.

Raise the threshold so it reflects your worst-case loop rather than your average:

config:
  scheduler:
    scheduler_health_check_threshold: '300'

That's the legitimate fix for a false positive — you're telling the probe what "hung" actually means for this workload. Keep it under the 5x60s window or you've just moved the goalpost.

Then look hard at max_tis_per_query: 128. Default is 16, and this is what's making your loops long. It bounds how many TIs are evaluated inside the critical section — the part holding row locks. Lowering it does not lower throughput. You schedule the same tasks in more, shorter passes; the loop only sleeps when idle:

idle_in_this_run = not num_queued_tis and not num_finished_events
if not is_unit_test and idle_in_this_run:
    time.sleep(min(self._scheduler_idle_sleep_time, next_event or 0))

With work pending it comes straight back around, no sleep. What you gain is a heartbeat write between chunks instead of only after all 128.

With 2 replicas that lever does double duty. A 128-TI critical section holds locks the other scheduler is blocking on, so your two schedulers are partly serializing against each other — likely the same DB contention you noted getting masked. Try 32 and watch the scheduler.scheduler_loop_duration timer. If p99 drops under 30s and your queued-tasks/sec holds, you've solved it without capping task counts.

Last one, if your mapped DAGs are large: check max_dagruns_per_loop_to_schedule (default 20). The _do_scheduling docstring is explicit that raising it helps small DAGs and hurts throughput on >500-task DAGs. If you've bumped it, put it back.