Reduce scheduler memory for asset-triggered Dag runs - #69849
Conversation
boring-cyborg
Bot
added
the
area:Scheduler
|
cc @Lee-W, @jason810496. ✨🙌 |
Lee-W
left a comment
Lee-W
left a comment
There was a problem hiding this comment.
Thanks! I remember we did something similar elsewhere. the overall idea looks good to me
c250d56 to
bb68de6
Compare
|
cc @Lee-W. 🫡 |
eladkal
added
type:bug-fix
Large asset event windows can exhaust scheduler memory because every event, including JSON metadata, is materialized as an ORM object before its Dag run association is persisted. Concurrent arrivals must also remain isolated from the event window used to derive each run's timestamp. Signed-off-by: viiccwen <vicwen@apache.org>
bb68de6 to
e042524
Compare
|
Hello @Lee-W, since upstream had been heavily modified, need review again. 🙏 |
| selected_asset_events.c.timestamp <= triggered_date, | ||
| selected_asset_events.c.event_id <= max_selected_event_id, |
There was a problem hiding this comment.
This looks like dead weight unless the reader understands the CTE is re-evaluated fresh in this second, independent query and could pick up events inserted after the aggregate read. Add a comment explaining this is a snapshot bound against a race, not a no-op.
| consumed_asset_event_count = ( | ||
| session.scalar( | ||
| select(func.count()) | ||
| .select_from(association_table) | ||
| .where(association_table.c.dag_run_id == dag_run.id) | ||
| ) | ||
| or 0 | ||
| ) |
There was a problem hiding this comment.
We can probably use rowcount instead? If we must use a separate query, it’d be a good idea to at least add a guard on log level (with isEnabledFor)
| select(literal(dag_run.id), selected_event_ids.c.event_id), | ||
| ) | ||
| ) | ||
| session.expire(dag_run, ["consumed_asset_events"]) |
There was a problem hiding this comment.
Worth a comment explaining why manual expiration is needed.
|
I think this is close to mergeable with some nits. |
5 participants
Description
closes #69848
Asset-triggered Dag run creation currently materializes every matching
AssetEventORM object, including its JSON metadata, before writing the Dag run association rows. Large event windows can therefore cause scheduler memory spikes far beyond the raw payload size.This change writes the association rows with a server-side
INSERT ... SELECTwhile preserving the existing event predicates, transaction boundary, exact event membership, and relationship behavior.Scheduler-side
AssetEventORM and JSON payload materialization drops from O(events) to approximately O(1). The database still performs O(events) work to select event IDs and persist one association row per event; this change moves that set operation out of Python rather than eliminating it.The partitioned-asset path uses the same set-based association helper. No schema or configuration change is needed.
Benchmark
The benchmark invokes the real scheduler path on SQLite, PostgreSQL, and MySQL with a 512-byte JSON
payload per event. Setup is outside the measured interval.
The 1,000-event result includes first-call and tracing warm-up overhead.
Notes
consumed_asset_event_countis designed for logging, can discuss leave it or not.Was generative AI tooling used to co-author this PR?
{pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.