Fix Grid view failing to load large Dags on MySQL - #71342
Conversation
|
This slightly changes tie-breaking logic; previously the ordering was by id, now it’s by timestamp. They should agree since the id is UUIDv7, so I’m just adding this as a hint for debugging, in case we hit an edge case in the future. |
|
Hi maintainer, this PR was merged without a milestone set.
|
Backport successfully created: v3-3-testNote: As of Merging PRs targeted for Airflow 3.X In matter of doubt please ask in #release-management Slack channel.
|
* Fix Grid view failing to load large Dags on MySQL * Remove unnecessary comment
Grid view showed no tasks for large Dags when the metadata backend is MySQL.
The API server logged:
Root cause
The grid endpoint used
_get_latest_serdagto load the latest serialized Dag.That query did
select(SerializedDagModel) ... ORDER BY ... LIMIT 1, which loads the whole row - including the bigdata/data_compressedblob columns.On MySQL,
ORDER BYuses filesort, and filesort copies each candidate row (the blob too) into the sort buffer. The sort buffer has a fixed size (sort_buffer_size, 256 KB by default). When a single serialized Dag is bigger than that, the sort fails with "Out of sort memory", so the whole request fails and the Grid view renders nothing.PostgreSQL is not affected because it does not copy large values into the sort buffer.
Fix
Use the existing
SerializedDagModel.latest_item_select_objecthelper.On MySQL it first finds the latest row's
id(ORDER BY onidonly, no blob), and then loads that single row by primary key. The blob is never put into the sort buffer, so the error is gone.Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 4.8) following the guidelines
{pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.