Skip to content

Fix Grid view failing to load large Dags on MySQL - #71342

Merged
henry3260 merged 3 commits into
apache:mainfrom
WonYong-Jang:fix-grid-out-of-sort-memory-mysql
Aug 10, 2026
Merged

Fix Grid view failing to load large Dags on MySQL#71342
henry3260 merged 3 commits into
apache:mainfrom
WonYong-Jang:fix-grid-out-of-sort-memory-mysql

Conversation

@WonYong-Jang

@WonYong-Jang WonYong-Jang commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

Grid view showed no tasks for large Dags when the metadata backend is MySQL.
The API server logged:

ERROR 1038: Out of sort memory, consider increasing server sort buffer size

Root cause

The grid endpoint used _get_latest_serdag to load the latest serialized Dag.
That query did select(SerializedDagModel) ... ORDER BY ... LIMIT 1, which loads the whole row - including the big data / data_compressed blob columns.

On MySQL, ORDER BY uses filesort, and filesort copies each candidate row (the blob too) into the sort buffer. The sort buffer has a fixed size (sort_buffer_size, 256 KB by default). When a single serialized Dag is bigger than that, the sort fails with "Out of sort memory", so the whole request fails and the Grid view renders nothing.

PostgreSQL is not affected because it does not copy large values into the sort buffer.

Fix

Use the existing SerializedDagModel.latest_item_select_object helper.
On MySQL it first finds the latest row's id (ORDER BY on id only, no blob), and then loads that single row by primary key. The blob is never put into the sort buffer, so the error is gone.


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: Claude Code (Opus 4.8) following the guidelines


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@boring-cyborg boring-cyborg Bot added the area:API Airflow's REST/HTTP API label Aug 9, 2026
@WonYong-Jang
WonYong-Jang marked this pull request as ready for review August 9, 2026 07:14
Comment thread airflow-core/src/airflow/api_fastapi/core_api/routes/ui/grid.py Outdated
@uranusjr

Copy link
Copy Markdown
Member

This slightly changes tie-breaking logic; previously the ordering was by id, now it’s by timestamp. They should agree since the id is UUIDv7, so I’m just adding this as a hint for debugging, in case we hit an edge case in the future.

@henry3260 henry3260 added the backport-to-v3-3-test Backport to v3-3-test label Aug 10, 2026
@henry3260
henry3260 merged commit fe5107f into apache:main Aug 10, 2026
78 checks passed
@github-actions github-actions Bot added this to the Airflow 3.3.2 milestone Aug 10, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Hi maintainer, this PR was merged without a milestone set.
We've automatically set the milestone to Airflow 3.3.2 based on: backport label targeting v3-3-test
If this milestone is not correct, please update it to the appropriate milestone.

This comment was generated by Milestone Tag Assistant.

@github-actions

Copy link
Copy Markdown
Contributor

Backport successfully created: v3-3-test

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status Branch Result
v3-3-test PR Link

pierrejeambrun pushed a commit that referenced this pull request Aug 10, 2026
#71370)

* Fix Grid view failing to load large Dags on MySQL

* Remove unnecessary comment
(cherry picked from commit fe5107f)

Co-authored-by: WonYong Jang <zcx5674@gmail.com>
Co-authored-by: Rahul Vats <43964496+vatsrahul1001@users.noreply.github.com>
vatsrahul1001 added a commit that referenced this pull request Aug 13, 2026
#71370)

* Fix Grid view failing to load large Dags on MySQL

* Remove unnecessary comment
(cherry picked from commit fe5107f)

Co-authored-by: WonYong Jang <zcx5674@gmail.com>
Co-authored-by: Rahul Vats <43964496+vatsrahul1001@users.noreply.github.com>
dabla pushed a commit to dabla/airflow that referenced this pull request Aug 14, 2026
* Fix Grid view failing to load large Dags on MySQL

* Remove unnecessary comment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:API Airflow's REST/HTTP API backport-to-v3-3-test Backport to v3-3-test

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants