Skip to content

Make HttpSensor defer when response_check is provided - #71922

Open
yuseok89 wants to merge 1 commit into
apache:mainfrom
yuseok89:fix-deferrable-httpsensor-response-check
Open

Make HttpSensor defer when response_check is provided#71922
yuseok89 wants to merge 1 commit into
apache:mainfrom
yuseok89:fix-deferrable-httpsensor-response-check

Conversation

@yuseok89

@yuseok89 yuseok89 commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

closes: #40209

HttpSensor(deferrable=True) with a response_check silently fell back to the synchronous poke loop, occupying a worker slot for the whole wait. It now defers like any other deferrable sensor: the triggerer waits for the endpoint to respond without error, the check runs on the worker, and if it returns False the task defers again, keeping the poke_interval pacing.

The callable stays on the worker; only the response is carried back, reusing the JSON serializer already in triggers/http.py. This is the same pattern S3KeySensor uses for check_fn.

Verified with a live Dag

An HttpSensor(deferrable=True, poke_interval=20) polling this Airflow's own api-server, with a response_check that only passes two minutes after the run was triggered.

Before the change the task never left running, since a response_check forced it onto the synchronous path.
After the change it goes straight to deferred, wakes every 20 seconds to evaluate the check on the worker, and succeeds once the check passes:

image
Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)
    • Opus 4.8

  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst, in airflow-core/newsfragments. You can add this file in a follow-up commit after the PR is created so you know the PR number.

@yuseok89
yuseok89 force-pushed the fix-deferrable-httpsensor-response-check branch from b500a52 to 0ced540 Compare August 22, 2026 16:42
@yuseok89
yuseok89 marked this pull request as ready for review August 23, 2026 23:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Deferrable HttpSensor does not move to Triggerer when using response_check

1 participant