Fix TimeSensor start_from_trigger behavior - #69925
Conversation
|
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
|
be6a7a9 to
caf1636
Compare
aaron-y-chen
left a comment
There was a problem hiding this comment.
Thanks for the PR :)
| if start_from_trigger: | ||
| raise ValueError( | ||
| "TimeSensor does not support start_from_trigger=True. The target moment is " | ||
| "computed fresh from the current wall-clock time on every Dag parse, so baking " | ||
| "it into the serialized trigger arguments makes the serialized Dag hash change " | ||
| "on every parse. Use deferrable=True instead, which computes the target moment " | ||
| "at task execution time and does not have this problem." | ||
| ) |
There was a problem hiding this comment.
I'm afraid this could cause a backward compatibility issue. Perhaps we should not raise an error directly in the initialization phase.
There was a problem hiding this comment.
Good point, you're right, a hard error at parse time is too risky here. Updated it: start_from_trigger=True now just emits a deprecation warning and gets ignored (falls back to False), instead of raising. So existing DAGs using it won't break on upgrade, they'll just see a warning telling them to switch to deferrable=True. Updated the test and docs to match. Let me know if that works better.
caf1636 to
e15a008
Compare
| def poke(self, context: Context) -> bool: | ||
| self.log.info("Checking if the time (%s) has come", self.target_datetime) | ||
|
|
||
| # self.target_date has been converted to UTC, so we do not need to convert timezone | ||
| return timezone.utcnow() > self.target_datetime | ||
| target_datetime = self.target_datetime | ||
| self.log.info("Checking if the time (%s) has come", target_datetime) | ||
| return timezone.utcnow() > target_datetime |
There was a problem hiding this comment.
This change doesn’t seem to be meaningful? (Unless it’s doing a race condition)
I’d revert it.
There was a problem hiding this comment.
This one's intentional, your parenthetical guess is right. target_datetime is now a computed property (calls datetime.now() fresh on every access), so calling self.target_datetime twice in this method could return two slightly different instants if the wall clock ticks past the target moment between the log call and the comparison. Storing it in a local variable once guarantees the logged value and the compared value are the same instant. I'd like to keep this as-is rather than revert, let me know if you still see it differently.
| # of the DateTimeTrigger | ||
| self.target_time = target_time | ||
| self.deferrable = deferrable | ||
| self.start_from_trigger = False |
There was a problem hiding this comment.
Can this attribute be removed if it’s always False? And can it be set to True?
There was a problem hiding this comment.
No, it can't be set to True anymore, it'll always be False now. With the switch to a deprecation warning instead of a hard error, passing True just triggers the warning and silently falls back to False. Keeping the attribute itself around for backward compatibility, since TimeSensorAsync in this same file already does the same thing (deprecated flag kept as an inert attribute instead of being removed outright).
There was a problem hiding this comment.
Keeping the value is a good idea, but I don’t like it being an attribute since the user would try to change it and see it’s silently ignored. I would change it to a property instead (with a deprecation warning) so setting it emits an error.
There was a problem hiding this comment.
Good point, updated it: start_from_trigger is now a property with a setter instead of a plain attribute. Reading it still always returns False, but setting it (whether at construction or afterward, e.g. op.start_from_trigger = True) now emits the same deprecation warning each time rather than only warning once and then silently ignoring further attempts. Tests updated to cover both the constructor path and direct assignment after construction.
e15a008 to
059a167
Compare
Agreed, let's settle this on the issue so it's not split across two threads. For what it's worth, my read on the tradeoff: #69746's TimeOfDayTrigger genuinely fixes more than mine does, it caught and fixed the class-level My case for the simpler deprecation approach: For reference, the current state on this PR: |
Closes: #69543
What does this PR do?
TimeSensorcomputed its target datetime in__init__, which runs at DAG parse time usingdatetime.now(self.dag.timezone). Whenstart_from_trigger=True, this value was stored directly instart_trigger_args.trigger_kwargs, which is serialized as part of the DAG. Because the value changed on every parse, the serialized DAG hash also changed on every parse, continuously creating new DAG versions.This PR:
ValueErrorwhenstart_from_trigger=Trueis used withTimeSensor, since the target moment cannot be safely computed at parse time.deferrable=Trueremains fully supported and unaffected.target_datetimeinto a property that is computed on access instead of being fixed during__init__.TimeSensordocumentation to describe the actual behavior, which uses the current wall-clock time in the DAG's timezone, instead of the previously documenteddata_interval_end/run_after-relative behavior that was never implemented.Was generative AI tooling used to co-author this PR?
Generated-by: Claude (Anthropic) following the guidelines