-
Notifications
You must be signed in to change notification settings - Fork 2k
Open
Description
When a task has a shutdown_delay
, we emit a "Waiting for shutdown delay" event at the start of the delay. Then when that delay expires we emit another event for "Killing" where we terminate the task. But this order is confusing because it looks like we're sending SIGINT much earlier than we should be. Example using a 30s task-level shutdown_delay
:
Recent Events:
Time Type Description
2025-09-22T15:23:33-04:00 Killed Task successfully killed
2025-09-22T15:23:33-04:00 Terminated Exit Code: 137, Exit Message: "Docker container exited with non-zero exit code: 137"
2025-09-22T15:22:58-04:00 Waiting for shutdown delay Waiting for shutdown_delay of 30s before killing the task.
2025-09-22T15:22:58-04:00 Killing Sent interrupt. Waiting 5s before force killing
2025-09-22T15:21:41-04:00 Started Task started by client
2025-09-22T15:21:40-04:00 Task Setup Building Task Directory
2025-09-22T15:20:48-04:00 Received Task received by client
When shutdown_delay
is set at the group level, we don't get the event at all, so the task events look like the allocation is just hung rather than pending shutdown, although it does shut down as we'd expect. Example using a 30s group-level shutdown_delay
:
Recent Events:
Time Type Description
2025-09-22T15:40:50-04:00 Killed Task successfully killed
2025-09-22T15:40:50-04:00 Terminated Exit Code: 137, Exit Message: "Docker container exited with non-zero exit code: 137"
2025-09-22T15:40:45-04:00 Killing Sent interrupt. Waiting 5s before force killing
2025-09-22T15:39:04-04:00 Started Task started by client
2025-09-22T15:39:03-04:00 Task Setup Building Task Directory
2025-09-22T15:38:19-04:00 Received Task received by client