Skip to content

task events missing or misleading around shutdown_delay #26829

@tgross

Description

@tgross

When a task has a shutdown_delay, we emit a "Waiting for shutdown delay" event at the start of the delay. Then when that delay expires we emit another event for "Killing" where we terminate the task. But this order is confusing because it looks like we're sending SIGINT much earlier than we should be. Example using a 30s task-level shutdown_delay:

Recent Events:
Time                       Type                        Description
2025-09-22T15:23:33-04:00  Killed                      Task successfully killed
2025-09-22T15:23:33-04:00  Terminated                  Exit Code: 137, Exit Message: "Docker container exited with non-zero exit code: 137"
2025-09-22T15:22:58-04:00  Waiting for shutdown delay  Waiting for shutdown_delay of 30s before killing the task.
2025-09-22T15:22:58-04:00  Killing                     Sent interrupt. Waiting 5s before force killing
2025-09-22T15:21:41-04:00  Started                     Task started by client
2025-09-22T15:21:40-04:00  Task Setup                  Building Task Directory
2025-09-22T15:20:48-04:00  Received                    Task received by client

When shutdown_delay is set at the group level, we don't get the event at all, so the task events look like the allocation is just hung rather than pending shutdown, although it does shut down as we'd expect. Example using a 30s group-level shutdown_delay:

Recent Events:
Time                       Type        Description
2025-09-22T15:40:50-04:00  Killed      Task successfully killed
2025-09-22T15:40:50-04:00  Terminated  Exit Code: 137, Exit Message: "Docker container exited with non-zero exit code: 137"
2025-09-22T15:40:45-04:00  Killing     Sent interrupt. Waiting 5s before force killing
2025-09-22T15:39:04-04:00  Started     Task started by client
2025-09-22T15:39:03-04:00  Task Setup  Building Task Directory
2025-09-22T15:38:19-04:00  Received    Task received by client

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions