Skip to content

BM-2707: fix(broker): stop supervisor respawning tasks during shutdown, add force shutdown - #1791

Open
jonastheis wants to merge 1 commit into
mainfrom
jonas/fix-shutdown-task-respawn
Open

BM-2707: fix(broker): stop supervisor respawning tasks during shutdown, add force shutdown#1791
jonastheis wants to merge 1 commit into
mainfrom
jonas/fix-shutdown-task-respawn

Conversation

@jonastheis

Copy link
Copy Markdown
Contributor

Summary

Fixes two shutdown UX issues in the broker:

  1. Supervisor restart loop during shutdown — after CTRL+C, the task supervisor would keep respawning tasks (ChainMonitorV2, offchain market monitor) on recoverable errors instead of exiting cleanly. This happened because Supervisor::spawn() entered the Recover arm with no check for whether the cancellation token was already set. Now it checks cancel_token.is_cancelled() first and breaks the loop instead of spawning a replacement.

  2. No way to force immediate shutdown — the graceful shutdown can take up to 2 hours waiting for in-progress orders to drain. A second CTRL+C during shutdown now cancels the critical token immediately. The first shutdown message also hints at this.

Changes

  • crates/broker/src/task.rs — add cancel_token.is_cancelled() check at the top of the Recover arm in Supervisor::spawn()
  • crates/broker/src/lib.rs — update first-signal messages to hint about force shutdown; wrap Phase 1 + Phase 2 in tokio::select! that races against a second ctrl_c()

…rce shutdown

- Supervisor now checks cancel_token.is_cancelled() before spawning a
  replacement on Recover errors, stopping the restart cascade visible
  in logs after CTRL+C
- First shutdown signal now hints "Press CTRL+C again to force
  immediate shutdown"
- Second CTRL+C during shutdown immediately cancels critical tasks
  instead of waiting for the grace period
@github-actions github-actions Bot changed the title fix(broker): stop supervisor respawning tasks during shutdown, add force shutdown BM-2707: fix(broker): stop supervisor respawning tasks during shutdown, add force shutdown Mar 25, 2026
@linear

linear Bot commented Mar 25, 2026

Copy link
Copy Markdown

@austinabell austinabell left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sigkill was basically the pattern to use if you wanted to force shutdown. Was intentional to avoid shutting down with two sigint signals, to avoid the case where an in progress order wouldn't complete. Did you verify that docker compose doesn't send multiple sigints when shutting down? Risk is that accidentally or with some other infra setup, the broker would accidentally be shut down mid order and be slashed, but I think this is fine if at least our default flow doesn't do this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants