Skip to content

Make ingester and compactor decommission timeouts configurable - #6640

Merged
guilload merged 2 commits into
mainfrom
guilload/configurable-decommission-timeout
Jul 30, 2026
Merged

Make ingester and compactor decommission timeouts configurable#6640
guilload merged 2 commits into
mainfrom
guilload/configurable-decommission-timeout

Conversation

@guilload

@guilload guilload commented Jul 29, 2026

Copy link
Copy Markdown
Member

Description

Per title.

How was this PR tested?

Added unit tests

@guilload
guilload force-pushed the guilload/configurable-decommission-timeout branch from 9664b23 to 3615a19 Compare July 29, 2026 16:35
@guilload
guilload marked this pull request as ready for review July 29, 2026 16:44
@guilload
guilload requested review from a team as code owners July 29, 2026 16:44
@guilload
guilload requested a review from loutPhilipps July 29, 2026 16:44
@guilload
guilload force-pushed the guilload/configurable-decommission-timeout branch from 3615a19 to 04abf8c Compare July 29, 2026 16:44

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3615a19d1e

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

// amount of time with which we can fully decommission an ingester. This will be
// most noticeable when using Quickwit locally.
Duration::from_secs(10)
Duration::from_secs(15)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Guard ingester timeouts below the drain delay

When ingest_api.decommission_timeout or QW_INGEST_DECOMMISSION_TIMEOUT is set below this 15s delay, shutdown will always time out before the ingester can even switch from Retiring to Decommissioning and close shards, even if the queues are already empty. That makes the new timeout knob ineffective for short Kubernetes termination grace periods; consider rejecting values at or below this delay or making the delay part of the configured shutdown budget.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stupid comment

@guilload
guilload force-pushed the guilload/configurable-decommission-timeout branch 2 times, most recently from 9e514d8 to 8233cda Compare July 30, 2026 15:31
@guilload
guilload force-pushed the guilload/configurable-decommission-timeout branch from 8233cda to 5de4895 Compare July 30, 2026 15:32
@guilload
guilload merged commit 9ad552b into main Jul 30, 2026
9 checks passed
@guilload
guilload deleted the guilload/configurable-decommission-timeout branch July 30, 2026 15:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants