Shorten decommissioned/dead node grace periods to prevent digest overflow - #6653
Conversation
| bytes = { version = "1", features = ["serde"] } | ||
| bytesize = { version = "2.4", features = ["serde"] } | ||
| bytestring = "1.5" | ||
| chitchat = "0.11.1" |
There was a problem hiding this comment.
i also increased the dead node cache size in chitchat.
…flow When a cluster is unstable for a long period of time, nodes accumulate in the marked-for-deletion and failure-detector state without being garbage collected, since both grace periods were set to 2 hours. This let the chitchat digest grow past 500 nodes after a gossip round, exceeding the largest transmittable digest size and causing nodes to fail. Shortening both grace periods to 15 minutes lets stale nodes be GC'd promptly, keeping the digest bounded.
835a134 to
bb6fa43
Compare
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 835a1344b2
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| bytesize = { version = "2.4", features = ["serde"] } | ||
| bytestring = "1.5" | ||
| chitchat = "0.11.1" | ||
| chitchat = "0.11.2" |
There was a problem hiding this comment.
This workspace dependency now requires chitchat 0.11.2, but the committed quickwit/Cargo.lock still pins the package to 0.11.1. Any build or test path that honors the checked-in lockfile, such as --locked/--frozen or vendored/reproducible builds, will either fail resolution or not be able to use the dependency version this fix relies on, so the lockfile needs to be regenerated and committed with this manifest change.
Useful? React with 👍 / 👎.
3 participants
Summary
When a cluster is unstable for a long period of time, nodes accumulate in the marked-for-deletion and failure-detector state without being garbage collected, since both grace periods were set to 2 hours. This let the chitchat digest grow past 500 nodes after a gossip round, exceeding the largest transmittable digest size and causing nodes to fail.
MARKED_FOR_DELETION_GRACE_PERIODfrom 2 hours to 15 minutesdead_node_grace_periodfrom 2 hours to 15 minutesTest plan
cargo clippy --workspace --all-features --tests(blocked locally:chitchat = "0.11.2"isn't published on crates.io yet)cargo nextest run -p quickwit-cluster --all-features