Skip to content

feat(controllers): add exponential backoff rate limiter to all contro… - #388

Open
geoffrey1330 wants to merge 2 commits into
mainfrom
fix-exponential-backoff-controller-requeue
Open

feat(controllers): add exponential backoff rate limiter to all contro…#388
geoffrey1330 wants to merge 2 commits into
mainfrom
fix-exponential-backoff-controller-requeue

Conversation

@geoffrey1330

@geoffrey1330 geoffrey1330 commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator

…llers

Add exponential backoff rate limiter to all controllers

Fixes #138.

All 18 controllers were requeueing on errors with fixed intervals, causing
thundering-herd retry storms against the backend when it was degraded. When
every controller fails at the same time, they all retry at the same wall-clock
time — keeping sbcli saturated and delaying recovery.

This wires an exponential backoff rate limiter (base 5 s, max 5 min) into
every controller's SetupWithManager via a shared defaultControllerOptions()
helper. When a controller returns ctrl.Result{}, err, successive retries for
the same object back off to 10 s, 20 s, 40 s … up to 5 min, then reset once
the object reconciles successfully.

RequeueAfter returns used for intentional polling (waiting for node online,
drain slot checks, sync intervals) are unaffected.

Fix: prevent duplicate backend calls when status patches fail after mutations

A subtle risk introduced by exponential backoff: if a backend POST succeeds
(e.g. trigger migrate, promote, suspend) but the subsequent status patch fails,
returning ctrl.Result{}, err causes the controller to back off — then retry
with Triggered=false still in status, firing the same backend call again.

Status patches that follow a successful backend mutation now return
ctrl.Result{Requeue: true}, nil instead of ctrl.Result{}, err. This
requeues immediately without triggering backoff, retrying only the status patch
until it lands. Once Triggered=true is persisted the controller moves forward
normally — no duplicate calls.

Applied across StorageNodeOps, SimplyblocksStorageCluster, and
VolumeMigration controllers.

Test Result

user@users-MacBook-Pro-2 ~ % kubectl -n simplyblock get storagecluster                           
NAME                  STATUS   UUID                                   CONFIGURED   AGE
simplyblock-cluster   active   38c2c881-afdc-4b95-813b-81ff2b15d009   true         6m27s
user@users-MacBook-Pro-2 ~ % kubectl -n simplyblock get storagenodesets
NAME               TOTAL   ONLINE   AGE
simplyblock-node   3       3        6m33s
user@users-MacBook-Pro-2 ~ % kubectl -n simplyblock get storagenode    
NAME                      WORKER                          SOCKET   NODEIDX   UUID                                   STATUS   HEALTH   AGE
simplyblock-node-9cxvxo   vm03.simplyblock3.localdomain   0        0         3fd8eb7c-ba8b-41b6-9d0e-9e011cd1b16a   online   true     5m11s
simplyblock-node-mk7iwn   vm04.simplyblock3.localdomain   0        0         74c99f22-dd3c-4df5-b1cd-4e0a540a915c   online   true     4m30s
simplyblock-node-tc85nl   vm02.simplyblock3.localdomain   0        0         df2486fc-f8be-41a3-a97f-38a3c430f2ae   online   true     6m35s

@noctarius noctarius added this to the 26.4 milestone Aug 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: Operator Uses Fixed Requeue Intervals — No Exponential Backoff on sbcli 5xx Errors

2 participants