Summary
Similar to how StorageNodeOps was introduced to model discrete, trackable operations on storage nodes (suspend, resume, restart, migrate), we should introduce a ClusterOps custom resource to handle cluster-level operations that are currently embedded directly in the SimplyblocksStorageCluster reconciler.
Motivation
- The cluster controller currently mixes steady-state reconciliation with long-running, multi-step operations (e.g.
Activate, Expand), making it hard to track progress, retry safely, or audit history.
StorageNodeOps proved this pattern works well: a dedicated CR with a phase/status state machine, a Triggered guard to prevent duplicate backend calls, and clear Succeeded/Failed terminal states.- Without a dedicated CR, partial failures (e.g. backend POST succeeds but status patch fails) are harder to recover from safely.
Proposed Scope
- New
ClusterOps CRD with:
spec.type — operation type (Activate, Expand, etc.)status.phase — state machine mirroring StorageNodeOps (Pending → Running → Succeeded / Failed)status.triggered — guard flag to prevent duplicate backend calls on retry
SimplyblocksStorageCluster controller delegates operation execution to ClusterOps instead of handling it inlineClusterOps controller owns the full lifecycle of each operation
Benefits
- Per-operation history and auditability
- Safe retries with exponential backoff without risk of duplicate backend calls
- Cleaner separation of steady-state reconciliation from one-shot operations
- Consistent operator pattern across node-level and cluster-level operations
Related
StorageNodeOps controller — reference implementationSimplyblocksStorageCluster controller — current inline operation handling
Summary
Similar to how
StorageNodeOpswas introduced to model discrete, trackable operations on storage nodes (suspend, resume, restart, migrate), we should introduce aClusterOpscustom resource to handle cluster-level operations that are currently embedded directly in theSimplyblocksStorageClusterreconciler.Motivation
Activate,Expand), making it hard to track progress, retry safely, or audit history.StorageNodeOpsproved this pattern works well: a dedicated CR with a phase/status state machine, aTriggeredguard to prevent duplicate backend calls, and clearSucceeded/Failedterminal states.Proposed Scope
ClusterOpsCRD with:spec.type— operation type (Activate,Expand, etc.)status.phase— state machine mirroringStorageNodeOps(Pending→Running→Succeeded/Failed)status.triggered— guard flag to prevent duplicate backend calls on retrySimplyblocksStorageClustercontroller delegates operation execution toClusterOpsinstead of handling it inlineClusterOpscontroller owns the full lifecycle of each operationBenefits
Related
StorageNodeOpscontroller — reference implementationSimplyblocksStorageClustercontroller — current inline operation handling