Not sure this is top priority, but keeping the info about this issue identified when working on PR #244.
Currently, promoteOrphanedReplicas skips TAKEOVER when persistence is configured, assuming the crashed pod will return with the same node ID and data. This is correct for transient failures (OOMKill, node reboot), but leaves the cluster stuck in degraded state when the primary cannot come back:
- Pod unschedulable (node drained, resource pressure, taint added)
- PVC bound to a failed node (local storage, zone loss)
- PVC manually deleted
- Node permanently removed from the cluster
Expected behavior:
After a configurable timeout, or based on pod status checks, the operator should detect that the primary is not coming back and promote a replica via TAKEOVER (accepting potential data loss for the unreplicated writes).
Alternatives:
- Timeout-based: If a primary remains in fail state for longer than a threshold (e.g., spec.failoverTimeout, default 5m), proceed with TAKEOVER even with persistence enabled.
- Pod-status-based: Check if the primary's pod is in an unrecoverable state (Unschedulable, PVC Lost, node deleted) and trigger TAKEOVER immediately.
- Annotation escape hatch: Allow manual intervention via valkey.io/force-takeover: "true" on the ValkeyCluster resource.
- ...
Risks:
TAKEOVER trades consensus for liveness. If the "dead" primary is actually partitioned (still serving clients on the other side of a network split), promoting a replica creates two owners for the same slots with competing epochs. With persistence, the old primary could resume serving stale data when the partition heals.
To mitigate this, the operator should:
- Confirm via the Kubernetes API that the old primary is actually gone (pod deleted, node gone), not just unreachable over the cluster bus.
- Fence the old primary (delete its pod) before issuing TAKEOVER, preventing it from coming back as a competing primary.
Context:
Not sure this is top priority, but keeping the info about this issue identified when working on PR #244.
Currently,
promoteOrphanedReplicasskipsTAKEOVERwhen persistence is configured, assuming the crashed pod will return with the same node ID and data. This is correct for transient failures (OOMKill, node reboot), but leaves the cluster stuck in degraded state when the primary cannot come back:Expected behavior:
After a configurable timeout, or based on pod status checks, the operator should detect that the primary is not coming back and promote a replica via TAKEOVER (accepting potential data loss for the unreplicated writes).
Alternatives:
Risks:
TAKEOVER trades consensus for liveness. If the "dead" primary is actually partitioned (still serving clients on the other side of a network split), promoting a replica creates two owners for the same slots with competing epochs. With persistence, the old primary could resume serving stale data when the partition heals.
To mitigate this, the operator should:
Context: