What steps did you take and what happened?
We use Cluster API with CAPV, to reproduce the issue we just need to trigger a KubeadmControlPlane while watching kube-node-lease leases. It doesn't happen to every worked node every time, but it's fairly consistent and the time it takes for the node to have its lease renewed varies.
- Start watching node leases, for example:
- kubectl get leases -n kube-node-lease -o wide --watch
or
- poll spec.renewTime on specific leases (e.g. kubectl get lease -n kube-node-lease -o jsonpath='{.spec.renewTime}{"\n"}' in a loop).
- Trigger a control-plane rollout using either:
-
Option A: delete one control-plane Machine (or otherwise force replacement of a CP node), or
-
Option B: clusterctl alpha rollout restart kubeadmcontrolplane --namespace
During the rollout, kubelet traffic to the API server (including lease updates) showed timeouts or long delays (e.g. client-side waits on the order of tens of seconds). In kube-node-lease, spec.renewTime for worker leases stopped advancing on a normal ~10s cadence and only moved again after a much longer gap, in some cases well over 100 seconds which made those nodes appear unhealthy or flaky from the controller-manager’s perspective
What did you expect to happen?
During a control-plane rollout, I expected worker leases in kube-node-lease to keep renewing about every ~10s and no long gaps in spec.renewTime. Replacing a CP node should not cause kubelet/API traffic requests to stall for tens of seconds
Cluster API version
capi-system v1.12.2
capv-system v1.15.2
Kubernetes version
kubernetes v1.34.4
Anything else you would like to add?
No response
Label(s) to be applied
/kind bug
One or more /area label. See https://github.com/kubernetes-sigs/cluster-api/labels?q=area for the list of labels.
What steps did you take and what happened?
We use Cluster API with CAPV, to reproduce the issue we just need to trigger a KubeadmControlPlane while watching kube-node-lease leases. It doesn't happen to every worked node every time, but it's fairly consistent and the time it takes for the node to have its lease renewed varies.
or
Option A: delete one control-plane Machine (or otherwise force replacement of a CP node), or
Option B: clusterctl alpha rollout restart kubeadmcontrolplane --namespace
During the rollout, kubelet traffic to the API server (including lease updates) showed timeouts or long delays (e.g. client-side waits on the order of tens of seconds). In kube-node-lease, spec.renewTime for worker leases stopped advancing on a normal ~10s cadence and only moved again after a much longer gap, in some cases well over 100 seconds which made those nodes appear unhealthy or flaky from the controller-manager’s perspective
What did you expect to happen?
During a control-plane rollout, I expected worker leases in kube-node-lease to keep renewing about every ~10s and no long gaps in spec.renewTime. Replacing a CP node should not cause kubelet/API traffic requests to stall for tens of seconds
Cluster API version
capi-system v1.12.2
capv-system v1.15.2
Kubernetes version
kubernetes v1.34.4
Anything else you would like to add?
No response
Label(s) to be applied
/kind bug
One or more /area label. See https://github.com/kubernetes-sigs/cluster-api/labels?q=area for the list of labels.