Handle Helm chart failures during node recreation/redeployment

[Adam-D-Lewis]

Problem

When Kubernetes nodes are recreated during redeployment, Helm charts can fail to roll out due to image pulling issues, causing Terraform deployments to timeout. This was observed in PR #3125.

Observed behavior:

• Keycloak Helm chart (and others) enter failed state when new nodes can't pull Bitnami images due to repository changes
• Terraform deployment times out waiting for Helm chart rollout
• force_update=true doesn't resolve the issue as expected
• Manual intervention required: deleting failed Helm charts in k9s and redeploying

Root cause:

• Bitnami Docker image repository has changed and the old repository no longer exists

Affected charts:

• Keycloak Helm chart
• MinIO Helm chart (conda-store related)
• PostgreSQL Helm chart (conda-store related)
• MinIO Loki Helm chart

Proposed solutions

For the next release, we should address this through one of:

1. Documentation: Add troubleshooting guide for users experiencing this issue during redeployment
2. Automation: Implement logic to detect and delete failed Helm charts before redeploying
3. Configuration: Improve Helm chart resilience during node recreation (e.g., better image pull policies, retry mechanisms)

References

• Original issue encountered in PR patch images used so Nebari is usable while we wait to update helm charts #3125

[viniciusdc]

Thanks for opening this issue, @Adam-D-Lewis. One thing I am concerned about is whether we should delete the Helm charts preemptively to avoid problems with a broken registry during deployment. However, I have a concern that Nebari may attempt to recreate the PVCs, which could potentially lead to data loss. -- As context, I think Helm adds some prefix to its resources that might not properly sync with pre-existing resources from before the removal.

Let's try to patch the existing broken sections of the helm values of each resource through the nebari upgrade, instead of deleting all.

[Adam-D-Lewis]

Yeah, we could potentially run something like what's below for the upgrade step

helm upgrade <release-name> <chart> \
  --reuse-values \
  --set image.tag=new-tag

[viniciusdc]

I wonder if we could get by with kubectl instead of helm, since we currnetly don't install that as part of nebari.

Maybe patch the affected deplyments/statefulsets ?

  kubectl patch deployment my-deployment