You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
description: Complete step-by-step guide to restoring Kubernetes applications from Longhorn backups stored in MinIO, including scaling strategies, PVC management, and real-world lessons learned.
6
+
description: Oops, My App Died! The Complete Human's Guide to Rescuing Kubernetes Applications from Longhorn Backups in MinIO
7
7
image:
8
8
path: /assets/img/posts/k8s-longhorn-restore.webp
9
9
alt: Kubernetes Longhorn backup restoration guide
10
-
draft: true
10
+
draft: false
11
11
---
12
12
13
13
14
-
# Restoring Your Kubernetes Applications from Longhorn Backups: A Complete Guide
14
+
# Restoring Your Kubernetes Applications from Longhorn Backups
15
15
16
-
When disaster strikes your Kubernetes cluster, having a solid backup strategy isn't enough—you need to know how to restore your applications quickly and reliably. Recently, I had to rebuild my entire K8S cluster from scratch and restore all my applications from 3-month-old Longhorn backups stored in MinIO. Here's the complete step-by-step process that got my media stack and observability tools back online.
16
+
When disaster strikes your Kubernetes cluster, having a solid backup strategy isn't enough—you need to know how to restore your applications quickly and reliably. Recently, I had to rebuild my entire K8S cluster from scratch and restore all my applications from Longhorn backups stored in MinIO. Here's the complete process that got my media stack and observability tools back online.
17
17
18
18
## The Situation
19
19
20
-
After redeploying my K8S cluster with Flux GitOps, I found myself with:
20
+
After redeploying my K8S cluster with [Flux GitOps](https://merox.dev/blog/homelab-tour/), I found myself with:
21
21
- ✅ Fresh cluster with all applications deployed via Flux
22
22
- ✅ Longhorn storage configured and connected to MinIO backend
23
23
- ✅ All backup data visible in Longhorn UI
@@ -42,7 +42,7 @@ Before starting, ensure you have:
42
42
- Kubernetes cluster with kubectl access
43
43
- Longhorn installed and configured
44
44
- Backup storage backend accessible (MinIO/S3)
45
-
- Applications deployed (scaled up or down doesn't matter)
45
+
- Applications deployed (scaled up or down doesn't really matter)
46
46
- Longhorn UI access for backup management
47
47
48
48
## Step 1: Assess Current State
@@ -128,7 +128,7 @@ For each backup, click the **⟲ (restore)** button and configure:
128
128
Once restoration completes, the restored Longhorn volumes need PersistentVolumes to be accessible by Kubernetes:
129
129
130
130
```yaml
131
-
# Example for Jellyfin - repeat for all applications
131
+
# Example for Jellyfin - repeat for all applications you want to be restored
- **Jellyfin**: Complete media library, metadata, and user settings
229
-
- **Grafana**: All dashboards, data sources, and alerting rules
230
-
- **Prometheus**: Historical metrics and configuration
231
-
- **Loki**: Log retention policies and stored logs
232
-
- **QBittorrent**: Torrent configurations and download states
233
-
- **Sonarr**: TV show monitoring and quality profiles
234
-
235
-
### Important Considerations
236
-
237
-
1. **Data Age**: My backups were 3 months old, so any data created after that point was lost. Plan backup frequency accordingly.
238
-
239
-
2. **Storage Sizes**: Pay attention to backup sizes vs. current PVC sizes. My Prometheus backup was 45GB while the current PVC was only 15GB—the restore process required updating the PVC size.
240
-
241
-
3. **Volume Naming**: Longhorn creates restored volumes with specific names. The PV `volumeHandle` must match exactly.
242
-
243
-
4. **Application Dependencies**: Some applications have interdependencies. Restore core infrastructure (Prometheus, Grafana) before application-specific services.
244
225
245
226
## Alternative: CLI-Based Restoration
246
227
@@ -260,23 +241,9 @@ spec:
260
241
261
242
## Conclusion
262
243
263
-
Restoring Kubernetes applications from Longhorn backups requires careful orchestration of scaling, PVC management, and volume binding. The process took about 45 minutes for 6 applications, but the result was a complete restoration to the previous backup state.
264
-
265
-
Key takeaways:
266
-
- **Always scale down applications first** to prevent corruption
267
-
- **Understand the relationship** between Longhorn volumes, PVs, and PVCs
268
-
- **Test your backup restoration process** before you need it
269
-
- **Document your PVC naming conventions** for faster recovery
270
-
- **Monitor backup age** vs. acceptable data loss
244
+
Restoring Kubernetes applications from Longhorn backups requires careful orchestration of scaling, PVC management, and volume binding. The process took about 30 minutes for 6 applications, but the result was a complete restoration to the previous backup state.
271
245
272
246
Having a solid backup strategy is crucial, but knowing how to restore efficiently under pressure is what separates good infrastructure management from great infrastructure management.
273
247
274
-
## Next Steps
275
-
276
-
Consider implementing:
277
-
- **Automated backup validation** to ensure restorability
278
-
- **Backup age monitoring** with alerts
279
-
- **Documentation of critical PVC mappings**
280
-
- **Regular disaster recovery drills**
281
248
282
-
Your future self will thank you when disaster strikes again.
249
+
Your future self will thank you when disaster strikes again. 😆
0 commit comments