NOTE: Terraform files will be drastically updated later as Atlantis is brought online.

GitOps 🛠️

About

This repository contains ArgoCD, Helm, and Terraform files for declarative deployments with Kubernetes, specifically k3s.

You can use these files to stand up your own on-prem Kubernetes cluster. While this repository was built to be run on Raspberry Pi devices, it should be equally valid anywhere Kubernetes can run.

If you want to implement this for yourself, please follow the setup document (which is actively being updated).

Features

• App-of-apps: A root Argo CD Application deployment schema which recursively manages child apps
• Namespace deployments: argocd, cert-manager, kube-system, logging, longhorn-system, monitoring, and applications-eng
• Cert-manager: X.509 certificate management for Kubernetes
• Longhorn: Distributed on-prem file storage with multiple storage classes
• Metal LB: An on-prem native software load balancer
• Monitoring: Prometheus Operator with Grafana using storage PVC through Longhorn
• n8n: Workflow automation platform with persistent storage
• vLLM: Runtime for AI models on a GPU node
• Dashboard UI for:
  • Argo CD: Controlling deployments and rollbacks
  • Grafana: Building dashboards against Prometheus data
  • Longhorn: Controlling the distributed block storage setup
  • n8n: Creating and managing automated workflows
  • Open WebUI: A ChatGPT-like interface paired with the vLLM deployment for inference
  • Prometheus: Querying against raw data from pods/nodes/deployment resources

Project Management

Work for this repository is housed in this Trello board.

Project Structure

├── argocd/                                      # ArgoCD application definitions
│   ├── apps/                                    # Application-level deployments
│   │   ├── applications/                        #
│   │   ├── argocd/                              #
│   │   ├── cert-manager/                        #
│   │   ├── kube-system/                         #
│   │   ├── logging/                             #
│   │   ├── longhorn-system/                     #
│   │   └── monitoring/                          #
│   ├── namespaces/                              # Namespace-level deployments
│   └── root/                                    # Root ArgoCD application deployment
├── helm/                                        # Helm charts
│   ├── argocd/                                  #
│   ├── cert-manager/                            #
│   ├── grafana/                                 #
│   ├── longhorn/                                #
│   ├── n8n/                                     #
│   ├── nvidia-device-plugin/                    #
│   ├── open-webui/                              #
│   ├── prometheus/                              #
│   ├── prometheus-operator/                     #
│   ├── prometheus-service-monitors/             #
│   ├── vllm/                                    #
└── terraform/                                   # Terraform configurations
    ├── namespaces.tf                            # Kubernetes namespace definitions
    ├── provider.tf                              # Terraform provider configuration
    └── storage-classes.tf                       # Longhorn storage class definitions

Built With

Hardware

The cluster this repo's files runs on uses Raspberry Pi 5 devices, specifically the 16gb version.

Here's the hardware list of what each of the control/worker nodes is using:

1. Raspberry Pi 5
2. NVMe + POE+ Pi 5 Hat and Active Cooler
3. Samsung 2TB NVMe SSD
4. 256gb Micro SD Card

The GPU node I am running for model inference is quite different and uses the following hardware:

1. Ncase M3 Case
2. MSI B650I Edge Wifi Motherboard
3. AMD 9800x3D CPU
4. 128gb DDR5 Corsair RAM
5. 8TB Western Digital NVMe SSD
6. Nvidia RTX 5090 FE GPU
7. Corsair SF1000 PSU
8. NZXT Kraken Elite 280mm AIO
9. Noctua 120mm Fans

I built it to be beefy enough to handle inference but also lightweight enough for me to unplug, take with me while traveling, and use as a personal computer.

Software

• Argo CD
• Cert Manager
• Grafana
• Grafana Loki
• Grafana Promtail
• Helm
• Kubernetes/K3s
• Longhorn
• Metal LB
• n8n
• Nvidia Container Toolkit
• Open WebUI
• OpenFaaS (coming soon)
• Prometheus
• Prometheus Operator
• Sealed Secrets
• Terraform
• Traefik
• vLLM

Acknowledgements

• Edede Oiwoh for inspiring me to build a home cluster and for bouncing ideas around
• rpi4cluster.com for tips on GitOps with Raspberry Pi setups (even if the notes weren't current and Helm/Argo configurations weren't file-based)
• Twitter (now X), Loom, and Tesla for teaching me proper GitOps processes and giving me a chance to move mountains with them