Homelab GitOps Infrastructure
“Two Nodes, One Repo, Zero Manual kubectl”
Cloud sandboxes are too abstract. You click a button and a Kubernetes cluster appears. You never feel what happens when the CNI silently kills networking and you have to debug it from a dead SSH session. I built this homelab to feel it.
The hardware
Two nodes on a home LAN:
- Raspberry Pi 4B with 8GB RAM: runs the K3s control plane. ARM64. Low power, completely silent, sits on a shelf.
- Old AMD64 box with 4GB RAM: runs all the actual workloads. Prometheus, Grafana, Loki, ArgoCD, Home Assistant, everything.
Everything is defined in this repository. If the cluster burns down, I can rebuild it from git clone and make.
The stack
K3s instead of vanilla Kubernetes. It actually runs on a Pi without eating all the RAM. Single binary, embedded etcd, bundles everything needed.
Cilium 1.18.9 as the CNI. eBPF-based networking with no kube-proxy. Hubble provides observability at the network layer. Pinned to 1.18 specifically because 1.19.x silently kills networking on ARM64. That was a fun week of debugging.
ArgoCD for GitOps. The app-of-apps pattern bootstraps everything: one root application deploys the entire cluster. I never run kubectl apply manually.
MetalLB gives services real IPs on the LAN. No NodePorts, no port forwarding. Services are reachable at their actual addresses.
Traefik handles ingress. Serves Grafana, ArgoCD UI, and Home Assistant on HTTPS with certificates from cert-manager.
Prometheus, Grafana, Loki, OpenTelemetry for the observability stack. Metrics, logs, and traces all feeding into dashboards. Loki keeps 28 days of everything.
Home Assistant running as a StatefulSet with a 10Gi PVC for home automation.
Things I learned the hard way
Cilium 1.19.x is broken on ARM64. Silent host network death. The SSH daemon stays up, so the machine looks alive, but all TCP connections timeout. No errors in any log. I had to physically attach a keyboard and monitor to figure out what was happening. Pinned to 1.18.9 until upstream fixes it.
Do not run application pods on the control plane with Cilium. The ClusterIP for the API server becomes unreachable from pods on the same node. Just keep the control plane tainted and let it be a control plane.
4GB RAM is tight. With Prometheus, Loki, Grafana, ArgoCD, and Home Assistant all competing for memory on the worker node, I had to tune aggressively. Loki’s memcached caches went from 10GB to 128MB. Traefik went from 256MB to 384MB. It works, but barely. The next upgrade is more RAM.
ArgoCD Helm diffs cannot handle multi-line ConfigMap data. Helm renders it as quoted strings. Kubernetes stores it as block scalars. ArgoCD sees them as totally different resources and marks the application as out of sync forever. Fixed with ignoreDifferences in the Application manifest.
kube-bench on K3s shows false positives. The tool was written for kubeadm paths. K3s puts everything under /var/lib/rancher/k3s/. Wrote a filter script to suppress the irrelevant findings.