Building a Kubernetes cluster using managed services like EKS or GKE hides a lot of the magic. I wanted to understand the magic, so I decided to build a GitOps-driven cluster on bare-metal infrastructure.

This isn’t a tutorial. Instead, these are my notes on what broke, what I learned, and what finally made the architecture click for me.

The Goal

The goal was simple: I wanted a state where I could accidentally delete my entire cluster, run a single script, and have everything—infrastructure, networking, and applications—rebuild itself exactly as it was, entirely from a Git repository.

The Architecture (and the mistakes)

I chose Terraform for the base infrastructure (provisioning nodes) and FluxCD for the cluster state.

My first mistake was trying to make Terraform do too much. I initially tried using the Terraform Helm provider to deploy cluster-level services. It created a nightmare of state dependency.

Lesson learned: Terraform is for infrastructure. Flux is for what runs on it. Never cross the streams.

The “Aha” Moment with Flux

Flux’s reconciliation loops were confusing until I stopped thinking of them as scripts and started thinking of them as a thermostat.

When you define a Kustomization in Flux, you aren’t saying “run kubectl apply.” You are setting the thermostat to 72 degrees. Flux is the AC unit constantly checking if the room is 72 degrees. If someone manually edits a deployment (opening a window), Flux immediately corrects it.

The Networking Rabbit Hole

I used Cilium as my CNI. BGP (Border Gateway Protocol) always sounded like dark magic used by ISPs. Configuring Cilium to announce LoadBalancer IP addresses to my local router via BGP was the most satisfying moment of the project.

Watching my router automatically learn routes to pods without any NAT was beautiful. It made the network feel alive.

Conclusion

This project completely changed how I view infrastructure. It stopped being a collection of servers and started being a declarative, evolving system. I am currently working on expanding this setup to handle multi-cluster GitOps, which you can follow in my Garden.