Things I learned breaking my Kubernetes cluster
A small post-mortem on RBAC and DNS failures in a bare-metal setup.
I intentionally broke my local cluster this weekend to see what recovering a control plane actually feels like when you don’t have AWS doing it for you.
Three things I learned:
- DNS is always the first thing to die. CoreDNS loops are incredibly easy to cause if your upstream resolv.conf points back into the cluster network.
- RBAC is harder than it looks. I accidentally locked myself out of
cluster-admin. Fixing it meant dropping down to the raw certificate level. - Etcd needs love. Running etcd on spinning disks for a control plane is a terrible idea. The disk IO latency causes leader elections to constantly fail.
Next week: attempting to intentionally partition the network.