AWS Builder Center

What Building a Mini-Kubernetes Taught Me About Control Planes

Docker and Podman feel like magic until you build one yourself. I implemented a minimal container runtime in C using only Linux kernel primitives — namespaces, cgroups v2, capabilities, seccomp, and pivot_root — and came away understanding what “containers aren’t real” actually means.

What Building a Mini-Kubernetes Taught Me About Control Planes

I spent a few weeks building a container orchestrator from a proof of concept up to something that actually works: Pods, ReplicaSets, Services, a REST API modeled on Kubernetes, and containers running under Podman. Roughly 1,000 lines of Python.
Kubernetes is around two million lines. So I want to be careful about what I'm claiming here. What I got wasn't a Kubernetes replacement — it was an answer to a question I'd been carrying around for a while: what is a control plane, actually? I'd used  kubectl apply  plenty of times without a clear mental model of what happened after I hit enter.
Here's what I found.

Declarative isn't a style choice, it's an architecture

Before this, I thought "declarative configuration" meant a nicer way to write commands. You describe what you want in YAML instead of running  docker run  fifteen times. Syntax sugar.
That's wrong, and building the thing is what made it obvious.
The API server in my orchestrator does almost nothing. It validates a request, writes a resource into an in-memory store, and returns. It doesn't start containers. It doesn't talk to Podman at all. When you POST a ReplicaSet asking for three replicas, the response comes back before a single container exists.
The actual work happens somewhere else entirely — in controllers, each running its own loop, each comparing what the store says should exist against what Podman says does exist, and closing the gap.
That separation is the whole design. Writing intent and realizing intent are different jobs, running at different times, and they only ever meet through a shared description of the desired state.

The reconciliation loop is the entire trick

My controllers run every second. Each one does the same three things:
  1. Read desired state from the store.
  1. Query the world to find actual state.
  1. Make one step toward closing the difference.
The ReplicaSet controller reads that you want three pods with the label  app: health , counts how many exist, and creates or deletes to match. Then it sleeps and does it again. Forever.
What surprised me is how much you get for free from this. I never wrote code for "handle a container crashing." I never wrote a recovery path. But if you  podman rm -f  one of my pods, another appears within a second — not because anything detected the failure, but because the next reconciliation pass counted two where it wanted three.
Self-healing isn't a feature in this design. It's a side effect of never assuming your last action worked.
That reframed error handling for me. In imperative code, a failed operation is a problem you must handle right there, because the moment has passed. In a reconciler, a failed operation is fine. You'll be back in a second. The loop is the retry.

Ownership, and the problem of telling pods apart

Once a ReplicaSet is creating pods, a question appears that I hadn't anticipated: which pods are yours?
If I create a standalone pod labeled  app: health  by hand, and a ReplicaSet wants three pods matching  app: health , does my pod count toward the three? Should the ReplicaSet delete it when scaling down?
Kubernetes solves this with owner references, so I did too — each pod created by a controller carries a reference to its creator, and the controller only counts and deletes pods it owns. But I want to flag that I only understood why that field exists after hitting the ambiguity myself. Reading the API docs,  ownerReferences  looks like bookkeeping. It isn't. It's what stops two controllers from fighting over the same object.

Service discovery, without writing a DNS server

Services were the part I most expected to be painful. Load balancing, virtual IPs, DNS resolution inside the cluster — that sounded like a lot.
It turned out Podman had already solved it. When you attach containers to a user-defined network, you can give each one network aliases, and containers can reach each other by those names. So a Service in my system is mostly a matter of registering the service name as an alias on every pod that matches its selector.  curl http://health-service:2000  from another container then just works, and the container runtime handles the round-robin.
The lesson I took from this is less about Kubernetes and more general: a large fraction of "hard" distributed systems problems in a project like this are already solved one layer down. The skill is recognizing which layer owns the problem before you start writing code for it.

Two things that bit me

Concurrency. Multiple controller threads plus a web server touching one in-memory store is exactly the setup where state gets quietly corrupted. Every read and write in my resource store goes through a reentrant lock. Reentrant specifically, because controller code paths call into each other and a plain mutex deadlocks the moment one locked method calls another.
State that outlives the process. My control plane keeps everything in memory, so restarting it wipes the store — but the containers it started are still running, orphaned. The fix was to prefix every managed container's name and sweep for leftovers on startup. It's a small detail that taught me something real about why production control planes persist their state to etcd rather than RAM.

What I deliberately didn't build

No multi-node scheduling. No rolling updates. No persistent state, no autoscaling, no health probes. One container per pod.
Every one of those omissions is a place where the real system is meaningfully harder than mine, and I'd rather be precise about that than imply otherwise. What I have is a working model of the core loop, not a distributed system.
But the core loop turns out to be the part that explains the rest. Once you've written a reconciler, a Deployment stops looking like a mysterious resource type and starts looking like a controller that manages ReplicaSets the same way a ReplicaSet manages Pods. Same pattern, one level up.
That's the thing I actually took away. Kubernetes isn't a pile of features. It's one idea — compare desired to actual, take a step, repeat — applied recursively until it covers a data center.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article