Booting a Kubernetes Homelab From Cold Metal

A few years ago my home server Gideon 2 died. One PSU failure took down media, documents, backups, passwords — every VM I had on it. The whole experience left me allergic to single points of failure in my own house. The chassis is still racked today, powered down and hollowed out: I've donated the motherboard, CPU, RAM and PSU to people who'll get more use from them than I will. What's left is twelve drives of mixed capacity (4TB to 14TB) still in their cages, and a tangle of PCIe cards and SAS cables dangling in an empty shell — the carcass left where it died.

I wanted to rebuild it as a cluster. Not "a bunch of machines I SSH into one by one" but a real cluster, where nothing is special, no node feels precious, and I never have to look at it. Plug new hardware in, power it on, walk away.

That's what Project Iluvatar is. A homelab that boots itself from cold metal, with the whole thing reproducible from one Git repo.

The Goal

Problem: Every homelab is a snowflake

Most homelab K8s setups start the same way. Flash an SD card. Install the OS. SSH in. Configure networking. Install k3s. Configure storage. Repeat for every node, plus all the workloads on top, plus the backups, plus the auth. Then a node dies, you reach for that one SD card you imaged differently six months ago, and the whole house of cards wobbles.

Every step is manual. Every node is slightly different. The thing you built doesn't exist anywhere except on that hardware.

Solution: Make the cluster reproduce itself

What if no node held any state worth caring about? What if a fresh, unconfigured Raspberry Pi could plug into the network and the cluster would just... boot it? Configure it? Recruit it?

That's Iluvatar. Git is the source of truth. The cluster bootstraps itself from Git. Adding a node is plugging it in.

The harder bar: reproducible after the world ends

"Reproducible" is easy to say and easy to quietly cheat on. The bar I actually hold this to: imagine the internet is gone and isn't coming back. Could I still rebuild this from cold metal?

That framing kills a lot of the usual shortcuts. No Let's Encrypt — ACME has to phone home, and a cert that silently expires while the lab sits offline for months is a time bomb waiting in a cupboard. No leaning on a hosted control plane that could disappear, change hands, or start charging. No "just pull it from Docker Hub" if Docker Hub isn't there that day.

It's a deliberately paranoid bar for a homelab, and I don't fully clear it yet — charts and images still pull from upstream at build time (it's on the backlog). But it's the North Star: every external dependency becomes a question of "what happens when this goes away?", and the answer should always be "I cached it, and I carry on."

Two Clusters, One Repo

The architecture is two co-located clusters with very different jobs. They share a single Git repo, share the same network, but otherwise stay out of each other's way.

Ainur - the Pi bootstrap cluster

A small Raspberry Pi cluster that exists for one reason: to bring the rest of the lab to life. It serves DHCP, DNS, TFTP, HTTP and the GitOps seed. Anything you plug into the network can netboot from it and self-configure. The Pis themselves netboot too, there are no SD cards in the rack.

Three Pi 5s on GeeekPi P33 M.2 NVMe PoE+ HATs (one control plane with 16GB, two workers with 8GB), each with a 128GB NVMe drive. Power and network come over a single PoE+ cable, so each node is one wire away from being deployed. About £100 per node.

Arda - the x86 workload cluster (work in progress)

A Talos Linux cluster on cheap refurbished x86 boxes that will eventually run everything else. Talos is an immutable, API-driven Linux specifically for Kubernetes: no SSH, no package manager, no editable filesystem. Configuration is declarative, the node is disposable. The plan is for the Pis to bring it up via Sidero Omni and ArgoCD to take over from there.

Hardware is HP ProDesk 400 G4 USFF Mini PCs (i5-8500T, 16GB RAM, 256GB NVMe for Talos, 4TB SATA SSD for Longhorn), around £110 a unit. They're physically in the rack and powered, but the Talos-via-Omni bring-up is still being finalised (currently mid-migration from Sidero Omni v1 to v2). The design is settled, the wiring isn't.

Eä - the physical environment

A Ubiquiti Dream Machine SE acts as router and DHCP server. A USW-Pro-24-PoE switch handles the network plus per-port PoE power, which is great because it means I can power-cycle a Pi from a UI on my phone. A JetKVM gives me remote console / image-mount on any node when I need to debug something headless.

Total idle power for the whole lab is around 50W. The two clusters together pull less than my old single home server did sitting idle.

The full picture, from the project docs:

The Self-Reproducing Bootstrap

The thing I'm most pleased with is that the cluster hosts the netboot stack that boots the cluster. It's circular on purpose, and it's the bit that makes the rest of the design work.

The very first Pi is brought up by a Multipass VM on my workstation that serves a minimal TFTP/NFS image. As soon as that first Pi is alive, it joins the bootstrap cluster, takes over the netboot services as a Kubernetes DaemonSet, and the VM goes away. Every subsequent Pi netboots from the Pis already in the cluster. The x86 nodes will netboot from them the same way once the Talos/Omni side is in.

The whole thing assembles itself in layers, and the only "scaffolding" outside the cluster is one short-lived VM that exists for about ten minutes.

How it actually works

A few pieces conspire to make this not-a-disaster:

• kube-vip advertises a stable Virtual IP (192.168.1.59 on my network) at layer 2 via ARP. The DHCP server hands out that IP as the boot server. Whichever Pi is currently the leader replies. Nodes that boot don't care which physical Pi answered, they just need the IP to be there.
• TFTP, NFS and HTTP run as hostNetwork DaemonSets pinned to the right ports. One pod per node, every node can serve. Boot files live on Longhorn RWX PVCs so any pod can read them.
• A Builder Job fetches the upstream Ubuntu Server ARM image, extracts it, patches cloud-init to point at the cluster, and stages the result into the PVCs. Declaratively. From Git.
• Cloud-init NoCloud lives at the HTTP endpoint. New Pis pick up their user-data and meta-data from the VIP, format their NVMe, run the k3s leader/agent election, and join.

The Nerdy Naming

A small Tolkien-flavoured indulgence. From the docs:

In the Ainulindalë, Ilúvatar sings the world into being; the Ainur join the song and shape it.

Same vibe here: this repo (Iluvatar) starts the music. It brings the first node(s) to life and lays down the platform. Then Ainur, the cluster and its controllers (Argo CD, Longhorn, Omni, etc.), pick up the melody and bring the rest of the lab into existence, continuously.

The names map cleanly onto the architecture:

• Iluvatar - the Git repo, the song that starts everything
• Eä - the physical environment, network, racks
• Ainur - the Pi bootstrap cluster
  • Valar - the bootstrap primaries (PXE, Git, ArgoCD seed)
  • Maiar - the bootstrap secondaries (resilience + aux services)
• Arda - the x86 main cluster
  • Valinor - control plane nodes
  • Istari - worker nodes

It's silly. I love it.

What Actually Runs On It

Everything that holds the cluster together is on Ainur today: ArgoCD, Headlamp, Dex, Longhorn, cert-manager, trust-manager, kube-vip, Sidero Omni, the in-cluster netboot stack.

Two pieces of connective tissue make that list behave like one platform instead of nine separate apps:

One CA, every app, no manual labelling. Self-hosted clusters can't use Let's Encrypt, and the usual answer is clicking around pasting a CA cert into every namespace — exactly the toil this project exists to kill. Instead a Root CA is generated once at first boot, cert-manager issues from it, and trust-manager distributes the bundle to every namespace labelled to receive it. Every app that asks for a cert gets one; every app that needs to trust the CA gets it. Zero manual steps.

One login, every app. Dex provides an OIDC issuer that Headlamp, ArgoCD, Sidero Omni and anything else I want gated all use as their identity provider. Log into one, you're into all of them. No per-app user database.

Once Arda is on, the workload plan is:

• Jellyfin - self-hosted media server with hardware-accelerated transcoding from the i5's iGPU
• Paperless-ngx - document ingestion, OCR, search. Replaces a filing cabinet
• Bitwarden - self-hosted password manager with shared vaults across the family
• Crafty Controller - Minecraft server manager with world backups and mod controls
• LlamaGPT - a local LLM I can use when I don't want to send a prompt to a third party

What's Not Done

Honest list, from my internal backlog. The project deliberately tracks what's not hardened, so:

• Arda / x86 cluster - the hardware is in the rack but the Talos-via-Omni bring-up is still being finalised. This is the next big chunk and gates a lot of the workload plan.
• Observability (Grafana / LGTM) - no Prometheus, no Grafana, no Loki yet. Queued behind Arda since it needs somewhere to actually run that isn't competing with the Pis for resources. I'd notice a fire by my phone alerts going stale.
• SOPS for secrets - dev-grade values are sitting in plaintext in the public repo. The Dex admin password is the literal bcrypt(ubuntu). Brute-forceable on a laptop. Trusted-LAN-only mitigates this for now, real fix is queued.
• Velero for backups - Longhorn replicates but there's no off-cluster backup. A bad replica race could lose a volume. For Minecraft worlds and family password vaults that's "what was the point of this cluster again," so it's the obvious next thing.
• DR drill - I claim it's self-reproducing. I haven't actually torn it down and watched it rebuild itself from scratch in a long time. That drill is overdue.
• Re-measure the timings - I used to be able to quote times for the bootstrap from cold metal to ArgoCD ready. They've drifted as the workload list grew and I haven't sat with a stopwatch in months.
• etcd encryption at rest - currently off
• Renovate bot - K3s and Helm chart versions are pinned but not automatically PR'd for CVEs
• True offline reproducibility - the "world has ended" bar from up top isn't actually met yet. Helm charts and container images still pull from upstream at build and boot time. The fix is caching them into the bootstrap VM and the generated netboot images, so a cold rebuild needs exactly zero internet. It's the through-line behind a lot of the other choices (self-signed CA, no ACME), and the thing that would make the paranoid framing true instead of aspirational.

The list is on display. None of it is a surprise.

Built To Be Honest

The thing I'm proudest of, after the ouroboros bit, is that the project doesn't pretend. That runs two ways: it writes down the parts that were genuinely horrible to get working, and it's just as blunt about the corners that are cut on purpose.

The bits that fought back

A few of these workarounds are the kind of thing I wanted recorded somewhere so I never have to rediscover them at 1am.

NFSv3 needs port 111. So does the host.

The Pi firmware can't boot from NFSv4 (there's an old Launchpad bug), so we're stuck on NFSv3, which requires rpcbind listening on UDP/111. Trouble: every Linux host runs its own rpcbind on UDP/111 by default. Two processes can't bind the same UDP port, so the pod fails.

The fix is a privileged init container that uses nsenter to break out into the host namespace and disable the host's rpcbind, freeing the port for the pod to take. The pod's NFS server then comes up cleanly. As crimes go, it's a small one.

TFTP needs predictable ports too

TFTP starts on UDP/69 but immediately negotiates ephemeral data ports for the actual transfer. By default those ports are random, which makes firewall rules impossible. The fix is pinning the data ports to a known range (5000-5010), exposing them via hostPort, and the firewall stays stable across pod restarts.

inotify caps bite when you least expect them

rpc.mountd opens an inotify watch per export. Stock kernel limits are too low for a few exports plus everything else on the node. Another init container bumps fs.inotify.max_user_instances and max_user_watches before NFS starts.

Cloud-init from a PVC, k3s tokens from a hostPath

The HTTP bootstrap pod serves two things from two sources: cloud-init from a shared PVC (so every DaemonSet pod sees the same content) and the k3s leader IP + agent token from a host-local path (produced by the leader-election scripts on each node). Both end up under one nginx via a sneaky bind-mount layer — PVC at /, hostPath at /k3s/ — and the nginx config doesn't have to care.

The corners I cut on purpose

The flip side of documenting the hard bits is being just as clear about the soft ones.

It runs on a LAN I trust, and it leans on that hard. The Root CA's private key sits on every node. The Dex admin password is dev-grade — literally bcrypt(ubuntu), brute-forceable on a laptop — sitting in plaintext in a public repo. The k3s agent join token is served unauthenticated over HTTP; anyone who can reach the boot VIP on the LAN can enrol a node. None of that is an oversight. It's a single-site hobby cloud where the trust boundary is the front door, and BACKLOG.md is explicit about exactly which threat models it does and doesn't address.

It doesn't claim to be production-grade, it doesn't try to be multi-tenant, and it doesn't pretend it has a real HA story below three nodes. The docs flag when something is "spicy" and link the upstream bug. The CA distribution is automatic — and I'll happily admit the private key is everywhere.

It's a hobby cloud, on one site, on hardware I can touch, with every trade-off that comes with that. Designing it any other way would be cosplay.

Closing

The full technical write-up, with PXE boot from scratch, ephemeral K3s clusters, self-signed CA chains, Cloudflare tunnels, Talos via Sidero Omni and the rest, is on the project's documentation site. Architecture diagrams, build-log style guides, photos of the homelab evolving back to about 2014, and the gnarly NFS/TFTP details I've kept light here.

The repo is open source: github.com/andrewiankidd/project-iluvatar.

This is the home server I should have built five years ago instead of trusting one chassis with everything. The PSU that killed Gideon 2 was, in hindsight, the most useful component I've ever owned. It forced me to learn how to do this properly.

Some strangers on the internet seemed to like it too:

"What a beautiful thing to look at, absolutely love it!"

"This is some sort of Zen garden."

"One of the best looking setup I have seen"

Repos and docs

• project-iluvatar - the source repo with everything (bootstrap, manifests, docs source)
• Documentation site - architecture, guides, and a build-log of the journey
• BACKLOG.md - honest tracking of gaps, debt and "would be nice" items
• andrewkidd/nfs-server - Alpine-based NFS container, ARM + x86, used by the netboot stack
• andrewkidd/tftp-server - Alpine-based TFTP container, same story
• r/homelab discussion - more photos of the rack and home office setup, and discussion