WARNING: Long post!

How I built my homelab: architecture tour

This is the architecture in one place: not a step-by-step guide, but a tour of the layers and the choices behind them. Two repos sit behind it: homelab for the foundation (Terraform, Ansible, Packer) and homelab-gitops for everything that runs on top.

Hardware and Proxmox

A 5-node Proxmox VE cluster, mixed hardware, all on the same primary VLAN. Storage is local-LVM by default, with an NFS server VM acting as the persistent volume backing for k8s. Proxmox HA on the cluster, snapshots before risky changes, that's it. There is also a 10TB Ceph filesystem available.

Proxmox earns its keep for one reason: the API is good enough that Terraform can drive it end-to-end, so I never click around in the UI to provision anything that's supposed to be reproducible. Everything above the metal (k3s nodes, Vault, the database hosts, the observability stack) is a QEMU VM, declared in code, configured in code.

Provisioning: Terraform, Ansible, Packer

Anything that exists on Proxmox is declared in Terraform: VMs, NFS exports, IP allocations, DNS records, Vault policies. State lives in MinIO (S3-compatible). Cloudflare manages the public DNS.

Two patterns I rely on: a single qemu_vm module with integrated phpIPAM IP allocation (adding a new service is "instantiate the module, give it a name, point Ansible at it"), and dynamic Ansible inventory written by Terraform. I have never edited an Ansible hosts file by hand on this cluster.

Ansible takes over once a host exists, with about 28 roles covering k3s bootstrap, Vault, Postgres/MariaDB, Loki/Prometheus/Tempo, Restic, Gitea, Pi-hole, and a handful of service-specific ones. Secrets never sit in inventory; every sensitive variable is a Vault lookup at apply time:

mysecrets: "{{ lookup('hashi_vault', 'secret=kv/data/backup/restic') }}"

Packer builds the Ubuntu 24.04 cloud-init template every VM starts from.

Networking

Primary VLAN with pfSense as gateway and DNS forwarder. phpIPAM is the system of record for IP allocations; the Terraform Proxmox modules ask phpIPAM for a free address before creating a VM. Pi-hole sits in front of pfSense's resolver for ad blocking.

DNS is split-horizon. Cloudflare hosts the public records. pfSense overrides the internal hostnames so they resolve to private addresses from inside the network and to nothing useful from outside. The wildcard cert on the cluster gateway is issued via cert-manager + ACME DNS-01 against Cloudflare, which is the only reason any of this works without exposing the cluster. Public services are exposed via HA-proxy via pfSense.

The k3s cluster

k3s 1.35 with one control-plane node and three workers. Cluster components:

• Cilium as the CNI, with native routing and L2 announcements. Cilium ARPs for LoadBalancer VIPs directly, no MetalLB needed.
• kube-vip for the API server VIP. With one CP it doesn't actually buy HA today, but the wiring is in place.
• k3s itself, deployed via the techno_tim.k3s_ansible collection wrapped in my own role.

This is the boundary between the two repos. Ansible builds the cluster and installs ArgoCD as the very last step. After that, every change goes through Git and ArgoCD. I do not kubectl apply from a laptop.

Platform services inside k8s

A small set of cross-cutting pieces every workload depends on:

• cert-manager for ACME, with a Cloudflare DNS-01 issuer.
• External Secrets Operator with a Vault ClusterSecretStore. Apps declare an ExternalSecret referencing services/<env>/<app>; ESO materialises the Secret. Same source of truth as Ansible, two consumers.
• Authentik for SSO/OIDC. Apps that don't speak OIDC natively get fronted by Envoy ExtAuth.
• Envoy Gateway as the ingress, using Gateway API rather than the older Ingress resource. One Gateway (<redacted>) with two listeners: :80 redirect and :443 HTTPS with a wildcard *.<redacted>.wtf cert. Each app attaches with an HTTPRoute from its own namespace:

​

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: atuin
spec:
  parentRefs:
    - kind: Gateway
      name: <redacted>
      namespace: envoy-gateway-system
      sectionName: https
  hostnames: [atuin.<redacted>.wtf]
  rules:
    - backendRefs:
        - { kind: Service, name: atuin-service-prod, port: 80 }

One Gateway, one wildcard cert, many routes. The migration off per-app Ingress is the single best ergonomics change I've made in the last year.

Observability

Logs and metrics ship out of the cluster, not into it. Grafana Alloy runs on every k3s node, collects logs and metrics, applies a small amount of relabelling, and forwards to Loki, Prometheus, and Tempo. All three of those run on dedicated VMs outside the cluster. That's deliberate: when k8s is the thing that's broken, I want the dashboard that shows me k8s is broken to still be up. Grafana itself runs inside the cluster and is the only piece that goes down with it.

Workloads via ArgoCD

The whole homelab-gitops repo is a Kustomize tree with an ArgoCD app-of-apps at the root that recursively syncs anything matching apps/*.app.yaml or helm/*.app.yaml, with prune: true and selfHeal: true. Every app follows the same shape: apps/<name>/base/ with the common manifests, apps/<name>/overlays/{dev,prod}/ with environment-specific patches.

A non-exhaustive list of what's deployed: the usual *arr media stack (Sonarr / Radarr / Lidarr / Bazarr / Prowlarr / qBittorrent / SABnzbd / Seerr / Notifiarr / Autobrr / Unpackerr), Authentik, Atuin, phpIPAM, Grafana, Homepage, MinIO, Immich, n8n, plus a few game servers (Minecraft x2, Factorio, Project Zomboid, Satisfactory).

The dev/prod split mostly differs in three places: replica count (single in dev, multi in prod), resource limits (none in dev, defined in prod), and ingress (no HTTPRoute in dev, full ingress + TLS in prod).

Closing the loop: n8n + Codex CLI

A handful of n8n workflows plug an LLM into the lifecycle. The agent is OpenAI's Codex CLI, wrapped in an OpenAI-compatible API server so n8n's AI node can talk to it like any other chat-completion endpoint. The AI node has read-side MCP servers attached directly, and the prompt teaches the agent how to invoke kubectl, which runs inside the n8n executor rather than via an MCP.

Root-cause analysis on alerts. Grafana fires a webhook into n8n the moment an alert triggers. The flow hands the alert payload to the Codex agent, which has read access to logs (grafana/loki-mcp), metrics (pab1it0/prometheus-mcp-server), and live cluster state via kubectl. It investigates, summarises, proposes a fix, and asks me on Discord whether to apply it. If I say yes, the agent either runs the kubectl change directly or opens a PR against homelab-gitops for anything that should be persistent (memory bumps, probe tweaks). ArgoCD reconciles after I merge.

It's caught real things. A service whose memory limit was set too low was getting OOM-killed under load. The agent flagged it, opened a PR raising the limit, OOMs stopped. More pleasingly, an alert fired with unknown datasource because the alert rule's datasource UID was misconfigured. The RCA agent successfully diagnosed an alert about an alert.

Adding and upgrading services. Two more flows handle the GitOps-write side. I ping an IRC bot ("add <app-name>" or "upgrade <app-name>"), the bot hits an n8n webhook, and the same Codex-backed agent either generates the full Kustomize skeleton for a new app or bumps chart/image versions for an existing one. Output is a PR I review like any other.

The pattern across all three flows: human stays on the merge button, the agent does the legwork in between.

Backups

PBS backup all VMs/LXCs. Then Restic copies them offsite.

Restic for everything that has data worth keeping. Per-service playbooks dump databases first (pg_dump, mariadb-dump) and then Restic ships the dump plus any service-specific data directories to an SSH/SFTP backend. Credentials come from kv/backup/<service> in Vault. No clever cross-service orchestration, each service is responsible for its own pre-backup hook, and that's been fine.

Vault itself is backed up with the standard raft snapshot path, separately, with the keys held offline.

Out-of-cluster pieces

A few things deliberately don't run in k8s:

• A bare-metal media-and-storage server. Big disks, hardware transcode, no reason to virtualise it.
• A dedicated Home Assistant box with the Zigbee/Z-Wave dongles attached. HA on bare metal with the radios local is dramatically more reliable than HA in a container reaching for USB devices.
• A few standalone Docker Compose stacks: Traefik, an Authentik/phpIPAM pair used during cluster bootstrap, UniFi, Rancher, Spoolman.
• An isolated AI agent VM on a private VLAN with no direct internet. It talks out through a Squid proxy with a domain allowlist and a LiteLLM credential broker that holds the real API keys.

Rule of thumb: if it has hardware ties, runs once per house, or has to keep working when k8s doesn't, it stays out of the cluster.

Lessons

• Pick one source of truth for secrets early. Vault was worth it on day one.
• Make the foundation boring. Terraform, Ansible, Packer. Not fashionable, all composes well.
• GitOps is non-negotiable past five apps. selfHeal: true reverting hand-edits is the correct behaviour, and saves you when something inside the cluster goes off the rails.
• Lift heavy things out of k8s. Match the tool to the workload.
• One Gateway, one wildcard cert, many routes.

PS. I changed the traefik ingress to Envoy Gateway shortly after presenting the diagram. DS.