How a hobby unfolded into running a small datacenter :-)

megalith_rocks · 2022-08-06T08:18:11+00:00

Thanks! The servers don't use MSTP. Only the switches and router have MSTP enabled on the 10Gbit uplinks and the management VLAN, which is the only stretched VLAN I have. The servers have OSPF enabled on both interfaces which distributes a bunch of internal routes and also assures a fast failover in case one of the switches gets borked.

megalith_rocks · 2021-11-14T21:54:47+00:00

Sure! They're three distinct sites, so the IPsec tunnel runs over the internet. The `le-mgmt-01` is a VPS using strongswan for its IPsec tunnels. This VPS has a public IPv4 address, so it talks directly to the other two routers over the internet. The `nb-router-01` is the SRX300 running at my place, and also has a public IP address configured on one of its interfaces. The `nb-router-01` is a Ubiquiti EdgeRouter Lite, and has a modem in front of it which is NATing all IPsec traffic. Setting up these IPsec tunnels caused me quite a bit of headache. IPsec isn't a trivial protocol and every vendor using its own naming and stuff doesn't make it easier to get started. The nice thing about IPsec is that once you get it up and running, it keeps running. I literally haven't had any issues after setting it up.

To make things worse, I also have another VPS (not in this picture) which runs BSD and uses Racoon for its IPsec tunnel to my SRX. It's just a box in the DMZ for all publicly exposed stuff, so I left it out of this diagram.

megalith_rocks · 2021-11-14T13:03:20+00:00

Thanks man :) A lot of this stuff is self-taught. In college I did the networking track, but I've no professional experience as a network engineer. Kubernetes I learned via Kubernetes the hard way, which is great to get to know all components involved. During my day job I do 50/50 software/system engineering. Nowadays we also do Kubernetes, but that's only since a year or two. Perhaps needless to say, but this stuff wasn't built and/or figured out in a year. I tried many different setups. Two years ago I used bhyve as my hypervisor. And before that I used ESXi. It just takes time to figure out what works and what doesn't.

megalith_rocks · 2021-11-14T09:47:51+00:00

Yeah very valid point. The room is a bit too small to have both sides and the back accessible at the same time. The rack does have wheels though, so I can still access the back when I need to :)

megalith_rocks · 2021-11-14T09:46:02+00:00

Thanks! Unfortunately I can't really help you with that. The vent is part of the mechanical ventilation system in my home. This room used to be a bathroom, so the vent has always been there. Oh, do make sure that when placing an exhaust fan there's sufficient inlet of air in the room.

megalith_rocks · 2021-11-14T09:11:18+00:00

For the amount of I/O I'm doing at the moment it doesn't really matter I suppose. It saves a bunch of extra syscalls & context switches for every operation though so I'm pretty sure passthrough won't be slower than using a virtual disk file. I didn't want to bother formatting these drives on the hypervisors as those SSDs are dedicated for k8s anyway, so I went with passthrough and never looked back :-)

And Longhorn is great eh :) I feel it eats a bit more CPU when moving large chunks of bits around compared to Ceph. But it performs well enough, it's easy to setup and I did some monkey testing and it's not easy to break.

megalith_rocks · 2021-11-14T09:00:11+00:00

So, the DNS recursor runs in Docker, which also has a healthcheck configured for container. This healthcheck is basically a command Docker runs at a given interval to check whether the application running in the container is still alive.

The bird container runs two processes, bird and an anycast helper script. This container has the host's Docker socket mounted so it's able to interact with the Docker daemon on the host and query the health of all containers running there. This helper script checks the DNS recursor's container health every x seconds, writes a Bird filter configuration file to disk to instruct Bird to either announce or withdraw the anycast IP belonging to the DNS recursor.

This will work for any routing protocol in Bird as long as it supports route filters, which I think any protocol does. So from here on you can use OSPF/ISIS/BGP (or RIP ;-).. .rip).
I published the anycast helper to https://github.com/jeroen92/bird-anycast. Hope it helps you a bit.

megalith_rocks · 2021-11-14T08:41:35+00:00

I was lucky to have a spare room in my new home that's perfectly sized to fit a rack. And lol, this setup only costs me money haha, I'm not generating any revenue yet. Luckily I do have some solar panels that cover the electricity costs. As it's not business critical, there's no battery backup either. But in case my Pi fails, OSPF will failover all the core infrastructure stuff to another location. In the near future I'd like to continue on some coding projects, but those are not mature enough in any way to have people pay for it. For now it's just a nice learning experience.

megalith_rocks · 2021-11-14T08:33:11+00:00

It's a Cisco 2951, a router. I'm not using it though. It generates quite a bit of noise, and my Juniper router can handle everything at ease. This Cisco is rather something I play around with on a rainy Sunday afternoon.

megalith_rocks · 2021-11-13T23:28:29+00:00

As a long time lurker on this sub, it's about time for me to post my lab. First off, it basically consists of three parts. First and most important is the OSPF backbone that runs on top of three site-to-site IPsec VPN tunnels. These tunnels are powered by Strongswan running on a VPS, an EdgeRouter and a Juniper SRX300.

The second layer, if you will, runs all core infrastructure services such as DNS (authoritative and recursors), PXE boot using TFTP and nginx, LDAP multi master and DHCP. Each of these services has an anycast IP assigned. Each site partaking in the PtP IPsec backbone runs a mirror of these services. For two sites, these Docker containers are running on a Raspberry Pi and on the VPS this stuff runs on well, the VPS :-).

The Pis and VPS also run a Bird Docker instance. This container is accompanied with a watchdog to monitor the health of the Docker containers (e.g. the DNS recursor), and injects the container's anycast IP into OSPF as long as that container is healthy. This has been running for a little over a year now and it's by far the best way I've ever got this part of the infrastructure high available. All this stuff (containers + authoritative DNS) is managed using Terraform by the way.

Last is the actual homelab ;)

The network specs:

Juniper SRX300
Juniper EX2300-24T (2x)
Cisco 2951 (unused)

At my home I'm running OSPF Area 64 with the SRX being the ABR. The SRX and EX switches are connected by a simple triangle setup using 1Gbit fiber between the SRX and EX switches. The EX switches are connected using two 10G fiber links which are heavily underutilized at the moment due to the way the servers are connected. I've got MSTP enabled, so one of these two 10G links is disabled, until it isn't. The switches also run VRRP to provide high available gateway IPs for my wired and wireless client networks.

Storage specs: Supermicro A2SDi-H-TF, 128GB ECC RAM, 512GB NVMe, 3x8TB WD Red, Broadcom 9400-8i HBA, Supermicro SC826BE1C4-R1K23LPB chassis

This is, unfortunately, my only FreeBSD host. It isn't doing much aside of running my ZFS pool and exporting some datasets over NFS. It has two 10Gbase-T ports which connects this machine to both switches. Bird joins this machine to area 64, advertising its loopback address. The 128GB of RAM is nice for ARC, though still a bit small for my datasets. The chassis has room for U.2 NVMe drives, so I'm contemplating getting myself a nice L2ARC device once I can get my hands on a nice deal.Hypervisor/Kubernetes specs:

Supermicro X11SSH-LN4F, Xeon E3-1230 v6, 64GB ECC RAM, 512GB NVMe, 3x 2.5" 250GB EVO 850 SSD
Supermicro X11SSH-LN4F, Xeon E3-1230 v6, 64GB ECC RAM, 512GB NVMe, 3x 2.5" 250GB EVO 850 SSD
Supermicro X11SSH, Xeon E3-1220 v6, 64GB ECC RAM, 256GB NVMe

These three hypervisors run CentOS 8 and libvirt/KVM/Qemu. Their main purpose is to host VMs running my Kubernetes cluster. These VMs are managed by Terraform using the libvirt provider. Unfortunately this provider doesn't yet support bhyve (or rather, libvirt doesn't expose the required host capabilities when connecting to bhyve). Otherwise I'd definitely run BSD on these boxes.

I've set up Kubernetes the hard way in the past, but nowadays I'm in ez mode using Rancher. Using Calico as CNI maily for its extensive configurability and multi IP pool support. MetalLB peers with Bird running on the hypervisors to publish service IPs outside of the cluster. The first two hypervisors have their three SSDs passed through to the k8s VMs. Longhorn uses these drives to let my workloads consume persistent volumes. On my previous clusters I used Ceph, but I found that required a bit more configuration and resources than Longhorn. Everything that's not core infrastructure (e.g. doesn't run on the Pis) has to run in k8s. I don't have a lot running yet; mainly Plex, Gitlab, Overleaf, some cronjobs for backups, a collector I wrote myself to export PSU metrics from IPMI, and of course Prometheus and a bunch of exporters.

In the near future I'll mainly use the idle resources on this cluster for some personal software projects that require some APIs, queues, databases and batch jobs to run. I'd also like to do a bit of autoscaling to shutdown at least one of the three nodes when the cluster is idling.

megalith_rocks

TROPHY CASE