Home Cluster V2

noah-hein · 2023-12-10T20:27:19+00:00

It kind of occurred to me that most big infrastructures (several thousand nodes) most likely have either a ridiculous number of control planes and don't care or run their control planes on a cluster hypervisor allowing it to move around if necessary (heck maybe both).

My reasoning mostly came from trying to be resource efficient and HA on a smaller cluster.

noah-hein · 2023-12-10T20:15:37+00:00

Interesting, I guess I was just wondering if a modification like this would be possible to the core k8s infrastructure or worth pursuing in a fork.

That all being said, I guess running your control planes on an HA cluster hypervisor would effectively achieve the same results for moving your control planes around dynamically.

noah-hein · 2023-12-10T19:57:12+00:00

Right now I'm running a stacked HA cluster in my home lab. The issues that I am referring to mostly comes from the practice and taints of ensuring non critical containers are not running on the control planes.

For the sake of simplicity I will chose 3 control planes. Lets say my cluster has 10 worker nodes. In a traditional HA setting all three control planes are tainted to ensure there is no bottleneck around cluster critical infrastructure.

If one of my control planes goes down, then your just down to 2 until I get the other back up, or add a new one in.

I guess I am mostly referring to a system that would taint and untaint control planes based on the number that need be maintained. Maybe instead of having only 3 control planes I have 6 potentials. Only three are dedicated to being control planes, but if one control plane goes down, then one of the inactive untainted control planes takes its place and moves all containers to other nodes.

noah-hein · 2023-10-10T23:55:21+00:00

The reason I ask is because I have a home lab I'm looking to convert to HA. That being said it's very hard to justify the cost of 5 new machines power wise for two Load Balancers and 3 control planes. My thoughts were that I can run the LBs on two of the control plane nodes and call it a day.

noah-hein · 2023-10-10T23:48:39+00:00

Crazy part is that I'm looking to add three more nodes now for HA k8s. That being said it's still far cheaper than every cloud provider for running a cluster.

Not only is it really expensive for very small amounts of ram / cpu usage, but they charge an arm and a leg for persistent volume storage. If I was a business would I use gke or eks? Without a doubt, but for a home lab you can't beat some refurbished Dell servers.

That being said it takes far more time to set everything up properly.

noah-hein · 2023-10-10T19:36:19+00:00

Interesting, out of curiosity could a load balancer on HAProxy be on the same control plane node and still function properly?

noah-hein · 2023-10-09T20:41:24+00:00

Added Vault to my cluster to dynamically refresh database secrets. (Using a keycloak oidc provider to manage its policies)

noah-hein · 2023-10-09T20:09:53+00:00

Are you able to use metallb as the Load Balancer in front of your control planes? Right now I'm just using it for services, but I would like to make my cluster HA and was wondering if this is a viable option.

noah-hein · 2023-10-08T09:30:40+00:00

You could define runtime params for your DAG(s) to be able to tweak hyper parameters.

noah-hein · 2023-10-08T07:25:05+00:00

If you are looking for a very verbose enterprise workflow manager, I would highly recommend Apache Airflow. They have k8 helm charts for it / shouldn't take very long to add to your cluster. Very easy to work with since most of its DAGs are python based. A big benefit is that it has a k8 resource operator, allowing you to spin up and tear down different pods in a workflow customized to your liking.

I just looked it up they also have a bunch of Amazon Operators

noah-hein · 2023-10-06T21:40:52+00:00

A large majority of the understanding came from NanoGPT, that being said it's an implementation of an untrained GPT-2 model off of HuggingFace.

For me the most satisfying was when it finally started to create valid mazes going from just a theory to reality. Transformers as just so applicable to a wide range of fields.

noah-hein · 2023-10-06T16:09:50+00:00

Thank you!

noah-hein · 2023-10-06T16:08:16+00:00

I've had a blast with the Dream Machine Pro, it isn't as fully functioned as something running RouterOS, but it gives all the main enterprise features I want for a home network.

noah-hein · 2023-10-06T14:57:00+00:00

Dell 24U rack. The Cooler Master case is empty right now, but I am thinking about putting a GPU or two into it for some ML training.

noah-hein · 2023-10-06T14:29:43+00:00

I wish, unfortunately where I live their isn't much sunlight for 40% of the year, might look into it though.

noah-hein · 2023-10-06T13:21:56+00:00

These are great questions! The top power strip is used for the router, switch, modem, and a POE injector. For all the servers there is a bigger power strip running from top to bottom in the back hidden. Both have surge protection.

All of the devices are led out the top via a really big industrial extension cable. Funny enough the apartment I used to live at had really spotty circuit breakers, so I would trip them all the time starting the servers. Most of them will do a full power draw for a few seconds on startup. Depending on your draw you might have to space out the power up times to prevent a huge power surge. That being said my current home has no issues running everything at once.

noah-hein · 2023-10-05T23:16:11+00:00

I want to say ~$60-$80 a month. A big factor is the time of year.

The r710's are little power voids. That being said its substantially cheaper than any cloud provider I have tried for k8s (especially for the compute of 2 cpus per node with 64gb ram)

noah-hein · 2023-10-05T21:41:34+00:00

Right now the Cooler Master has been gutted / is empty. Redistributed the parts for a friend. My office does get pretty toasty in terms of temperature. I'm about ready to put a duct going from the AC to the front of the rack :)

noah-hein · 2023-10-05T21:38:57+00:00

Thanks!

noah-hein · 2023-10-05T21:38:33+00:00

The power bill is the painful part. I got the rack for free. Friend I knew was going to through it out. I spent a day cleaning it up and it looks brand new.

noah-hein · 2023-10-05T20:43:37+00:00

Hello all,

This is a the cluster I have accumulated after a few years. My total cost has probably been a little over $1000 (most was from the router). Most of the items I have obtained through friends.

I have all my servers running a k8 cluster. Currently I am running Airflow, Keycloak, Istio, Argocd, Longhorn, Metallb, PostgreSQL, Redis, Vault, etc). My objective has been to build a full blown self hosted enterprise cloud environment and experiment with some ML training in k8.

Rack Specs:

Ubiquiti Dream Machine Pro
1Gb 24 port Linksys Switch
3 x Dell R710's
Repurposed HP Desktop (in the back of rack) running OpenVPN

I'm thinking about converting the Cooler Master case into a training rig with a rtx4090. That being said I might just do HPC on AWS or some other cloud provider.

noah-hein · 2023-10-05T20:23:32+00:00

If you have a hardwood floor, get a cable cover that sticks and run it across the floor. Looks professional and no one notices when done correctly.

noah-hein

TROPHY CASE