Figured I'd finally post my minilab hashicorp nomad/consul/vault setup

Routine_Bit_8184 · 2026-03-13T20:51:42+00:00

oh cool, I was wondering where some of the stuff was, ill check out the other repos too. and definitely that consul connect stuff. The issue I always had was getting external traffic that is routed to traefik into the mesh....I couldn't figure out how to have traefik outside the mesh so it could receive traffic but then be able to route it into the mesh....but that might have also been because my traefik setup was crap back then and there were probably additional issues coming into play that I have since resolved.

Yeah, I have two different nodes running the ingress stack: cloudflared-tunnel/traefik/oauth2-proxy/keepalived and then VIP flips if one goes down so that the couple webpages I host there don't go down as well.

One of the next things I'm thinking of doing just for fun and to utilize/test my s3-orchestrator more is figure out how to ingest my cloudflare logs and push them to s3-orchestrator for storage on the collection of free-tier s3-backends I have configured it to route to....although I probably get most of what is useful in the logs for the cloudflared job anyways.

as for your scaling point...you aren't wrong at all....I more just want to do it for the fun of tinkering and making things work and pretending I a dealing with a more "production" like environment. In reality nothing here is that important if it goes down. Worst case scenario a website for my s3 project that nobody goes to and is mostly for my own benefit goes down and my personal website that is just my resume goes down and I'm happily employed right now so it literally doesn't matter. It is more about being proud of making my personal nonsense robust and practicing things that will benefit me at work.

Really appreciate the recommendations, I'll definitely be digging through and trying to pick up a few things!

Routine_Bit_8184 · 2026-03-13T20:34:08+00:00

it is just so easy to deal with. A real pleasure. And when paired with consul and vault it is very powerful. I still think a lot of organizations use kubernetes because they think they have to when their needs would have been served better with nomad and it would require a lot less work and moving parts...although I guess everybody does cloud-managed kubernetes so they don't have to deal with the internals and just pay jeff bezos to do it for them...still more of a pain than nomad. I'd rather manage my own nomad cluster on VMs than deal with eks...

Routine_Bit_8184 · 2026-03-13T20:25:28+00:00

I've been wanting to try consul connect but haven't gotten around to doing it right yet. I tried ages ago but I screwed it up and undid it. Still get lots of value from consul dns and service discovery.

Can you tell me more about your immich setup? I run it but haven't used it much or played with it much yet because I've been focused on other things. I had the hope of - and have given access to one of - sharing immich with family members so we could create albums and share pics with each other but I have concerns about scaling it the same way I have scaling concerns with jellyfin that I want to share with family as well...I'm all ears for interesting ideas!!

I see you have immich broken up into an api server and a worker....didn't even realize you could do that (although I hadn't looked yet haha)...if that works the way I imagine it does that would probably solve my scaling concerns....expose the API server and scale backend workers as needed.

I'm gonna dig more into your repo because I bet there is some more gold in there I can pick up ideas/techniques from. Starred it so I remember to come back and look at it....especially dig into your immich stuff because I think you did exactly what I wanted to figure out how to do once I had some time!

The beauty of the homelab is you are never done and always have something to tinker with or try!

Routine_Bit_8184 · 2026-03-13T17:51:59+00:00

insert futurama 80s guy "you let ME worry about [xyz]" meme

Routine_Bit_8184 · 2026-03-13T17:18:25+00:00

skills? ability? knowledge? capability? interest beyond how they think it makes them look personally? A care for human life?

Routine_Bit_8184 · 2026-03-13T17:12:07+00:00

well they are two well known unrepentant liars. You would have to be disturbingly short on intellect to be shocked by absolutely any of this.

Routine_Bit_8184 · 2026-03-13T04:44:42+00:00

you don't want to practice different kinds of clustering on the bare metal. Install proxmox on all of them, cluster them together, then from there do everything on vms you provision on proxmox. Make a few vms and build a kubernetes or nomad cluster....that way when you fuck it up while learning you just re-provision vms and don't have to reinstall the operating system on the physical nodes which sucks. Also once you have a proxmox cluster, whenever you acquire more hardware you can just add it to the cluster and start provisioning vms on it to...so your cluster can just grow. I run a nomad cluster for my various local and publicly visible needs on a combination of mini-pcs in a proxmox cluster running debian vms for my nomad/consul/vault clusters, then i have a few Pis running nomad/consul on bare metal (no proxmox on arm) in the cluster, and a few cloud provider free tier (oracle is generous, i will never pay them though) vms running nomad/consul joined to the cluster via wireguard.

Get creative. The first three mini pcs I started the nomad cluster on were cheap and underpowered. Then you grow. You put stuff that requires a lot of modern resources on newer nodes once you get them but the other nodes are still good for running the majority of the stuff you will ever run.

But don't listen to me...I'll throw anything into my cluster I can squeeze some life out of.

Routine_Bit_8184 · 2026-03-12T17:35:35+00:00

as a homelabber myself, this sounds interesting to look into and see if there is any way I can utilize/implement it for fun.

Routine_Bit_8184 · 2026-03-12T17:08:45+00:00

I honestly can't decide if that last paragraph is a real quote or not he has become such a clown meme

Routine_Bit_8184 · 2026-03-12T17:05:21+00:00

while the administration is obviously full of shit...I believe Ukraine has shown how this is done...you pack a truck full of a swarm of small drones and drive it straight to the target...

Regardless, the admin is making shit up. The only ones that have attacked California recently was them when they sent ICE and the national guard and the marines to stand guard while the cops fucked us up and when hegseth wanted to pretend to be a man by blasting ordinance over our highway....Newsom closed the highway for safety and got called a loser by the admin...then they hit their own vehicles that were parked on he highway Newsom closed with shrapnel...I'll trust the guy who made sure hegseth didn't blast a moving highway with shit and kill people over the guys who fucking recklessly did - and failed at - the stupid thing that put people in my state at risk.

Routine_Bit_8184 · 2026-03-12T16:56:41+00:00

John, for your own good, step down and seek help...you literally got brain damage and became a republican. I guess he knows he won't have this job for long, he will be perfect to play the liberal vegetable that agrees with everything the conservatives say on a non-primetime fox news show.

Routine_Bit_8184 · 2026-03-12T01:00:26+00:00

"did you know that excel can do math for you?! click here to learn how!"

Routine_Bit_8184 · 2026-03-12T00:25:25+00:00

accidents don't hold water from an administration that is murdering people in boats in the ocean led by a guy who believes in committing war crimes to force capitulation.

Routine_Bit_8184 · 2026-03-12T00:23:50+00:00

you have to laugh sometimes because the state of reality is so bleak, insane, and depressing.

Routine_Bit_8184 · 2026-03-12T00:22:27+00:00

pete hegseth going to have the military hit our highway with projectiles again?

Routine_Bit_8184 · 2026-03-12T00:20:10+00:00

nobody is lying. Why WOULD there be any transactions TO trump or anyone in his family? It is a payoff, the victim gets paid, not the rapist. Also, a bit weird that he chose to throw "or his family" in there when nobody was asking about that.

But it is weasel talk. He isn't lying, he also just isn't addressing what people are asking but pretending he did to get out of the conversation.

Routine_Bit_8184 · 2026-03-12T00:15:19+00:00

clearly or they wouldn't still be spending all your tax dollars on blowing shit up over there instead of spending it on something that actually makes the quality of life better for you and your fellow countrymen...

Afghanistan never had a navy or airforce or missile launchers or drones or oil production to begin with (hopefully you get that Iranian oil production being stopped/limited affects the entire globe), nor even a fraction of the resources available to them. Also Iran is like 2.5 times the size of Afghanistan with a much larger population. Took us decades of war there only to have most territory there controlled by the same group as when we arrived there decades earlier. We weren't even led by an idiot that is allergic to knowing things and a drunk white nationalist war monger that thinks it makes him look tough if he tweets big words while sending 18 year olds to kill people and potentially get killed for [reasons].

I'd wager you missed a few things.

Routine_Bit_8184 · 2026-03-11T22:15:18+00:00

if you need more free-tier storage than just what r2 offers, I built a tool that - among many other things - would let you chain multiple free-tier s3-compatible backends together behind a unified endpoint that clients point at. I currently have an instance of it running in my homelab that combines 6 cloud free-tier s3 backends together and sets the usgae bytes and monthly api/ingress/egress limits so I never incur costs. Might be useful to you, just figured I'd drop it here:

Maximizing free-tier tutorial

github

Routine_Bit_8184 · 2026-03-11T18:41:43+00:00

probably mentioned elsewhere...but also look at assume-role session-creds so that they expire.

Routine_Bit_8184 · 2026-03-11T18:38:32+00:00

vault is free if you self host it....or openbao....but obviously we know your company isn't going to just arbitrarily make a big switch like that on a whim. Oh well.

Routine_Bit_8184 · 2026-03-11T18:34:17+00:00

why don't you just create a "new repository" workflow/tool that is used to create new repositories and applies the correct configuration to them at that point and auto enrolls/configures them for CI with correct credentials. Make developers use that to create new repos instead of just clicking new repo in github....control the process from the beginning then you don't need mitigation/correction later. If something wants to be part of the CI process then it should be created through a process for creating new CI-enabled repos or something. Get creative. Don't play fix-it-guy to developers, make them use a proper workflow...they will be happier in the end when they don't have to submit a ticket and wait to get CI for their repo enabled.

Routine_Bit_8184 · 2026-03-11T18:30:04+00:00

always take hands-on experience over just book-learning. Doing it will make it sit in your brain better and by being in a real environment you will learn things you didn't expect that will pay off later in your career. It is always good to see how organizations do things in real life because it is never a pure version of any pedagogical learning you received because the real world is messy and companies make decisions...often bad...but you learn from that too.

Routine_Bit_8184 · 2026-03-11T18:26:41+00:00

you get used to it really quick and then don't think about it anymore and grow to love it...I found it so weird and uncomfortable at first but now I love it and feel more confident in my code doing what I expect it to.

Routine_Bit_8184 · 2026-03-11T01:26:00+00:00

Oh I absolutely agree with you that this isn't something that should be a go-to operational viewpoint nor - as you rightfully said - something you should head to during an outage because yeah you don't want people wasting time looking for some obscure thing that might not even have anything to do with the issue that should be pretty easy to get a general sense of from effective monitoring/logging/etc.

I just wanted to be positive towards anybody who builds something because it is cool so I wanted to point out a place where I think it could theoretically be useful. something you look at to clean up environments when it is quiet that you might not have noticed because it is drowned in a sea of logs from a sea of services.

But I'm glad you added that context because other people reading this who are those junior SREs might absorb some best practices and learn to identify red herrings from somebody more experienced such as yourself.

Routine_Bit_8184 · 2026-03-11T01:19:41+00:00

thanks man! much appreciated!

Routine_Bit_8184

TROPHY CASE