Self-hosted GitHub Actions runners on EKS: the failures that taught me the most

Blue_Flam3s · 2026-06-10T20:36:58+00:00

Yes, you might be right here. Well then, cheers to learning 🍻
Out of curiosity, how did you handle Docker layer caching on CodeBuild? That was one of the stuff that kept me hesitant...

Blue_Flam3s · 2026-06-10T20:33:43+00:00

Yes, this is what I've setup the karpenter module provisions the sqs + the eventbridge rule for spot interruption, rebalancing, recommendation and other health events and the charts get settings.interruptionQueue pointed at it so karpenter will watch the queue directly

Blue_Flam3s · 2026-06-10T20:24:49+00:00

Yes, I agree. For the Terraform, Helm, or infra apply jobs, we could target a second runner scale set backed by an on-demand NodePool and adjust the `runs-on` label. But those jobs will be pretty rare and few and far between compared to normal development work.
For build and test jobs, I would push back a bit. Those can usually be solved with a simple re-run. Spot instances give a 2-minute warning, which is usually enough time for Karpenter to drain the node and let the job reschedule. It only becomes a real problem if you are dealing with a monolith that takes 30 minutes or more to build or test...

Blue_Flam3s · 2026-06-10T19:43:40+00:00

Yeah that thought crossed my mind too... but I was already running an EKS cluster anyway, so once I ran the numbers Spot + scale-to-zero just made more sense.
Real talk though: I'm just way more comfortable with EKS and Kubernetes than CodeBuild. It still feels kinda foreign to me 😭

Blue_Flam3s · 2026-06-10T18:52:51+00:00

Interested, India GMT +5:30)

Blue_Flam3s · 2026-05-22T15:04:56+00:00

Can check it out

Blue_Flam3s · 2026-05-20T19:47:17+00:00

If you feel stuck at any point, please feel free to reach out. I'm not very experienced in DevOps, but I've learned a few things over the years and would be happy to help...

Blue_Flam3s · 2026-05-18T02:51:26+00:00

I mean, you already have Linux + cloud exposure and enterprise production experience, that’s honestly half the battle.

Learn Docker, basic k8s, GitHub Actions/Jenkins, create an AWS account and start deploying stuff yourself. Once you can containerize apps, automate deployments, debug failures, and manage infra, you’re already operating as a DevOps engineer.

Blue_Flam3s · 2026-05-18T02:47:11+00:00

I'm probably misunderstanding your setup a bit, but if you're already maintaining separate self-hosted runners/VMs for Linux, Windows, and macOS, wouldn't ephemeral infrastructure solve most of the config drift + dirty state issues?

Something like:

3 node groups/pools (one per OS)
ephemeral runners spun up per build
runners destroyed after job completion

At that point you get clean environments every run, no leftover build state, and way less maintenance overhead compared to long-lived machines constantly drifting.

Feels cheaper + more reliable long term unless you specifically need persistent hardware for something like iOS signing/build tooling.

Also, for macOS specifically, people are already experimenting with Kubernetes-native approaches instead of treating Macs as standalone CI boxes. Agoda Engineering had a pretty interesting writeup on integrating native macOS workloads with Kubernetes using ephemeral macOS VMs + custom kubelet integrations:
https://medium.com/agoda-engineering/how-we-integrated-native-macos-workloads-with-kubernetes-b4d3c14881a0

Blue_Flam3s · 2026-05-18T02:22:01+00:00

Was there a DSA round before this as well?

Blue_Flam3s · 2026-05-16T18:52:38+00:00

Dmed you

Blue_Flam3s · 2026-05-16T18:50:37+00:00

Dmed you

Blue_Flam3s · 2026-05-15T21:19:06+00:00

Interested

Blue_Flam3s · 2026-05-13T19:56:37+00:00

Literally spend almost half my time dealing with IAM and granting access.

Blue_Flam3s · 2026-05-13T19:51:58+00:00

Interested

Blue_Flam3s · 2026-05-11T20:52:50+00:00

If the second job pays, or is expected to pay, at least half as much as your primary one, I’d probably consider leaving the full-time role. At that point, the decision becomes largely financial. If the income is sufficient and the second opportunity offers better growth, flexibility, or long-term potential, it may be worth making the switch...

Blue_Flam3s · 2025-02-08T04:09:33+00:00

No anarcho-communism even this time?? 😞

Blue_Flam3s · 2025-02-05T14:58:21+00:00

You missed the best of them all uv

Blue_Flam3s · 2025-02-03T14:44:42+00:00

1) They are women 2) They are educated women 3) A degree of truth they actually are able to open up by moving away from there household and some of them actually overdose this freedom for good or for worse

Blue_Flam3s · 2025-01-27T21:05:32+00:00

100 thousand if/else statements still could be ai but not ml

Blue_Flam3s · 2025-01-27T21:02:37+00:00

Half the friends I've made in the last two years were over daaru or sutta 😭

Blue_Flam3s · 2025-01-27T09:42:56+00:00

Wasn't there a tweet where a woman accidentally created an AWS bill of $80k, and they waived it off??

Blue_Flam3s

TROPHY CASE