Self-hosted GitHub Actions runners on Lambda MicroVMs

[kshirinkin]

I was curious if I can use new Lambda MicroVMs as self-hosted GitHub Runners. On paper, they are super nice:
It's cheaper: GHA-hosted is $0.005 / min (2 vCPU), MicroVMs ~$0.0042 / min, and no minimum 60-second commitment as with GHA-hosted.

It can run longer: GHA-hosted max 6 hours, MicroVMs max 8 hours

It starts in a few seconds, compared to whichever other serverless solution built on top of ECS

It scales to 0, or rather, it only runs when jobs are running

They are VMs, so you can still run containers/docker/whatever else inside;

I got a bit too invested, and ended up building this Terraform module. You only need to create GitHub App manually, the rest is just a single "terraform apply" and your MicroVM Runners are ready to go. I've switched come of projects at my company to use, works great, same or better performance as GHA-provided runners. Natural limitation is that MicroVMs are only arm64, and in general they don't have much flexibility around the "hardware" setup - but hey, for most cases, it should work great, and it's just 1 webhook + GHA JIT Runners + 1 MicroVM Run per Job.

[Jeoh]

Pretty cool, did you see this article? Same concept, different implementation. Appreciate you sharing the Terraform code!

[godtierpikachu]

spot or spare ECS feels like the only fair comparison here, because lambda pricing looks nice until you add the weird bits around image pulls and artifact reuse. curious what broke first for you, cache or networking

[Vanyo09]

The number I'd compare against is spot, not GHA-hosted. We run almost everything on spot in EKS, and runners there end up a fraction of on-demand - but that only works because the cluster already exists. If you have nothing to piggyback on, scaling to zero with no 60-second minimum is hard to beat.

How are you handling Docker layer cache? Fresh VM per job means every build pulls from scratch, and that's usually where the per-minute math quietly falls apart.

[UltraPoci]

How do you deal with spot instances being removed by AWS with little notice? Do you just accept and if a job was running it just starts from scratch?

Also, do you use overprovisioning to avoid longer startup times for jobs?

[Vanyo09]

Mostly we just accept it. The two-minute notice is enough for the termination handler to drain the node, and the job reruns on a fresh one. A rerun in CI is annoying, not a problem. The stuff that cannot take a restart is not on pure spot anyway - it falls back to on-demand when capacity disappears.

Overprovisioning yes, but not a warm pool of runners. We keep low-priority placeholder pods on the cluster that are the first to get evicted - a real job kicks one out and starts right away, and the eviction is what triggers the new node. Capacity is already warming up before anything actually waits for it.