
Serverless GitLab Runners on AWS Lambda MicroVMs: killing our idle CI cluster
Our EKS runner cluster cost $270/month — 95% of it paying for idle. I rebuilt it on Lambda MicroVMs: every CI job gets its own Firecracker VM restored from a snapshot with Docker already warm, cutting job pickup from 2 minutes to ~4 seconds at ~$0.01 per job.
Series: Serverless CI on AWS Lambda MicroVMs (2 articles)
- 1Serverless GitLab Runners on AWS Lambda MicroVMs: killing our idle CI cluster This article
This is Part 1 of a 2-part series.- Part 1 (this post): replacing an idle EKS runner cluster with per-job Firecracker VMs — architecture, measured numbers, and the launch-week gotchas. - Part 2: MicroVM images are invisible to your vulnerability scanner — closing the CVE loop with Inspector, Step Functions, and AWS DevOps Agent. - Companion repo: everything in this series is reproducible fromgitlab.com/sawalanipawan/lambda-microvm-gitlab-runners— CDK stacks, runner image, pipelines, and a zero-to-green runbook.
TL;DR: Our GitLab runners lived on an EKS cluster costing ~$270/month — and 95% of that paid for nothing happening between pipelines. I rebuilt the whole thing on AWS Lambda MicroVMs (GA June 2026). Now every pending CI job gets its own fresh Firecracker VM, restored from a snapshot in which the Docker daemon is already running and common images are already pulled. The VM runs exactly one job and vanishes.
Results: job pickup 60–120 s → ~4 s, cost $270/mo fixed → ~$0.01 per job, and every job now gets its own kernel instead of a
privileged pod on a shared node.The whole thing in five steps
- GitLab marks a job pending and fires a webhook.
- A tiny receiver Lambda validates it and drops it on an SQS FIFO queue.
- A dispatcher Lambda mints a one-time GitLab runner via the API and calls
RunMicrovm.
- The MicroVM wakes from its snapshot (~5 s), with dockerd already warm, picks up exactly that one job, and runs it.
- The job ends → the VM terminates itself. A sweeper Lambda cleans up anything that slipped through. Nothing is left running.
That's the entire platform: three small Lambdas, one queue, one Dockerfile.
The itch
(Context: gitlab.com SaaS, ARM64 images for Fargate. Runner fleet: EKS Auto Mode + kubernetes executor.)
Three things bugged me about our EKS runner setup:
- The cost curve is upside down. Cost Explorer said ~$270/mo for the runner platform. The actual Spot compute doing CI work? About $14. The other 95% was an idle tax — control plane, manager node, NAT gateway, sitting there through nights and weekends.
-
privileged: true. Docker builds forced privileged job pods on shared nodes. Any job could root the node it ran on.
- The upgrade treadmill. Our infra repo carried three generations of runner stack (EKS 1.31 + Karpenter, 1.34 + newer Karpenter, then Auto Mode). None of that work ever delivered a feature.
First: what exactly is a Lambda MicroVM?
If you've only met Lambda as "functions", recalibrate.
Lambda MicroVMs (GA 22 June 2026) are full Firecracker micro-VMs with lifecycle control,
running on the same virtualization layer as regular Lambda. Three things make them different from a function:
- A real OS you control. Amazon Linux 2023, with full OS capabilities on request. That means you can run dockerd inside the VM, mount filesystems, create network namespaces.
- A lifetime measured in hours, not seconds. Up to 8 hours running or suspended; terminated on your API call.
- VM-level isolation. Each MicroVM is its own kernel. AWS's stated use cases: AI code sandboxes, dev environments, security scanning — and multi-tenant CI executors, which is exactly what a runner fleet is.
The lifecycle, end to end:
- You upload a
Dockerfilebundle to S3. No docker build on your side, no registry. Lambda builds the image itself, on a managed AL2023 base.
- Lambda boots it and waits for your
/readyhook. You return 200 only when everything you want "pre-baked" is initialized.
- Lambda snapshots the entire VM — memory and disk. This is the magic. The snapshot is not a container image; it's a running machine frozen mid-flight. Warm daemons, filled caches, open files: all captured.
-
RunMicrovmresumes a fresh copy of that snapshot in seconds. Each VM gets a private HTTPS endpoint and a/runhook that delivers a per-VM payload — your slot for per-job configuration.
- Networking is explicit. You attach "connectors": internet or VPC egress, and ingress that can be disabled entirely (
NO_INGRESS) or reduced to an IAM-gated shell channel.
- Terminate when done — all charges stop. (Suspend/resume exists too: ~1 s each way, storage-only billing while suspended.)
Sizing follows the snapshot. Memory is fixed per image (the snapshot is a RAM image), vCPU is proportional (2 GB = 1 vCPU), and the VM automatically bursts to 4× baseline under load. Billing is per second while running.
Why that fits CI like a glove
The superpower: you initialize an environment once, snapshot it, and every VM launch resumes from that exact state in seconds.
The trick most writeups miss: what you snapshot doesn't have to be your app. It can be infrastructure state. My image's entrypoint:
- starts
dockerd
- pre-pulls
maven,node,docker:cli,alpine(all linux/arm64)
- answers the
/readyhook → Lambda takes the snapshot
So every CI job starts with a hot Docker daemon and a primed layer cache — the two slowest parts of containerized CI, already done, forever.
Architecture

Design choices worth defending:
- The runner runs inside the VM as
gitlab-runner run-single --max-builds 1. Every GitLab feature — caches, artifacts, services, timeouts, OIDCid_tokens— just works, because it's a completely normal runner that happens to live for one job. I didn't write a custom executor (that API is in maintenance mode), and GitLab's autoscaler plugins want SSH access that MicroVMs deliberately don't offer.
-
NO_INGRESS. A CI runner only needs outbound connections. The job VM cannot accept an inbound connection from anywhere. Try that with a node pool.
- One-time runners, minted per job. The dispatcher creates an ephemeral runner via
POST /user/runnersand delivers its token through the VM's/runpayload. No long-lived runner token exists anywhere — which also killed the plaintextglrt-token we'd been carrying in git.
- No human ever touches a credential. GitLab tokens live in Secrets Manager as managed external secrets: AWS itself rotates them against GitLab's token-rotate API every 14 days, and the bootstrap value is revoked minutes after creation. The Lambdas re-read the secret every 5 minutes, so rotation is invisible.
- Idempotent end to end. SQS FIFO dedupes on job id;
RunMicrovmtakes a per-jobclientToken. A redelivered webhook cannot double-launch a VM. A lost webhook is healed by the sweeper, which re-queues any job GitLab still reports as pending.
What I learned the hard way (save yourself a day)
- Your Dockerfile becomes a container inside the VM, layered on a managed AL2023 base. Start
FROM public.ecr.aws/lambda/microvms:al2023-minimal; running dockerd requiresadditionalOsCapabilities: ["ALL"].
- Memory is fixed at image build time. The snapshot is a RAM image, so
RunMicrovmhas no size parameter — different sizes mean different images. Also check your applied "Max allocated memory" quota: the default is 400 GB, but my new account was silently clamped to 8 GB, and the Service Quotas API refuses requests below the default. That's a support-case conversation.
- Outbound UDP is blocked, so DNS dies — unless you point
resolv.confat Lambda's stub resolver,169.254.169.253. Symptom:docker pulltimes out inside the VM.
- Job containers can't reach that resolver over the docker bridge either. Run them with
--docker-network-mode host. With one job per VM there's nothing to isolate anyway.
- Lifecycle hooks arrive on port 9000, and
/runmust answer within 60 s — spawn your workload and return immediately.
- Debug image builds outside CloudFormation. A failed build surfaces as
NotStabilizedand rollback deletes the evidence. Iterate with the CLI (aws lambda-microvms create-microvm-image; shorthand keys are camelCase) and read build logs at/aws/lambda-microvms/\<name\>— then codify in CDK. My own failure was dnf5 pedantry:install_weak_deps=Falseis an "invalid boolean"; use=0.
- Runtime logs are double opt-in. Enable them per
RunMicrovmcall and give the execution rolelogs:*permissions — miss either and you get silence.
- Grant
lambda:PassNetworkConnectorto whoever callsRunMicrovm— a PassRole-style guard that only errors at dispatch time.
- Bundle the AWS SDK into your Lambdas (
externalModules: []in CDK): the runtime's built-in SDK predates the service. Related: the IAM policy simulator doesn't know the new actions and returns misleadingimplicitDeny— for brand-new services, trust the actual error messages.
Three more, earned during a second rollout of this design on a real production codebase:
- With host networking, docker doesn't put the container's hostname in
/etc/hosts— soInetAddress.getLocalHost()fails. Symptom: Quartz dies with "Cannot run without an instance id" and takes the whole Spring test context with it (2,585 cascaded test errors from one lookup). Fix: a fixed hostname via--docker-hostname+--docker-extra-hosts.
-
run-singlecaps job logs at 4 MB by default — large test suites get truncated mid-run. Set--output-limit(KB) to whatever your current fleet uses.
- CloudFormation's
AWS::Lambda::MicrovmImagewantsKey/Valuepairs inEnvironmentVariables, notName/Value— you'll only find out on first real use, since an empty list validates fine.
The numbers
Measured on the demo (July 2026, eu-west-1, 2 GB/1 vCPU baseline VMs):
Metric | EKS runner cluster | MicroVM runners |
|---|---|---|
Fixed monthly cost | ~$256 | ~$0 |
Cost per average job | opaque (shared) | ~$0.01 |
Push → job running (cold) | 60–120 s | ~25 s total; 4 s queue time |
RunMicrovm → VM RUNNING | — | 189 ms API call, ~5 s to RUNNING |
docker build with warm layers | minutes (pull first) | 14.7 s |
Isolation | privileged pods, shared kernel | 1 job = 1 kernel |
Break-even honesty: per vCPU-hour, MicroVM compute costs ~5–7× x86 Spot. The win is structural — deleting the idle floor, not beating Spot on rate. If your CI volume is ~5× ours, a packed cluster wins on raw compute. Measure your own idle ratio first.
What's still rough
- ARM64/Graviton only. Fine for us (we build ARM64 images anyway); a hard stop if your toolchain is x86-native.
- The service is weeks old. Expect API/CLI sharp edges (see above).
- 8 h max VM lifetime. Fine for CI; a hard cap for anything else.
Steal this
Everything is CDK TypeScript + three small Lambdas + one Dockerfile, published as a public companion repo:
gitlab.com/sawalanipawan/lambda-microvm-gitlab-runners — parameterized
for any AWS account and gitlab.com namespace (GitLab Free works; the security scanning uses open-source Trivy, no Ultimate needed). The runbook takes you from zero to a green pipeline in ~30 minutes.Coming in Part 2
This architecture forces an awkward security question: your runner image is now built inside Lambda and never lands in a registry — so Inspector, ECR scanning, and every registry-based scanner you own is blind to it. Part 2 covers the scan-twin pattern that restores continuous CVE visibility, and
the Step Functions + AWS DevOps Agent workflow that turns findings into agent-analyzed GitLab issues, a ready-for-review remediation MR, and Inspector-verified closure — with exactly one human step in the loop: the merge. (Spoiler: it took a live image from 23 HIGH CVEs to 0, and caught an Amazon Linux 2023 quirk that silently turns
dnf upgrade into a no-op.)If you run GitLab runners on a cluster today, do one thing before your next platform sprint: open Cost Explorer and compute your own idle ratio. Mine was 95% — and that number, not any of the architecture above, is what made this migration inevitable.
Series: Serverless CI on AWS Lambda MicroVMs (2 articles)
- 1Serverless GitLab Runners on AWS Lambda MicroVMs: killing our idle CI cluster This article
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article