Before the Freeze, After the Thaw: Lambda MicroVM Snapshot Design Patterns
Lambda MicroVMs start fast because they resume from a snapshot, but anything created at build time is shared. Learn what belongs before the freeze, what belongs after the thaw, and the patterns that keep tenants apart.
Lambda MicroVMs start fast because they resume from a snapshot. That's the feature, and it comes with one design rule worth knowing before you ship.
Picture this. You've built a per-tenant sandbox on Lambda MicroVMs. The first launch works and the tenth launch works. Startup is so fast you stop thinking about it. Then someone in a review asks a quiet question: why do two different tenants' sessions have the same session token?
Nothing crashed. Nothing paged. The app was doing exactly what you wrote. The catch is that the pattern you've relied on for years, "my app starts fresh every time it launches," quietly stopped being true.
To be clear, the platform keeps your tenants isolated: each MicroVM is its own VM, with its own memory and kernel. What got shared was a copy of a value your app created before the snapshot.
This piece is about that assumption, and about turning it into an advantage.
A restore is not a boot
When you create a MicroVM image, Lambda runs your Dockerfile, starts your app, and captures a Firecracker snapshot of the memory state of every running process, the disk, and any initialized network connections and file descriptors. Launching a MicroVM resumes that snapshot, so your process is already running when the first request arrives. Anything your app generated during the build, like unique IDs or secrets, is shared by every MicroVM restored from that snapshot.
That gives you two buckets for everything your app does at startup:
- Frozen: identical in every MicroVM. Loaded dependencies, warmed caches, parsed config, lookup tables. This is where the speed comes from, so do as much of it here as you can.
- Fresh: must be different per MicroVM. IDs, tokens, seeds, tenant identity, anything secret.
The whole skill is sorting your startup work into the right bucket. Most trouble starts with a line of code that looks like it belongs in the first bucket and secretly belongs in the second.
The six-line mistake
Here's the line that caused your quiet question in the review:
1
2
3
4
import random, time, uuid
INSTANCE_ID = str(uuid.uuid4()) # runs once, at image build
RNG = random.Random(int(time.time())) # seeded once, at image build
It looks harmless. It runs at import time, which means it runs during the image build, which means it ends up in the snapshot. Every MicroVM that resumes from that snapshot wakes up already holding the same
INSTANCE_ID and the same random number generator state.Now you try to reproduce the problem. You launch two MicroVMs from the image, compare their IDs, and they're different. Ticket closed?
Not so fast. In a test of six MicroVMs launched from one image, the build-time IDs came back like this:
1
2
VM 1: 4be6f28a VM 2: 4be6f28a VM 3: b8c64626
VM 4: b8c64626 VM 5: 4be6f28a VM 6: 4be6f28a
Four MicroVMs shared one ID and two shared another. Their kernel
boot_id values matched in the same pattern, so don't use boot_id as an identity either. A quick two-MicroVM check can easily land on different groups and look fine, which is exactly how this mistake survives code review.The "random" numbers matched too. Both builds happened to start within the same wall-clock second, so the clock-based seed was identical everywhere. A clock-seeded generator created at build time is wrong twice over.
One image, more than one snapshot
Remember the first time you cloned a VM from a template and found out every clone had the same SSH host keys? Same machine ID, same hostname, same everything the template had the day you sealed it. Nobody did anything wrong. The template just did what templates do.
A MicroVM snapshot is that template. And here's the twist that makes this mistake so slippery: sometimes there's more than one.
Back to your review. The quiet question was why two tenants shared a session token, and your first move was the obvious one: reproduce it. You launched two MicroVMs, compared their build-time IDs, saw two different values, and decided the report must be wrong. Green tick.
That's why the test lied to you. Launch six MicroVMs from the same image and you get the table above: four share one ID and two share another. Your two-MicroVM test simply landed in different groups.
So where do the groups come from? One image version can have more than one build, and you can list them:
1
2
3
4
aws lambda-microvms list-microvm-image-builds \
--image-identifier arn:aws:lambda:us-east-2:123456789012:microvm-image:my-image \
--image-version 1.0 \
--query 'items[].buildId' --output text
For this image it returned two build IDs. Each build records its own chipset generation and its own snapshot sizes, and asking for the details of both looks like this:
1
2
build 1: GRAVITON generation 3 memory snapshot 595,394,560 bytes
build 2: GRAVITON generation 4 memory snapshot 599,609,344 bytes
The build log tells the same story. Your app booted twice during the build,
/ready was called twice, and the two IDs in the table above match those two boots exactly. Think of it as two takes of the same photo: nearly identical, but each take has its own moment frozen in it.What's certain is that one image version produced two snapshot families and MicroVMs landed in both. What isn't visible from the outside is which MicroVM gets which family, or why. Two Graviton generations is the obvious suspect, but there's no AWS documentation that says so, so treat it as an observed pattern and not a rule.
The practical rule survives either way: treat every build-time value as shared by some of your MicroVMs, and never let a two-MicroVM smoke test convince you something is unique.
The fix: let the front desk hand out the keys
Think of a hotel. Every room is built from the same blueprint, and that's fine, because nobody expects rooms to have different layouts. What can't be shared is the key card. The front desk codes it when the guest checks in.
The
/run hook is your front desk. The snapshot is the blueprint: identical rooms, built once, ready fast. /run is the check-in, the moment this particular MicroVM learns who it is and gets its own keys. Lambda calls it with a JSON body containing the MicroVM's real ID plus whatever payload you passed at launch, and holds back external traffic until the hook returns HTTP 200. No guest gets into the room before the key works.Here's the check-in desk in code:
1
2
3
4
5
6
7
8
def handle_run(raw_body):
"""Create all per-MicroVM state. Called from the /run hook."""
body = json.loads(raw_body or b"{}")
payload = body.get("runHookPayload") or "{}"
FRESH["microvm_id"] = body.get("microvmId", "unknown")
FRESH["tenant"] = json.loads(payload).get("tenant", "unassigned")
FRESH["session_token"] = secrets.token_hex(16) # CSPRNG, drawn after the snapshot
RUN_DONE.set()
Four things happen: the MicroVM reads who it is (its ID), reads what the platform told it (the tenant), mints its own token after the snapshot so nobody else can hold a copy, and raises a flag that says "open for guests."
The hook server itself is just a tiny web server answering the platform's calls. Only
/run does real work here. The other three are placeholders for your own logic:1
2
3
4
5
6
7
8
9
10
if route == HOOKS + "/ready":
self._reply(200) # init is done: take the snapshot
elif route == HOOKS + "/run":
try:
handle_run(raw)
self._reply(200)
except Exception:
self._reply(500) # a failing /run can send the MicroVM to TERMINATING
elif route in (HOOKS + "/suspend", HOOKS + "/resume", HOOKS + "/terminate"):
self._reply(200) # flush on suspend, refresh on resume, clean up on terminate
Flush your buffers before a suspend, refresh credentials after a resume, tidy up on terminate. Note the failure path too: if check-in fails, you answer 500, and the MicroVM never opens for guests.
Launch two MicroVMs, one for Acme and one for Globex, and each answers with its own ID, its own tenant and its own token fingerprint. The blueprint stays clean because nothing unique ever touched it.
Worth pausing on: the hook port is separate from your app port. AWS's sample binds the hooks to0.0.0.0on their own port and exposes both ports in the Dockerfile, because Lambda calls hooks over the network namespace. A listener on localhost only will never be reached.
Wiring it up, from Monday morning
Picture the scene. A new tenant is onboarding, your platform team has until lunch, and you need a MicroVM image that behaves. Three steps get you there.
Step 1: the recipe. The Dockerfile uses the Lambda-provided base image, which is the recommended one if you use OpenSSL, since it includes a snapshot-compatible build.
1
2
3
4
5
6
FROM public.ecr.aws/lambda/microvms:al2023-minimal
RUN dnf install -y python3 && dnf clean all
WORKDIR /app
COPY app.py .
EXPOSE 8080 9000
CMD ["python3", "app.py"]
Port 8080 is for your tenants' traffic and 9000 is for the platform's hook calls.
Step 2: tell Lambda which calls your app answers. You enable the hooks when you create the image and declare the hook port in the same call:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
aws lambda-microvms create-microvm-image \
--name snapshot-demo \
--base-image-arn arn:aws:lambda:us-east-2:aws:microvm-image:al2023-1 \
--build-role-arn arn:aws:iam::123456789012:role/MicrovmBuildRole \
--code-artifact uri=s3://my-bucket/snapshot-demo.zip \
--hooks '{
"port": 9000,
"microvmImageHooks": { "ready": "ENABLED", "readyTimeoutInSeconds": 60 },
"microvmHooks": {
"run": "ENABLED", "runTimeoutInSeconds": 10,
"suspend": "ENABLED", "suspendTimeoutInSeconds": 10,
"resume": "ENABLED", "resumeTimeoutInSeconds": 10,
"terminate": "ENABLED", "terminateTimeoutInSeconds": 10
}
}'
Expect the build to take about two minutes, which is enough time for a coffee.
Step 3: one MicroVM per tenant. The tenant's name rides along in the payload:
1
2
3
4
5
6
7
8
aws lambda-microvms run-microvm \
--image-identifier arn:aws:lambda:us-east-2:123456789012:microvm-image:snapshot-demo \
--ingress-network-connectors arn:aws:lambda:us-east-2:aws:network-connector:aws-network-connector:ALL_INGRESS \
--egress-network-connectors arn:aws:lambda:us-east-2:aws:network-connector:aws-network-connector:INTERNET_EGRESS \
--execution-role-arn arn:aws:iam::123456789012:role/MicrovmExecutionRole \
--idle-policy '{"autoResumeEnabled":true,"maxIdleDurationSeconds":900,"suspendedDurationSeconds":300}' \
--run-hook-payload '{"tenant":"acme"}' \
--maximum-duration-in-seconds 900
Two details will save you a lost afternoon:
- Use the full image ARN for
--image-identifier. A bare image name is rejected byget-microvm-image, even though one example in the docs uses a name. - Build logs live in
/aws/lambda-microvms/<image-name>. That's where to look when a build misbehaves.
What survives suspend and resume
Think about a laptop lid. You close it at lunch with a dozen tabs and a half-written email, and you open it an hour later and everything is exactly where you left it. You didn't save anything. Nothing "restarted."
A MicroVM behaves the same way. Suspend one, resume it, and ask for its identity again: the token it minted at check-in is unchanged, because its memory came back exactly as it was. Suspend it again, send a request without resuming it, and it wakes itself up, as long as the idle policy has
autoResumeEnabled set. Your tenant just opens the laptop and carries on. CloudWatch shows /suspend and /resume firing each time.That extends the rule neatly: whatever you create at check-in lives as long as the MicroVM does, including every nap it takes in between.
Where does configuration go?
Say Acme asks for their own API key to be available inside their sandbox. Where does it go? You have three places, and picking the wrong one is how secrets end up in the wrong hands.
- The printed label on the box. These are image environment variables, set at build time. Every MicroVM from that image gets the same label, because it's printed before the box ships. That makes them fine for static, non-sensitive settings like a log level.
- The sticky note handed over at delivery. This is
runHookPayload: a note attached to this one MicroVM at launch, delivered in the/runbody. It's how the tenant name reached the check-in desk in the example above. - The safe. Runtime hooks execute under the execution role, so your
/runhook can open a safe (Secrets Manager or Parameter Store) with its own IAM permissions at runtime. That's where real secrets belong.
The tempting mistake is to put Acme's key in the Dockerfile as an environment variable. It feels tidy, and it quietly hands Acme's key to every other tenant's MicroVM, because it's printed on every label and frozen into every snapshot.
The right answer takes one extra step: the sticky note says "Acme's key is stored at this path," and the check-in desk opens the safe and fetches it. Put the reference in the payload and keep the secret behind IAM.
Two things to expect after the thaw
Randomness. Think of a card dealer who shuffles once before closing the casino and never again: every table opens with the same deck order. A fresh
uuid4() drawn on every request came back unique across all six MicroVMs in testing, so entropy read after the restore is healthy. AWS's guidance for Python is to use secrets or random.SystemRandom, and to avoid a default-seeded random.random(). Other languages have equivalents. Reading fresh entropy per call is the safe habit.Connections. Waking from a snapshot is like picking up a phone that went to sleep mid-call: the line is dead and your app has to redial. Outbound non-local connections are closed on run and resume, and most AWS SDKs reconnect transparently. For other clients, set timeouts and retries so they re-establish on their own. This one comes from AWS's guidance rather than from testing.
Keep the freeze small
Packing for a trip is a good model here. Everything in the suitcase has to come along every time you travel, whether you use it or not. Memory is restored eagerly while disk content is demand-paged, so a larger memory snapshot directly lengthens a restore, and AWS's rule of thumb is roughly one second per 500 MB.
Even a trivial standard-library Python server produced a memory snapshot of roughly 570 MiB in testing, plus about 308 MiB of installed code and a small disk snapshot. That's mostly the OS and runtime, so it's the baseline your app builds on. To see the numbers for your own image:
1
2
3
4
aws lambda-microvms get-microvm-image-build \
--image-identifier arn:aws:lambda:us-east-2:123456789012:microvm-image:my-image \
--image-version 1.0 \
--build-id "$BUILD_ID"
Look at
snapshotBuild.memorySnapshotSizeInBytes first. This is the trade-off in the design: warm what's worth warming, and don't preload every model variant "just in case."Design patterns: where freeze and thaw need extra care
Most snapshot surprises hide in the same few places. If your workload touches any of these, apply the rule on purpose: shared before the freeze, unique after the thaw.
- Tenant identity (session tokens, instance IDs, tenant mapping): create them in
/run, never at import time. Test with more than two MicroVMs, because one image version can hold several snapshot lineages. - Secrets and API keys: keep them out of the Dockerfile and image environment variables, since both are frozen into every snapshot. Pass a reference in
runHookPayload, fetch the secret from Secrets Manager or Parameter Store in/run, and re-fetch in/resumefor long-lived sessions. - Randomness and time: draw from
secretsorrandom.SystemRandomat the moment you need a value, and never seed from the clock at build. Timestamps captured at init are frozen too, so every MicroVM in a lineage believes it started at build time. Set "started at" values in/run. - Connection pools and long-lived clients (databases, queues, websockets): outbound connections are closed on run and resume. Set timeouts and retries, and rebuild pools in
/resume. This one comes from AWS's guidance and wasn't tested here. - Warm caches and models: this is the best use of the frozen bucket, but a bigger memory snapshot means a slower restore. Load what every tenant needs and load the rest on demand. Snapshots aren't refreshed by the service, so stale cached data or dependencies only change when you build a new image version.
- Suspended sessions: state created in
/runsurvives suspend and resume, so flush buffers in/suspendand re-check anything time-sensitive in/resume.
Final thoughts
If you remember one thing, make it this: shared state belongs before the freeze, unique state belongs after the thaw. Sort your startup work into those two buckets and the rest of Lambda MicroVMs gets easier to reason about, including suspend and resume, which follow the same logic.
The snippets in this post were run on Lambda MicroVMs in us-east-2 in October 2026. The connection behavior and the per-language randomness guidance come from AWS's documentation instead. This is a young service, so confirm behavior in your own account before relying on it.
Disclaimer: Please note that AWS is constantly evolving, and new features may be available since the release of this blog post. It's recommended to review the latest documentation to determine the most suitable solutions for your specific needs. This blog is a reference guide only. Additionally, ensure that the proposed solutions comply with your organization's security and compliance requirements, as some services may be relatively new and may not be fully compliant with all industry standards.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article