
Run an iperf2 server on a Lambda MicroVM, with Proxylity UDP Gateway
A Lambda MicroVM is a server that sleeps when idle. Build an iperf2 server on one, reached through Proxylity UDP Gateway, and learn where state should live, why event time beats processing time, and how to prove a measurement tool correct.
About the author: Lee Harding is the founder of Proxylity, which makes the UDP Gateway service used in this tutorial. The example code is open source, and the techniques here (capturing a protocol before implementing it, choosing where state lives, event time versus processing time, and proving a measurement tool correct) apply well beyond it.
iperf2 is one of the classic tools for measuring network performance. You run
iperf -s -u on one machine, iperf -u -c <server> on another, and the client reports throughput, packet loss, and jitter.This tutorial builds the server half on a Lambda MicroVM, so you get an iperf2 endpoint that any stock client on the internet can test against, and that costs next to nothing while nobody is using it. By the end, an unmodified iperf2 client will run a normal test against your endpoint and print a normal result:
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams[ 1] 0.0000-10.0196 sec 1.25 MBytes 1.05 Mbits/sec 5.790 ms 0/895 (0%)And you'll prove the numbers are right, with bash scripts that inject known faults and check that the same faults come back out.
Why do this?
Is a Lambda MicroVM a server? Arguably, yes. It's a small Linux virtual machine running your long-lived process, with its own memory that survives between requests. Anyone who has run a server will recognize it. What's different is the lifecycle. A MicroVM can be launched when the first request arrives, suspends automatically when it's been idle (keeping its memory), and costs near zero while suspended. There's no machine to size, launch, or leave running "just in case". That's the sense in which this tutorial calls it serverless: not that the server is gone, but that you stop paying for it when it isn't working.
Why iperf2? It's a good stress test for that idea. It's UDP, it's stateful, and it's timing-sensitive. The server has to remember every datagram of a test to report loss and jitter at the end. If a MicroVM can do this correctly, it can host many other stateful UDP workloads: firmware delivery, AAA protocols, game servers. And the result is easy to check, because we can compare it with a stock iperf2 server.
Why does it matter where the state lives? A normal Lambda function forgets everything between calls, so stateful work has to push its state out to a cache or database. A MicroVM can just keep it in memory. Whether that's the right trade is the design question this tutorial keeps returning to, using iperf2 as a concrete example.
Why would I want my own iperf2 server? To measure the network path between your places (a laptop, an office, a branch, a device in the field) and AWS. The client runs on your machine, as always. What you're missing is an endpoint in the cloud to measure against.
Why a gateway?
You might be thinking: if the iperf2 server runs on a MicroVM, why do I need a UDP gateway at all? Why not just run
iperf -s in there?Because you can't. A MicroVM sits behind NAT: it can make outbound connections, but it has no public IP address or open port that a client on the internet can send UDP to. The only way in is through the Lambda MicroVM API, over HTTPS. So even if you installed iperf2 in the image and ran
iperf -s -u , it would listen happily and no client could ever reach it. And running the client on your machine doesn't change that: the client needs a server address to send to.An iperf2 client sends raw UDP datagrams to an IP address and a port. Something with a public address has to sit in front, receive the datagrams, turn them into HTTPS requests to the MicroVM, and turn the MicroVM's replies back into UDP. That's the job of Proxylity UDP Gateway. The alternative is to build that front end yourself on EC2 instances or containers with public addresses, and then you're back to running servers.
This also explains the shape of the app. It isn't
iperf running in a MicroVM. It's a small program that re-implements the iperf2 server's side of the protocol, working from batches of datagrams delivered as HTTPS requests.The gateway is a commercial service, but at the volume of this example it's free, and Proxylity accounts have a setting that hard-stops processing before any charges can be incurred. That setting is the safety net for this tutorial.
What you'll build
Some terms first, because the rest of the tutorial uses them:
- A test is one run of the iperf2 client. It sends numbered datagrams at a set rate for a number of seconds, then sends a final datagram (the FIN) that means "that's all, tell me how it went".
- The server report is the answer: a single small datagram containing how many bytes and datagrams arrived, how many were lost or out of order, and the jitter. The client prints it.
- The listener is the public UDP endpoint the gateway gives you. The destination is where the listener sends what it receives, here the MicroVM.
- The app is the small C# program we write. It runs inside the MicroVM and plays the part of the iperf2 server: it counts datagrams per test and builds the server report. It's the thing being exercised; the network path to it is what you're measuring.
- The listener receives the client's datagrams, groups them into batches, and delivers each batch to the destination as JSON over HTTPS. Whatever the destination replies is sent back to the client as UDP.
- The destination is a Lambda MicroVM, started from an image you build, which keeps running and keeps its memory between requests.
- Inside it runs a ~10 MB C# program, compiled ahead of time to a single native executable. It keeps each test's statistics in memory and answers the end of the test with a standard iperf2 server report.
How the tutorial is laid out
The parts build on each other. Each one opens with a Goal and a Why. Where you're meant to do something, there's a Try it block, followed by You should see so you can check your work. Parts without those blocks are explanation, and you can read them without running anything.
- Learn what iperf2 really sends
- Decide where the state lives
- Write the app
- Run it locally, no AWS needed
- Package it as a MicroVM image
- Deploy and run a real test
- Prove it's right
What you'll learn
- How to find out what a protocol really sends before you write code for it
- Why where state lives is the first design decision for anything stateful, and why a MicroVM fits here
- Event time versus processing time, and why it matters as soon as packets are batched
- How to package an app as a Lambda MicroVM image, and the lifecycle hooks it needs
- How to prove a measurement tool is correct, not just that it runs
Prerequisites
- An AWS account, and the AWS CLI v2 (a version with the
lambda-microvmscommands) and SAM CLI configured for us-west-2 (or another region with Lambda MicroVMs)
- A Proxylity account. See the cost note below.
- Linux or WSL 2, with the .NET 10 SDK, a C toolchain (
clangorgcc),iperf(iperf2, not iperf3; this tutorial was tested with the 2.1.9 client),jq,xxd,zip, andcurl
-
sudo, only for the network-impairment proof near the end
All of the code is in the example repository: github.com/proxylity/examples/tree/main/iperf2-microvm . This tutorial walks through the important parts rather than every line.
Cost. The traffic in this tutorial is well inside Proxylity's free usage, and you can set your Proxylity account to stop processing before any charges would be incurred. On the AWS side you pay for the MicroVM while it runs (the template uses a 1 GB baseline and suspends it after 15 idle minutes). Logs are kept for 7 days. Delete the stack when you're done (see Clean up ).
1. Learn what iperf2 really sends
Goal. Know exactly what bytes an iperf2 client sends and what the server answers.
Why. It's tempting to start from documentation or memory. Don't. A server that's almost compatible fails in confusing ways, and the captured bytes become your first unit tests.
You don't need root or a packet capture tool. A tiny UDP proxy that sits between a real client and a real server, and logs every datagram as hex, is enough. The repository has one in
scripts/research/capture.py (about 40 lines of Python).Try it. Run a stock server, the proxy in front of it, and a stock client pointed at the proxy:
iperf -s -u -p 5002 &python3 scripts/research/capture.py 5001 127.0.0.1 5002 /tmp/cap/cap.jsonl &iperf -u -c 127.0.0.1 -p 5001 -t 2 -b 1MYou should see. The log has one line per datagram. Here's the beginning of the first datagram the client sent, and the start of the last one:
00000001 6af2292f 000236a9 00000000 48010098 ... <- first data datagramffffff4a 6af22931 0002807e ffffffff 48010098 ... <- final datagram (FIN)Reading it as big-endian 32-bit words:
Bytes | Meaning |
|---|---|
0-3 | Sequence number, low 32 bits. Starts at 1. |
4-7, 8-11 | When the client sent it: seconds and microseconds |
12-15 | Sequence number, high 32 bits (modern clients use 64-bit sequence numbers) |
16-19 | Client flags, in the first datagram |
The test ends with the FIN: a datagram whose sequence number is negative.
0xffffff4a is -182, meaning "that was datagram 182 of 182". The server answers it with a 128-byte server report (byte count, duration, lost, out-of-order, total, jitter), and the client prints it.Two more facts matter later, and both are covered in What surprised me : the client repeats the FIN until it gets an answer, and the stock server counts the FIN as a received datagram.
Save the captured bytes, including the stock server's report. They're the reference the app is held to.
2. Decide where the state lives
Goal. Choose the compute for the app by looking at what it has to remember.
Why. For anything stateful, this is the first design decision, and the rest of the design follows from it.
What the server has to remember
The server only learns how many datagrams the client sent when the FIN arrives, at the end of the test. To produce the server report at that moment, it has to have kept, for every datagram of the test: which sequence numbers have arrived (to spot loss, duplicates, and reordering), the byte count, when each arrived, and a running jitter estimate. That state changes with every datagram and is read as a whole when the FIN arrives.
So the question is where that state lives while the test runs.
Option A: a Lambda function as the destination
The gateway can deliver batches to a regular Lambda function instead of a MicroVM. A function doesn't keep memory between invocations, and the gateway may invoke several copies at once for the same test. So the statistics would have to live in an external store such as a cache, and every invocation would have to merge its batch into it atomically. That design works, but it needs a VPC, a cache, atomic merge scripts, and a replay of every stored record at the end to work out duplicates and ordering, all inside the 2-second window the client allows for the answer.
Option B: a Lambda MicroVM
A MicroVM is one long-lived process with its own memory. The state becomes a dictionary of flows (a flow is one test, identified by the sender's IP address and port plus the listener port) and a bitmap per flow, updated as datagrams arrive. No cache, no VPC, no merge scripts.
This tutorial uses option B.
Rule of thumb: decide where the state lives before you choose the compute. When state is hot, per-session, and read as a whole at the end, keeping it in one process's memory removes a whole class of problems.
One MicroVM for every test
The destination can start one MicroVM per tenant, either with a fixed
TenantId or with a TenantIdExpression that pulls a tenant ID out of each packet's bytes. It's tempting to give each test its own MicroVM. It doesn't work here: an iperf2 datagram carries no test identifier, and the expression only sees payload bytes, not the sender's address. A tenant computed from the bytes would scatter one test across several MicroVMs and break its statistics.So this design uses a fixed
TenantId . One MicroVM serves every test, and the flow key keeps tests apart in memory. The fixed ID turns out to be handy: changing it gives you a brand new MicroVM with empty memory, without rebuilding anything.Per-tenant MicroVMs are a great fit for protocols that do carry a session or device ID in the payload. iperf2 just isn't one of them.
3. Write the app
Goal. Build the code that plays the iperf2 server: read the gateway's batches, parse iperf2 datagrams, keep the statistics, and produce the server report.
Why. This is the part you'd reuse for another protocol. The details change, but the shape (contract, codec, state, clocks, locking) doesn't.
The app is an ASP.NET Core minimal API in C#. The full code is in
src/ ; here are the pieces worth understanding.How the gateway talks to your app
The listener delivers batches to the MicroVM as an HTTPS
POST on port 8080:{ "Messages": [{ "Tag": "opaque-correlation-tag", "ReceivedAt": "2026-11-10T00:15:31.1936828Z", "Remote": { "IpAddress": "203.0.113.10", "Port": 45678 }, "Local": { "Domain": "ingress-1", "Port": 3164 }, "Formatter": "base64", "Data": "AAAAAWryKS8AAjapAAAAAEgBAJg..." }]}Your app answers with replies matched to messages by
Tag . For iperf2, that's one reply: the server report, in answer to a FIN.{ "Replies": [{ "Tag": "opaque-correlation-tag", "Data": "<base64 server report>" }] }One batch can hold datagrams from many tests, and one test arrives across many batches.
Parse datagrams and write reports
The protocol code is small. Parsing reads the fields from Part 1:
var sequenceLow = BinaryPrimitives.ReadUInt32BigEndian(payload);var seconds = BinaryPrimitives.ReadUInt32BigEndian(payload[4..]);var microseconds = BinaryPrimitives.ReadUInt32BigEndian(payload[8..]);var sequenceHigh = payload.Length >= 16 ? BinaryPrimitives.ReadUInt32BigEndian(payload[12..]) : 0;Writing the report fills in the same fields at the offsets a stock server uses. The best test for both is the bytes you captured in Part 1. The unit tests feed in the captured FIN and check the result against the captured stock-server report, byte for byte:
[Fact]public void ReportMatchesStockServerAtEveryFieldWeFill(){ var ours = Iperf2Protocol.WriteServerReport(true, new(ReceivedBytes: 267_540, DurationMicroseconds: 2_017_196, LostDatagrams: 0, OutOfOrderDatagrams: 0, Datagrams: 182, JitterMicroseconds: 36)); var stock = Convert.FromHexString(StockReport);
Assert.Equal(stock.Length, ours.Length); Assert.Equal(stock[..56], ours[..56]);}These are called golden tests: the expected answer comes from the real system, not from your own understanding of it.
Keep the statistics in memory
This is the heart of the app:
FlowTable , a dictionary of flows. Each flow keeps counters, a bitmap of sequence numbers it has seen, and a jitter estimate.private void Apply(Flow flow, in Arrival arrival){ if (!flow.MarkSeen(arrival.Sequence)) // bitmap: was this sequence number seen before? { flow.Duplicates++; // yes: a duplicate. Note it, don't count it. return; }
flow.Received++; flow.Bytes += arrival.Length; if (arrival.Sequence < flow.HighestSequence) flow.OutOfOrder++; else flow.HighestSequence = arrival.Sequence;
// Jitter: the smoothed change in transit time (RFC 1889), the estimator iperf2 uses. var transit = arrival.ReceivedMicroseconds - arrival.SentMicroseconds; if (flow.HasTransit) flow.Jitter += (Math.Abs(transit - flow.PreviousTransit) - flow.Jitter) / 16; flow.PreviousTransit = transit; flow.HasTransit = true;}Two details matter more than they look.
Loss is only calculated at the end. It's tempting to compute loss as "highest sequence minus received" as you go. But a datagram that's late isn't lost yet. The FIN tells you how many datagrams the client sent, so loss is simply sent minus unique received, worked out when the test ends. A late datagram that fills a gap is never counted against you.
Duplicates are detected exactly. The bitmap means a duplicate is never counted as received data. That's not a nicety: you'll see in Part 7 that a stock iperf2 server doesn't do this, and a duplicate can hide a real loss.
Event time versus processing time
The gateway delivers batches concurrently, so your app can process a later batch before an earlier one. In the first local test of this project, a perfectly ordered test reported "20 datagrams received out-of-order". None were. The processing order was scrambled, not the arrival order.
Two clocks are involved, and each has its job:
- Event time: when the datagram reached the gateway. That's each message's
ReceivedAt, and it's what every measurement uses. Using the VM's clock instead would turn batching delay into fake jitter.
- Processing time: when your app gets around to it. That's the VM's own clock.
So arrivals wait in a small reorder buffer for 500 ms of event time, and are folded in oldest first:
pending.Sort(static (a, b) =>{ var byTime = a.Arrival.ReceivedMicroseconds.CompareTo(b.Arrival.ReceivedMicroseconds); return byTime != 0 ? byTime : a.Order.CompareTo(b.Order); // ties keep delivery order});The tie-breaker matters because
List.Sort isn't stable: two datagrams with the same timestamp could otherwise swap places and show up as reordering.The FIN uses the other clock. When the first FIN arrives, the app waits 150 ms before answering, so batches that are still in flight get counted. That wait is about delivery, so it's measured on the VM's clock:
// The grace gives batches that are still in flight time to land, so it is measured on this// machine's clock (processing time). ReceivedAt (event time) is only used for the statistics.if (now - flow.FirstFinSeenAt < (long)options.FinalizationGrace.TotalMilliseconds) return new(FinAction.Withheld, default, 0, fin.Uses64BitSequence);Withholding the reply is safe because the client keeps retrying the FIN. The next retry after the grace period gets the report. The report is then kept for 10 seconds, so any later retries get the same answer.
If you take one idea from this tutorial into stream processing, databases, or analytics, make it this one: measure with event time, wait with processing time.
Thread safety
ASP.NET Core handles requests on several threads, so two batches for the same flow can be processed at the same moment. The fix is one
lock around the flow table. Parse the batch outside the lock, then fold it in inside. The work inside is microseconds of arithmetic, and there's no I/O, so the lock is never held for long. A unit test hammers it from 8 threads to make sure no update is lost.4. Run it locally, no AWS needed
Goal. See a stock iperf2 client get a correct server report from your app, on one machine.
Why. A local loop is far faster than deploying, and this is where the protocol surprises showed up, long before any AWS bill.
tools/LocalGateway is a stand-in for the Proxylity gateway. It listens for UDP, groups datagrams into batches, posts them to your app concurrently in the same JSON format, and sends the replies back.Try it.
dotnet test tests/Iperf2.Testsscripts/dev/local-test.sh -t 5 -b 2MYou should see. The unit tests pass, and the client prints a server report:
[ 1] Sent 2234 datagrams[ 1] Server Report:[ 1] 0.0000-5.0022 sec 3.13 MBytes 5.25 Mbits/sec 0.051 ms 0/2233 (0%)That's a stock iperf2 client, talking to your app, through a stand-in gateway, on one laptop.
5. Package it as a MicroVM image
Goal. Turn the app into an image Lambda can start MicroVMs from.
Why. MicroVMs are restored from a snapshot of your running app, and they're driven by lifecycle hooks. Those two facts shape how the app is built.
Compile it to one native executable
I compiled the app with NativeAOT: ahead-of-time compilation to a single native executable. This is a choice, not a requirement; any language that can serve HTTP works. I picked it because there's no .NET runtime to put in the image, the image stays small, and startup is fast.
<Project Sdk="Microsoft.NET.Sdk.Web"> <PropertyGroup> <TargetFramework>net10.0</TargetFramework> <AssemblyName>iperf2-microvm</AssemblyName> <PublishAot>true</PublishAot> <InvariantGlobalization>true</InvariantGlobalization> <StripSymbols>true</StripSymbols> <OptimizationPreference>Size</OptimizationPreference> <JsonSerializerIsReflectionEnabledByDefault>false</JsonSerializerIsReflectionEnabledByDefault> </PropertyGroup></Project>NativeAOT trims away code it can't see being used, so reflection-based JSON serialization doesn't work. Instead, declare the types you serialize and let a source generator write the serializer at compile time:
[JsonSourceGenerationOptions(PropertyNameCaseInsensitive = true, DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull)][JsonSerializable(typeof(GatewayRequest))][JsonSerializable(typeof(GatewayResponse))]// ...one line per type you read or writepublic partial class AppJson : JsonSerializerContext;Two ports and the lifecycle hooks
The app listens on two ports:
- 8080 for packet batches from the gateway
- 9000 for the Lambda MicroVM lifecycle hooks:
/ready(called while the image is built) and/run(called when each MicroVM starts)
They're on separate ports on purpose. The token the gateway uses to reach your MicroVM is scoped to port 8080, so it can't call your hooks.
Snapshots, and why
/run matters. When Lambda builds the image, it starts your app, waits for /ready to return 200, and then snapshots the running process. Every MicroVM is restored from that snapshot. That makes startup very fast, but anything your app computes at startup (a random ID, a timestamp) is identical in every MicroVM.So per-MicroVM values come from the
/run hook instead. Lambda calls it with the MicroVM's ID, and the gateway adds a payload with the tenant ID and any settings you configured on the destination:app.MapPost(RunHookPath, async (HttpContext context) =>{ var run = await context.Request.ReadFromJsonAsync(AppJson.Default.RunRequest, context.RequestAborted); microvmId = run?.MicrovmId ?? microvmId; // ...read the tenant ID and settings from the payload... return Results.Text("{}", "application/json");}).AddEndpointFilter(OnlyOnPort(HookPort));Gotcha: route by port, not by Host header. The first deployment of this project failed while building the image: "Ready hook check failed: the application returned a client error (HTTP 4xx)". The app had separated its two ports with
RequireHost("*:9000") , which matches the HTTP Host header. Lambda's hook request doesn't put the port in that header, so every request was a 404. It had passed locally only because curl does.The fix is to check the port the connection actually arrived on:
static Func<EndpointFilterInvocationContext, EndpointFilterDelegate, ValueTask<object?>> OnlyOnPort(int port) => (context, next) => context.HttpContext.Connection.LocalPort == port ? next(context) : ValueTask.FromResult<object?>(Results.NotFound());The lesson is bigger than the fix: make your local test harness behave like the platform. The repository's local scripts now send a
Host header without a port, and fail if /ready doesn't return 200.Build the image
Try it.
scripts/publish.shIt runs
dotnet publish -r linux-x64 and stages the executable and a Dockerfile in build/image/ . The Dockerfile is three lines of substance:FROM public.ecr.aws/lambda/microvms:al2023-minimalCOPY iperf2-microvm /app/iperf2-microvmENTRYPOINT ["/app/iperf2-microvm"]al2023-minimal is Lambda's own minimal base for MicroVMs. A NativeAOT binary still needs the C library, so it has to be built against a version of glibc no newer than the base image's (2.34). publish.sh checks that and stops if not.Gotcha: architecture. MicroVMs default to ARM64. This example builds x86_64 (linux-x64), which needs base image version 3 or later;deploy.shpicks a version that supports your architecture. If the binary and the image disagree, the build fails or the app won't start ("Exec format error").
6. Deploy and run a real test
Goal. Put the listener and the MicroVM in your AWS account and run a stock iperf2 client against them.
Why. This is the point of the whole exercise: a normal client, on a normal network, getting a normal result from an endpoint with no server to manage.
The template
The SAM template (
template.yaml ) declares the image, two roles, a log group, and the listener. A few parts are worth understanding.The image builds from your staged folder, with the hooks enabled:
Iperf2MicrovmImage: Type: AWS::Serverless::MicrovmImage Properties: Name: !Sub "${AWS::StackName}-server" CodeUri: build/image/ BaseImageArn: !Sub "arn:${AWS::Partition}:lambda:${AWS::Region}:aws:microvm-image:al2023-1" CpuConfigurations: - Architecture: !Ref Architecture Hooks: Port: 9000 MicrovmImageHooks: { Ready: ENABLED, ReadyTimeoutInSeconds: 60 } MicrovmHooks: { Run: ENABLED, RunTimeoutInSeconds: 10 }The log group name matters. The image's logging setting applies to the build. At runtime, a MicroVM writes its stdout to Lambda's default group,
/aws/lambda-microvms/<image-name> , using its execution role. So the template creates the group with exactly that name, with 7-day retention, and gives the execution role permission to write to it. If the execution role can't write to the group, your app's runtime output silently disappears.If you'd rather use a group with another name, set the destination's
RuntimeLogGroupName argument and scope the execution role to that group instead. And UseTenantIdForLogStreamName: "true" names the log stream after the tenant (here, iperf2-1 ) rather than after each MicroVM, so your app's output stays in one stream even when you start a fresh MicroVM.The listener sends packets to the image, with a fixed tenant:
Destinations: - Role: Arn: !GetAtt DestinationRole.Arn Formatter: base64 Batching: Count: 1000 TimeoutInSeconds: 0.25 DestinationArn: !GetAtt Iperf2MicrovmImage.ImageArn Arguments: TenantId: !Ref TenantId IdleDurationSeconds: !Ref IdleDurationSeconds ExecutionRoleArn: !GetAtt MicrovmExecutionRole.Arn RunHookExtraData: '{"FinGraceMs":150,"LogBatches":true}'Batching delay never shows up in the measurements, because every datagram carries its own
ReceivedAt . But batching is critically important to ensure we don't exceed the MicroVM rate limits which are small: 40-160 requests per second depending on the vCPU configuration. A faster rate leads to 429 errors.The destination role is what the gateway assumes to start MicroVMs from your image and get tokens to reach them. Its permissions are scoped to this one image.
ClientRestrictions limits who can send packets to the listener; the deploy script fills it in with your public IP address.Deploy and test
Try it.
scripts/deploy.shAt the time of writing, SAM CLI can't yet upload the code for
AWS::Serverless::MicrovmImage by itself, so deploy.sh zips and uploads build/image/ , then deploys the template. It prints your endpoint. Then:scripts/warmup.shiperf -u -c <domain> -p <port> -t 10 -b 1MThe warmup runs a tiny two-datagram test first, so the MicroVM is running before you measure.
You should see.
------------------------------------------------------------Client connecting to ingress-1.proxylity.com, UDP port 3164Sending 1470 byte datagrams, IPG target: 11215.21 us (kalman adjust)UDP buffer size: 208 KByte (default)------------------------------------------------------------[ 1] local 172.19.89.134 port 35965 connected with 15.197.76.92 port 3164[ ID] Interval Transfer Bandwidth[ 1] 0.0000-10.0154 sec 1.25 MBytes 1.05 Mbits/sec[ 1] Sent 896 datagrams[ 1] Server Report:[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams[ 1] 0.0000-10.0196 sec 1.25 MBytes 1.05 Mbits/sec .579 ms 0/895 (0%)The first
Bandwidth line is the client's own view of what it sent. Everything under Server Report was calculated by your app in the MicroVM and sent back as a single UDP datagram. Don't worry that Sent 896 is one more than the 895 in the report: that's how the iperf2 client counts. Against a stock iperf2 server, the capture in Part 1 showed the same thing ( Sent 183 , report 0/182 ).Your domain and port will differ. The deploy script prints yours.
Want to see the state living in memory? Ask the MicroVM:
scripts/status.sh{"Status":"ok","Microvm":"microvm-...","Tenant":"iperf2-1","ActiveFlows":0,"CompletedFlows":0,"Batches":184,"Packets":802}That talks to the MicroVM's own HTTPS endpoint with a short-lived token for port 8080: the same way the gateway delivers packets.
7. Prove it's right
Goal. Show that the numbers the app reports are correct, not just plausible.
Why. A measurement tool that prints plausible numbers isn't the same as a correct one. If the app is wrong, every network you test with it gets the wrong answer, and nothing will tell you.
The repository includes bash scripts that check the results against independent sources of truth, and a check passes only when they agree.
Try it.
IMPAIR=1 scripts/prove/run-all.shYou should see a PASS/FAIL table for these six proofs:
Proof | What it checks |
|---|---|
Deterministic | Hand-built datagrams with known faults, sent from bash; the report must be exactly right |
Oracle | The same datagrams sent to a stock iperf -s -u on your machine; the answers must match |
Stock client | An unmodified client; the backend must have recorded the same numbers the client printed |
Impairment | Known loss, duplication, and reordering injected with tc netem ; the measured rates must match |
Parallel | Five tests at once, with two datagram sizes; no datagram may leak between tests |
State | A slow test spread over many deliveries; counts only ever go up, and memory is cleaned up afterwards |
Packets you can build by hand
The deterministic proof needs no iperf client at all. Bash can send UDP by itself through
/dev/udp , and printf plus xxd can build the bytes:exec 3<>"/dev/udp/$HOST/$PORT" # a UDP socket you can write to and read fromiperf2_datagram 5 100 | xxd -r -p >&3 # send datagram 5, 100 bytes longreply=$(timeout 0.02 dd bs=65535 count=1 status=none <&3 | xxd -p) # read one reply datagramThe script sends datagrams 1 to 50, but skips 10, sends 31 before 30, and holds 20 back until after 45. So the right answer is known in advance: 51 datagrams (50 + FIN), lost 1, out-of-order 2, 5000 bytes. The backend got exactly that, and so did a stock iperf2 server given the same packets.
Something more correct than the original
A second scenario sends datagram 5 twice and skips datagram 10. The correct answer is "lost 1". This backend says lost 1 and logs one duplicate. The stock iperf2 server says lost 0: it counts the duplicate as received data, and that cancels out the real loss. The bitmap from Part 3 is why this version gets it right.
Faults in, the same faults out
The impairment proof uses Linux traffic control to damage outgoing packets on purpose: drop 5%, duplicate 2%, and reorder 25%. Then it runs an ordinary 10-second test:
[ 1] 0.0000-10.0074 sec 2.38 MBytes 2.00 Mbits/sec 4.712 ms 87/1787 (4.9%)[ 1] 0.0000-10.0074 sec 103 datagrams received out-of-orderInjected | Measured |
|---|---|
5% loss | 87 of 1787 lost (4.87%) |
2% duplication | 37 duplicates detected (2.18%), none counted as data |
25% reordering | 103 datagrams out of order |
Random faults can only be checked statistically, so this proof accepts results within three standard deviations of the injected rate for the sample size. That's exactly why the deterministic proof exists too: it's where the answer must be exact.
In the reference run, all six proofs passed: 60 checks, 0 failures. Two more numbers from it are worth pointing out. The stock-client test arrived in 164 separate HTTPS deliveries, all folded into one in-memory flow, and a slow 20-second test spanned 176 deliveries to the same MicroVM, with the received count only ever going up. That is the state-in-memory design doing its job.
What surprised me
Reading about a protocol and watching it are different things. Watching the real client turned up three details that no amount of reading would have caught, and each one changes what a correct server does.
To see the first two, point the capture proxy from Part 1 at a port with nothing listening, so the FIN never gets an answer, and watch what the client does:
seq= -188 dt= 0.0 msseq= -189 dt= 10.2 msseq= -190 dt= 10.0 ms...data=11 fins=201 first=-12 last=-212 span=2033 ms- The client resends the FIN about every 10 ms for about 2 seconds. Your server has a 2-second window to answer. This is also why withholding the first reply for 150 ms (Part 3) is safe: another FIN is always coming.
- Every retry carries the next sequence number (-12, -13, -14, ...). If you take the largest FIN as the total, you'll report loss that never happened. The first FIN, the smallest number, is the real total.
- The stock server counts the FIN itself as a received datagram (182 = 181 data + 1 FIN). Matching that is what makes your report read
0/182on a clean run instead of1/182.
Other surprises came from the platform rather than the protocol: the
Host header gotcha in Part 5, and the processing-order jitter in Part 3. Capturing early and testing locally caught most of them before any AWS bill.Troubleshooting
- The image build fails at the ready hook. Your app isn't returning 200 on
/aws/lambda-microvms/runtime/v1/readyon the hook port. Check that it routes by the connection's port, not theHostheader (Part 5). CloudFormation rolls back a failed image and deletes its build log, soscripts/dev/debug-image-build.shbuilds the image outside CloudFormation to keep the reason.
- No runtime logs. Check the log group name and the execution role (Part 6).
- The client prints "did not receive ack of last datagram". No report arrived within about 2 seconds. Run
scripts/dev/recent.shto see whether the test's datagrams and FINs reached the MicroVM, and what it did with them.
- The first test after a while is slow. The MicroVM was suspended after 15 idle minutes and is resuming. Its memory survives a suspend.
Clean up
scripts/teardown.shMicroVMs are started by the gateway, not by CloudFormation, so the script terminates them first and then deletes the stack.
What's next
- Reverse mode (
iperf -u -c ... -R), where the server sends the traffic. The MicroVM can hold the schedule and publish outgoing datagrams through a Proxylity PacketSource, with its topic passed in through the destination's run hook settings.
- Find the limits. One MicroVM is one machine. How many packets per second and concurrent tests does it handle before the numbers degrade? How quickly does it scale out?
Wrapping up
We started by asking whether you can run an iperf2 server without running a server. The honest answer is that a Lambda MicroVM is a server, a small VM with a process and some memory. But it's one that starts when traffic arrives, goes to sleep when it stops, and costs almost nothing in between. And it fits this job well because the job needs a place to remember things while a test runs: a place Lambda functions don't have, and that a MicroVM provides.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article