
Tiny Models, Massive Impact: Running Distilled LLMs on AWS Lambda with Zero Cold Start
Key insights from AWS Community Day Bengaluru | July 11, 2026
Series: AWS Conferences Tour (42 articles)
- …
- 40Tiny Models, Massive Impact: Running Distilled LLMs on AWS Lambda with Zero Cold Start This article
I got the amazing chance to attend AWS Community Day Bengaluru recently on 11 July 2026. The event was full with excellent speakers and fantastic sessions all around, but one presentation totally altered the way I thought about generative AI architecture.
Next up was a talk from Samyuktha M S from IBM on executing distilled LLMs on AWS Lambda. LLM? When we hear the word, let's face it, we think of huge cloud costs, expensive GPU clusters, and sluggish API requests. Samyuktha, however, took a totally different approach: she reduced the model, ditched the GPUs, and ran the whole thing on a serverless architecture.
Here’s an in-depth look at everything I’ve learned, broken down so you can apply it to your own serverless applications.
The ₹40,000 Resume Screening Nightmare
To create the context, Samyuktha presented a difficulty faced by many engineering teams today. Suppose you are hiring for a single backend role. By Monday morning you had 800 applications in your mailbox. You have till the morning standup to come up with a shortlist.
Sound familiar?
The present urge is to wire those 800 CVs to a hosted frontier model like GPT-4o. This looks like a simple fix, yet it creates three giant headaches:
- The Cost: You want to screen 800 CVs? This will cost you around ₹40,000 for a single run with GPT-4o ($5.00 for 1 million tokens).
- The Latency: If you do 800 API requests in a row and each call takes ~800ms your job will creep along for ~11 minutes. Your standup might be over by the time it’s over.
- The Privacy Nightmare: You are dealing with 800 resumes with personal phone numbers, addresses, job histories and sending them off to a third party server. You will not have a good answer when a candidate simply writes you asking, “Where is my data stored?”
Stop Renting a Mansion for a Weekend Getaway
This is where the fun starts. Why isn’t a frontier model the right tool here?
Imagine buying food from Swiggy. But the company sends a giant 18-wheeler truck to deliver your one pizza, instead of a delivery guy on a scooter. It does the job but is highly wasteful.
What the resume screening assignment is basically about is pulling data (years of experience, tech stack, role), matching it to your job description, assigning a label to the candidate (strong, maybe, no) and providing a one-line justification.
It doesn’t need world knowledge, complicated multi-step reasoning or a 100K token context window. We want easy decision boundaries, not high IQ's. We have around 250MB of logic for this job and we are renting 1.7 trillion parameters to run that.
Enter TinyLlama: The 250MB Hero
Every step down the model size ladder is a trade-off, sacrificing broad knowledge you don’t need for a smaller footprint, Samyuktha explained.
But GPT-4o needs a hosted API, and a 70B Llama 3 model needs a dedicated GPU fleet. A model like TinyLlama (1.1 billion parameters) is small by nature. But we can make it smaller.
Quantization (32 bit floating point (FP32) to 4 bit integer (INT4)) can compress the model 8x per weight with only a ~0.5% drop in classification accuracy.
The ultimate footprint? A very capable model tuned for the task. 250MB. So it’s small enough to fit well inside an AWS Lambda container layer. No Gpu. There are no end points to stay warm. Amazon Bedrock is not needed.
AWS Lambda: The Perfect Host (With One Big Catch)
It’s hard to argue with AWS Lambda as a great host for an asynchronous, spiky application like a CV screener.
- Spiky Load: At 9 AM you get 800 CVs only. No traffic for the remainder of the day.
- Scale to Zero: You are billed per invocation. ₹0... When no one is hiring.
- Data Stays Home: The resumes never leave your AWS account or your VPC.”
However, there is a big catch. The Cold Start. To be fair.
The first time you call a Lambda function, it has to establish the environment. It takes roughly 200ms to start the runtime, 800–1200ms to pull the container, and then a painful 2000–4000ms to load the 250MB model into memory.
Time for chilly start? 3-6 seconds per CV. The model load stage alone is bigger than all the others combined. This is precisely why “just run the model on Lambda” has been a failing architectural pattern for years.
Crushing the Cold Start Wall with SnapStart
AWS Lambda SnapStart drastically changes the game.
Instead of having to re-build the environment from scratch every time, SnapStart takes a snapshot of your initialised environment, ensuring those 250MB model weights are already in RAM. On a fresh invocation, Lambda simply reclaims this pre-warmed memory.
This reduces initialisation time from 3-6 seconds to less than 200ms. It’s 30x faster.
Two Rules People Get Wrong with SnapStart
For this to function, Samyuktha pointed out two important rules:
- Load at the INIT scope, never in the handler. You need to load your model at the module level (eg
llm = Llama(...)so that it gets snapshotted. Make your actualhandlerfunction skinny.
- Keep it outside a VPC. Restoring a snapshot is fast, but if you put it inside a VPC, adding the Elastic Network Interface (ENI) adds seconds back to your execution time, completely ignoring the point.
Supercharging with ARM64
The solution is based on AWS Graviton3 (ARM64) architecture for even more performance. Moving to the NEON approach in
llama.cpp instead of the usual x86 AVX2 will bring you 34% quicker inference throughput, 2x more memory bandwidth, and a 20% cost decrease per millisecond.This is the same Dockerfile setup presented in the session to deploy this to Amazon ECR:
FROM public.ecr.aws/lambda/python:3.12-arm64RUN pip install llama-cpp-python --extra-index-url \ https://abetlen.github.io/llama-cpp-python/whl/cpuCOPY tinyllama-q4.gguf /opt/model.gguf # 250MB, own layerCOPY handler.py ${LAMBDA_TASK_ROOT}/
You merely set your Lambda to
arm64 , give it 3008MB of memory and turn SnapStart on for published versions.The Numbers Speak for Themselves: Traditional vs. Modern
Bottom line is? The live demo showed an impressive performance measure.
When comparing the latency of getting the first token (p50):
- GPT-4o (API): 820 ms
- Claude Haiku (API): 650 ms
- Bedrock Claude Instant: 480 ms
- Lambda + TinyLlama (Cold): 3,200 ms
- Lambda + TinyLlama (SnapStart): 118 ms (The clear winner)
Now let's have a look at the budgetary impact of running those 800 CVs. The original GPT-4o cost ₹40,000 to set up, took 11 minutes to do, and transported your data out of the nation.
The re-engineered AWS Lambda setup is around ₹40 (1,000x cheaper), completes in less than 2 mins thanks to huge concurrency without API queueing, and ensures your data never leaves your AWS account.
6 Traps to Avoid
If you’re considering doing this you may be wondering what the hidden risks are. Samyuktha has helpfully listed the six things that will bite you in production:
- Image > 10GB ceiling: AWS Lambda has a restriction on the number of containers it can process. Use a multi-stage build and maintain your model as a separate layer.
- SnapStart inside a VPC: As indicated, the ENI attachment kills your speed. If you must utilise a VPC, then use Provisioned Concurrency instead of SnapStart.
- Memory too low: TinyLlama needs roughly 1.5GB of memory while it is running. Set the RAM of your Lambda to exactly
3008MB, so you have enough CPU.
- Concurrent cold restores: If you have 800 requests all at once you can still have cold starts. Schedule pings from Amazon EventBridge to warm up your function.
- Context > 2048 tokens: limited models have limited context windows. You have to truncate forcefully otherwise the quality of the response drops off a cliff.
- Silent quality drift: You won’t know if the model begins to go wrong. Log all verdicts to Amazon CloudWatch and sample them manually once a week.
The Golden Rule for Task Matching

TinyLlama is great for filtering CVs but it is not a silver bullet. If your workload requires the following, you should explicitly avoid this arrangement and route to Amazon Bedrock instead:
- Multi-step reasoning (e.g., "Rank these 5 candidates and justify the trade-offs").
- Code generation or review (e.g., diffs, bug-finding, refactoring).
- Long context windows (e.g., processing full 40-page legal contracts).
- Post-2023 world knowledge that the tiny model was never trained on.
- Open-ended creative generation (e.g., writing marketing emails or creative copy).
Long story short, you can use a simple three-step rule to evaluate any AI task:
- Is it a narrow task? (Fixed output, no complex reasoning chains).
- Is it high volume? (Enough API calls to really care about the cloud costs).
- Is it privacy-bound? (Data must be strictly within your own AWS account).
If all three are YES, run a small model on Lambda. You own it. It’s inexpensive and lightning fast. If the answer to any of these is NO, then use the strong models offered by Amazon Bedrock.
Key Takeaways
- Stop overpaying for simple tasks: Basic data extraction using a giant frontier model like GPT-4o is expensive and too much. You are renting 1.7 trillion parameters for a work that fits in 250MB.
- Tiny models are highly capable: Quantized models, like TinyLlama (going from 32-bit to 4-bit), maintain almost all of the accuracy while fitting nicely within an AWS Lambda container layer.
- SnapStart kills the cold start wall: The old 3-6 second latency in loading a model on Lambda is gone. SnapStart takes a snapshot of the initialised environment and recovers the model from RAM in less than 200ms.
- Keep your data at home: Running a small model directly in your own serverless architecture means that sensitive data such as phone numbers and addresses on a CV never leaves your AWS account.
Conclusion
“Ultimately, the largest model is not always the correct model.” The optimal architecture is the one that ships, scalability to your traffic surges and doesn’t blow up your cloud cost.
This session was a great reminder that as engineers, we need to appropriately scale our tools. AWS gives us both ends of the range – Lambda for rapid, narrow, private activities and Amazon Bedrock for the complicated, reasoning-heavy workloads. The point is? Match the model with the assignment.
About the Author
As an AWS Community Builder, I enjoy sharing the things I've learned through my own experiences and events, and I like to help others on their path. If you found this helpful or have any questions, don't hesitate to get in touch! 🚀
References
Event: AWS Community Day Bengaluru
Topic: Tiny Models, Massive Impact: Running Distilled LLMs on AWS Lambda with Zero Cold Start
Date: July 11, 2026
Series: AWS Conferences Tour (42 articles)
- …
- 40Tiny Models, Massive Impact: Running Distilled LLMs on AWS Lambda with Zero Cold Start This article
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article