AWS Builder Center

Builder Center

Scale-to-zero ML on a student budget: running a deep learning model for ₹0 when nobody's looking

How we served a roof-detection ML model on SageMaker Serverless with Step Functions, so it costs nothing when idle, and what fought back along the way.

Scale-to-zero ML on a student budget: running a deep learning model for ₹0 when nobody's looking

How we served a roof-detection model with Amazon SageMaker Serverless Inference and orchestrated it with AWS Step Functions, for Environmental Hacks (WeMakeDevs x AWS, Track 03: Waste and Energy)
Here's a problem every student team building with ML on the cloud runs into. You have a model. It works beautifully on your laptop. Now it needs to live somewhere on the internet, answer requests, and not burn through your credits while you sleep.
The obvious move is to rent a machine with a GPU and leave the model running on it. It's also the move that quietly empties your account, because a provisioned endpoint bills you for every hour it is switched on, whether anyone calls it or not. During a hackathon, that's most hours.
We had $200 of AWS credits and a roof-segmentation model to deploy. This post is about how we made that model cost nothing while idle, and the surprising number of small things that stood between "works on my laptop" and "works on AWS".


What we were building

SolarRooftop AI tells a homeowner in Delhi whether their roof is worth putting solar panels on. You drop a pin, and about 20 seconds later you get a panel layout, a year-long shade map, annual energy, and a payback period in rupees.
Under the hood, every request runs a small pipeline:
  1. Fetch a satellite tile and building outlines for the address
  1. Segment the roof with our ML model (roof, water tanks, stair rooms, trees...)
  1. Simulate a year of sunlight and shadows, lay out panels, and work out the money
  1. Save the report
Step 2 is the one that needs a real neural network: a SegFormer-B2 our ML teammate trained in PyTorch, first on a German roof dataset and then fine-tuned on Delhi. Everything in this post is about making step 2 cheap, reliable and boring.


Why serverless inference was the right call

A few numbers settled the decision for us:
  • The model package ( model.tar.gz ) is about 100 MB.
  • Running it uses about 1.5 GB of memory.
  • One prediction on a 512 x 512 tile takes under a second on a plain CPU.
The model simply doesn't need a GPU. And our traffic is spiky: nothing for hours, then a judge or a teammate runs ten assessments in a row.
That is exactly the shape SageMaker Serverless Inference is built for. You give it a container and a memory size, and it handles the rest. When requests arrive, it spins up capacity. When they stop, it scales back down to zero, and you pay only for the time spent actually processing requests.
The trade-off is the cold start. The first request after a quiet period has to wait while SageMaker pulls the container image and loads the model. We measured it:
Call
Round trip
First call after idle (cold)
12.3 s
Next call (warm)
3.8 s
For a "check my roof" tool, a 12-second wait the first time is completely fine. Paying for a GPU around the clock to save those 12 seconds is not.


Bring your own container: a tiny contract

SageMaker has prebuilt containers, but we wanted full control over dependencies (OpenCV, shapely and a pinned PyTorch), so we used bring your own container. The contract SageMaker expects is refreshingly small:
  • It starts your image with the argument  serve .
  • It unpacks your  model.tar.gz  into  /opt/ml/model .
  • It expects an HTTP server on port 8080 with two routes:  GET /ping  (are you healthy?) and  POST /invocations  (do the work).
Our server is a little over 100 lines of Python using only the standard library's  http.server . It loads the model once at startup, so  /ping  only says "ok" when the model is actually ready, and the cold start is paid once per container instead of once per request.
Python
# simplified from ml/serve/serve.py

def main():

    if len(sys.argv) > 1 and sys.argv[1] != "serve":

        sys.exit("SageMaker starts the container with 'serve'")

    get_model()  # load at startup: /ping is fast and the cold start is paid once

    ThreadingHTTPServer(("0.0.0.0", 8080), Handler).serve_forever()
The image itself is  python:3.12-slim  plus the CPU-only PyTorch build, which keeps it far smaller than a GPU image. Smaller image, faster cold start.


Three resources and one setting

We define all our infrastructure with the AWS CDK in Python. A SageMaker endpoint is three resources: a model (image + weights + permissions), an endpoint config (how to run it), and the endpoint itself. The whole "serverless" part is a single block:
Python
sagemaker.CfnEndpointConfig(

    self, "SegEndpointConfig",

    production_variants=[sagemaker.CfnEndpointConfig.ProductionVariantProperty(

        model_name=model.attr_model_name, variant_name="AllTraffic",

        serverless_config=sagemaker.CfnEndpointConfig.ServerlessConfigProperty(

            memory_size_in_mb=3072, max_concurrency=2))],

)
That  serverless_config  replaces an instance type. No instance type means no machine sitting there billing you. A CDK unit test checks that no endpoint config ever gets an  InstanceType , so nobody on the team can accidentally switch us to a provisioned endpoint.


Step Functions: the conductor that makes cold starts harmless

Serverless endpoints have two moods you need to handle. They can be cold (slow to answer) and they can be busy (we capped concurrency at 2, so a batch of ten jobs will get throttled). If your code treats either of those as a failure, users see errors that aren't really errors.
We run the pipeline with AWS Step Functions and let it do the patient waiting for us. A small Lambda function calls the endpoint and translates SageMaker's errors into two kinds:
Python
class ModelBusy(Exception):

    """Throttled, not ready, or timed out during a cold start. Step Functions retries this."""



class SegmentationFailed(Exception):

    """The model rejected the input or returned invalid output. Not retried."""
Here's a neat trick: Step Functions sees the Python exception's class name as the error type. So the state machine can retry only the errors that are worth retrying:
Python
segment.add_retry(errors=["ModelBusy"], interval=Duration.seconds(10),

                  max_attempts=4, backoff_rate=2)   # waits 10 s, 20 s, 40 s, 80 s
A cold start or a throttle becomes a short wait. A genuinely bad input fails immediately, without wasting retries on something that will fail again.
A few other Step Functions choices that kept things cheap and simple:
  • Status updates without Lambdas. Step Functions can write to DynamoDB directly, so "status = segmenting" is a built-in task, not another function to pay for and maintain.
  • Pass addresses, not documents. Data passed between states is capped at 256 KB, and a full solar report can get big. Every step writes its output to Amazon S3 and only hands the next step the file's location.
  • Catch everything. Any unexpected error jumps to a state that marks the job as failed in DynamoDB with a readable message. The user sees "something went wrong" instead of a spinner that never stops.
  • "No roof here" isn't an outage. If the model finds no roof (say, someone pinned a road), the job fails politely for the user, but the execution still ends as a success, so our CloudWatch failure alarm doesn't email us about user mistakes.
One gotcha worth knowing: if the whole workflow hits its overall timeout, the Catch states don't run. Our API therefore treats any job stuck "running" for more than 12 minutes as failed, so nothing can hang forever.


What fought back

Scale-to-zero is great once it works. Getting there had a few surprises.
1. SageMaker rejected our image format. Docker Desktop's newer image store produces OCI image manifests. Lambda accepted them. SageMaker did not: "Unsupported manifest media type application/vnd.oci.image.manifest.v1+json". We now pre-build our images in the classic Docker v2 format before the CDK uploads them.
2. New accounts have smaller limits. We first asked for 4096 MB. On a new account, SageMaker Serverless allows at most 3072 MB per endpoint (and Lambda at most 3008 MB per function). Since the model needs about 1.5 GB, 3072 MB was plenty, and CloudFormation rolled back cleanly each time we got it wrong.
3. Old images can hide behind reused tags. The CDK names images by a hash of their build inputs and skips the upload if that tag is already in ECR. Changing only how an image is built doesn't change the hash, so the old OCI image could have been reused. Adding an  extra_hash  to the image asset forced a fresh tag.
4. Logs that live forever. SageMaker creates its endpoint log group itself, with no expiry. We create that log group in the CDK first, with 7-day retention, so old logs don't slowly turn into a bill.
5. Cold containers are slower than cold models. The model endpoint wakes up in about 12 seconds. Our solar-engine Lambda, a big container with geospatial libraries, took about 76 seconds on its very first cold call (about 5 to 10 seconds warm). If you are demoing a serverless app, warm it up with one request first. We wrote a small health-check script that does exactly that.


So what does it actually cost?

  • Idle: nothing is running, so nothing is billed. No GPU, no machine, no NAT gateway (we have a test for that too).
  • Per assessment: a few seconds of serverless inference, a few Lambda seconds and a dozen state transitions. That's a fraction of a cent.
  • Total so far: our AWS Budgets page, with credits excluded so it shows real usage, reads $0.00 at the time of writing. (Budgets updates a few times a day, so treat that as approximate.)
Two settings matter as much as the architecture: a budget alarm with credits excluded (otherwise free credits hide your real usage and the alarm never fires) and short log retention.


The end-to-end result

With everything warm, a full assessment (satellite fetch, model inference, a year of shadow simulation, panel layout and finance) completes in about 16 to 24 seconds. The very first one after a quiet spell takes up to two minutes while the containers wake up. And when the last user closes the tab, the whole system quietly goes back to costing nothing

If you are about to deploy a model on a student budget

  1. Check whether you need a GPU at all. If inference takes about a second on a CPU, serverless inference is probably your friend.
  1. Keep the image small: a slim base image plus CPU-only PyTorch, with the model loaded at startup.
  1. Let an orchestrator absorb cold starts. Retry "busy" errors with backoff and fail fast on real errors.
  1. Check your account's quotas before choosing sizes.
  1. Put guardrails in code: a test that forbids instance types, a budget alarm without credits, and log retention.
We used AI coding assistants (Claude Code and Codex) for scaffolding, debugging and tests. The design decisions and trade-offs were ours.
Built for Environmental Hacks by WeMakeDevs x AWS. Live app:
 https://main.da6yw5ko9zhmc.amplifyapp.com  (up while our AWS stack is running) ·
Architecture diagram

Code:  https://github.com/vedanttkhanna/Rest\_Azured 
# amazon-s3# api# aws-builders# aws-cloudwatch# aws-cdk
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article