AWS Builder Center

SageMaker Endpoint Autoscaling in Python: The Fast Way

A copy-pasteable SageMaker autoscaling tutorial in Python: register a scalable target, add a target-tracking policy, and use high-resolution metrics to scale GenAI endpoints up to 6x faster.

Series: Bedrock Builder's Log (6 articles)

  1. …
  2. 3
    SageMaker Endpoint Autoscaling in Python: The Fast Way This article
Lead Data Scientist

SageMaker Endpoint Autoscaling in Python: Do It the Fast Way

Most SageMaker autoscaling tutorials teach you the slow way and never tell you. They wire up  SageMakerVariantInvocationsPerInstance , call it done, and move on. That metric works fine for a scikit-learn model answering in 20 milliseconds. Point it at an LLM that takes 8 seconds per response and you'll watch your endpoint sit pinned at 100% while CloudWatch takes its sweet time deciding a scale-out is warranted — sometimes six minutes late, by which point your users have already rage-quit.
The fix is a metric most people don't know exists. Let's deploy an endpoint, attach autoscaling the ordinary way, and then upgrade it to the way you actually want for generative AI.

Prerequisites

You'll need an AWS account,  boto3  and the  sagemaker  SDK installed ( pip install sagemaker boto3 ), and an execution role with SageMaker permissions. Everything below runs from a notebook or a plain Python script.

Step 1: Deploy a real-time endpoint

We'll use a JumpStart model so there's no artifact packaging to fuss over. Swap in your own model whenever you like — the autoscaling half is identical regardless of what's behind the endpoint.
python
from sagemaker.jumpstart.model import JumpStartModel

model = JumpStartModel(model_id="huggingface-llm-mistral-7b-instruct")
predictor = model.deploy(
instance_type="ml.g5.2xlarge",
initial_instance_count=1,
endpoint_name="demo-autoscale-ep",
)
Cost note: an  ml.g5.2xlarge  runs a little over a dollar an hour in most regions, and you pay for every instance while the endpoint exists — not per request. That's exactly why autoscaling matters: idle instances are pure burn.

Step 2: The ordinary way (fine for classic ML)

Autoscaling lives in Application Auto Scaling, not SageMaker itself. Two calls: register the endpoint variant as a scalable target, then attach a policy.
python
import boto3

aas = boto3.client("application-autoscaling")
resource_id = "endpoint/demo-autoscale-ep/variant/AllTraffic"

aas.register_scalable_target(
ServiceNamespace="sagemaker",
ResourceId=resource_id,
ScalableDimension="sagemaker:variant:DesiredInstanceCount",
MinCapacity=1,
MaxCapacity=4,
)

aas.put_scaling_policy(
PolicyName="invocations-policy",
ServiceNamespace="sagemaker",
ResourceId=resource_id,
ScalableDimension="sagemaker:variant:DesiredInstanceCount",
PolicyType="TargetTrackingScaling",
TargetTrackingScalingPolicyConfiguration={
"TargetValue": 50.0, # avg invocations per instance per minute
"PredefinedMetricSpecification": {
"PredefinedMetricType": "SageMakerVariantInvocationsPerInstance"
},
"ScaleInCooldown": 300,
"ScaleOutCooldown": 60,
},
)
Target tracking behaves like a thermostat: it adds instances to pull the metric back toward  TargetValue  and removes them when demand falls. Set  ScaleInCooldown  longer than  ScaleOutCooldown  so you scale up eagerly and down cautiously — nobody wants to drop capacity right before the next spike.

Step 3: The fast way (what you want for GenAI)

Here's the part the tutorials skip. Invocation count is a bad proxy for LLM load, because one request can occupy a GPU for seconds. What you actually care about is concurrency — how many requests are in flight right now. SageMaker emits a high-resolution metric for exactly this, sampled every 10 seconds instead of once a minute.
python
aas.put_scaling_policy(
PolicyName="concurrency-policy",
ServiceNamespace="sagemaker",
ResourceId=resource_id,
ScalableDimension="sagemaker:variant:DesiredInstanceCount",
PolicyType="TargetTrackingScaling",
TargetTrackingScalingPolicyConfiguration={
"TargetValue": 5.0, # concurrent requests per model copy
"PredefinedMetricSpecification": {
"PredefinedMetricType":
"SageMakerVariantConcurrentRequestsPerModelHighResolution"
},
"ScaleInCooldown": 180,
"ScaleOutCooldown": 180,
},
)
Same two-call pattern, one different  PredefinedMetricType . AWS measured scale-out detection dropping from over six minutes to under 45 seconds on Llama-class models — roughly 6x faster to notice a spike.  TargetValue  here is concurrent requests per copy, not invocations per minute, so tune it to what one instance can comfortably keep in flight.
Even better, you can run both policies at once. Concurrency-based scaling reacts fast, and the invocation-based policy acts as a safety net: if a container crashes, in-flight concurrency can crater and trick a lone concurrency policy into scaling in at the worst moment. The invocation policy holds the line.

Step 4: Clean up (do not skip this)

An idle endpoint bills around the clock. Tear it down when you're done:
python
predictor.delete_model()
predictor.delete_endpoint()
Deregistering the scalable target is good hygiene too, via  aas.deregister_scalable_target(...)  with the same  ResourceId  and  ScalableDimension .

Why this matters / what I'd do

If you're serving classic, low-latency models, the invocation metric is genuinely fine — don't overthink it. But the moment there's an LLM behind the endpoint, default to  ...ConcurrentRequestsPerModelHighResolution  and run the invocation policy alongside it as a guardrail. The change is one string, and it's the difference between a five-minute scaling lag and a 45-second one. On a bursty workload, that lag is what decides whether you over-provision (and overpay) permanently, or scale on demand and sleep at night.
One caveat worth internalizing: autoscaling only adds instances. It can't rescue a cold start. If your container takes three minutes to pull a 14 GB model, no policy makes new capacity ready faster — so trim your image, cache artifacts, and keep  MinCapacity  at 1 rather than 0 for anything latency-sensitive.
Everything above — the deploy script, both scaling policies, a locust load test, and a one-command teardown — is in the companion repo:  aws-sagemaker-endpoint-autoscaling-demo . Clone it, point it at your account, and watch the scale-out difference for yourself.
What's your autoscaling  TargetValue  set to right now — and do you actually know if it's the concurrency metric or the invocation one? Drop it in the comments; I'd bet half of you are on the slow metric without realizing it.
#AWS #SageMaker #MachineLearning #MLOps #GenAI #Python

Series: Bedrock Builder's Log (6 articles)

  1. …
  2. 3
    SageMaker Endpoint Autoscaling in Python: The Fast Way This article
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article