AWS Builder Center
Optimize LLM Inference with vLLM KV Cache Offloading on Amazon SageMaker AI

Optimize LLM Inference with vLLM KV Cache Offloading on Amazon SageMaker AI

Learn how to improve LLM inference performance for long-context workloads like multi-turn conversations and RAG applications by deploying vLLM with KV cache offloading on Amazon SageMaker AI. This guide covers both the native vLLM connector and LMCache, with step-by-step deployment examples and configuration options for CPU and local disk based offloading.

Frontier AI Solutions Architect

Introduction

When deploying Large Language Models (LLMs) with vLLM on Amazon SageMaker AI, the Key-Value (KV) cache can become a significant bottleneck for long-context scenarios like multi-round conversations or Retrieval-Augmented Generation (RAG).
In this post, you'll learn how to configure KV cache offloading on Amazon SageMaker AI using vLLM. We'll cover both the native vLLM connector and the LMCache connector, show you how to deploy each option, and help you choose the right approach for your workload.

What is KV cache?

During LLM inference, the attention mechanism computes key and value tensors for each token in the sequence. KV caching stores these tensors in GPU memory so they do not need to be recomputed for previously processed tokens. This optimization significantly accelerates generation by avoiding redundant computation.

Why KV cache offloading?

For long contexts (thousands of tokens), the KV cache can consume substantial GPU memory, sometimes more than the model weights themselves. Each active request stores its KV cache in GPU memory, and for long contexts this can be multiple GBs per request. This memory consumption creates critical constraints in production deployments. vLLM uses continuous batching to maximize GPU utilization, processing multiple requests concurrently. However, memory constraints fundamentally limit how many requests can be batched together. Conceptually, the maximum concurrent requests is bounded by available GPU memory divided by the KV cache requirement per request. The actual calculation involves additional factors like PagedAttention blocks, activations, and memory fragmentation. This memory bottleneck directly impacts throughput. Fewer concurrent requests means lower GPU utilization and reduced tokens per second. The performance impact is substantial. Benchmarks demonstrate that KV cache offloading can deliver up to 9× throughput increases by addressing these memory bottlenecks, so you can achieve higher concurrency and better GPU utilization. KV cache offloading solves this by moving KV cache data from GPU memory to alternative storage tiers (CPU RAM, NVMe, S3, Redis), so you can achieve higher concurrency and better resource utilization without requiring additional GPU infrastructure.

Data Flow

GPU → KV Cache → [CPU Memory | Local Storage | Remote Storage]
↑ ↓
└────────── KV Cache Retrieval ──────┘


Solution overview

This guide demonstrates how to deploy vLLM with KV cache offloading on Amazon SageMaker AI. You'll configure one of two connector options:
  • Native vLLM connector: CPU offloading using CUDA DMA, included in AWS Deep Learning Containers
  • LMCache connector: Advanced connector with multiple storage backends (CPU, Local disk, S3, Redis,...) and enhanced optimizations
Both options reduce GPU memory pressure and increase throughput for long-context workloads. You'll use the AWS Python SDK to deploy endpoints with the appropriate configuration for your use case.

Configure Amazon SageMaker AI vLLM deployment with KV cache offloading

KV cache CPU offloading with native vLLM connector

vLLM introduced a KV Offloading Connector framework with a pluggable architecture that so you can choose different implementations. Starting with vLLM v0.11.0 (released October 2, 2025), vLLM includes a native  OffloadingConnector  that provides asynchronous KV cache offloading to CPU memory using CUDA DMA. The native connector is advertised by the vLLM team to achieve 2-22× Time-To-First-Token (TTFT) reduction and up to 9× throughput increases with high cache hit rates under concurrent load. Simple setup with no additional dependencies required.

LMCache: Advanced KV cache connector

LMCache provides KV cache offloading for vLLM with additional capabilities beyond the native connector. You can use LMCache's connector implementation ( LMCacheConnectorV1 ) instead of vLLM's native connector. LMCache Connector comes with additional optimizations (chunking, layer-wise pipelining) and multiple storage backends (CPU, NVMe, S3, Redis, ...) LMCache research reports 1.9-8.1× TTFT reduction and 2.3-14× throughput increases for long-context workloads compared to baseline vLLM, with approximately 50% cache hit ratios in production RAG and multi-turn conversation scenarios.

Prerequisites

Before you begin, ensure you have the following:
  • AWS Account with permissions to:
  • Create and manage Amazon SageMaker endpoints (Create models, endpoint configurations, endpoints InvokeEndpoint)
  • Create and pass IAM role to SageMaker as execution role
  • Create Amazon ECR repositories and push Docker image (only needed for LMCache option)
  • AWS CLI configured with appropriate credentials
  • Docker installed for building custom container images (only needed for LMCache option)
  • Python 3.12+ with the following packages:
  •  boto3  v1.42.2+
  •  sagemaker  v3.0.0+ Python SDK
  • Hugging Face Token with access to gated models:
Note: The resources created in this blog post (Amazon SageMaker endpoints and Amazon ECR repositories) incur charges to your AWS account. This example shows usage of a ml.g6e.2xlarge, which has a hourly cost of $2.80 in us-east-1. Remember to delete your endpoint when done testing (see the Cleanup section).

Deploying to Amazon SageMaker

Deploy with vLLM native connector

The native connector provides asynchronous CPU offloading. No additional setup required.
import boto3

# Initialize SageMaker client
session = boto3.Session()
sm_client = session.client(service_name='sagemaker')
region = session.region_name

# Configuration
# See https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-roles.html for role setup
execution_role = "arn:aws:iam::YOUR_ACCOUNT_ID:role/YOUR_ROLE_NAME"

# You can find the latest containers at the https://aws.github.io/deep-learning-containers/.
image_uri = f'763104351884.dkr.ecr.{region}.amazonaws.com/vllm:0.15.1-gpu-py312-cu129-ubuntu22.04-sagemaker'

# Simple configuration - native offloading enabled by default
env = {
"SM_VLLM_MODEL": "meta-llama/Llama-3.1-8B-Instruct", # Need to have accepted Meta EULA on HuggingFace
"HF_TOKEN": "your-huggingface-token",
"SM_VLLM_KV_OFFLOADING_SIZE": "10", # GiB - activates KV offloading
"SM_VLLM_DISABLE_HYBRID_KV_CACHE_MANAGER": "True",
}

# Create model
model_name = "vllm-kvcache-native-model"
create_model_response = sm_client.create_model(
ModelName=model_name,
ExecutionRoleArn=execution_role,
PrimaryContainer={
'Image': image_uri,
'Environment': env
}
)

# Create endpoint configuration
endpoint_config_name = "vllm-kvcache-native-config"
sm_client.create_endpoint_config(
EndpointConfigName=endpoint_config_name,
ProductionVariants=[{
'VariantName': 'AllTraffic',
'ModelName': model_name,
'InitialInstanceCount': 1,
'InstanceType': 'ml.g6e.2xlarge'
}]
)

# Create endpoint
endpoint_name = "vllm-kvcache-native-endpoint"
sm_client.create_endpoint(
EndpointName=endpoint_name,
EndpointConfigName=endpoint_config_name
)

print(f"Creating endpoint {endpoint_name}. This will take several minutes...")

Note on required flags: The native connector requires disabling Hybrid KV Cache Manager. This is due to a known limitation in vLLM v0.15.1 where the  OffloadingConnector  does not yet support the Hybrid Memory Allocator (HMA). A bug report has been filed ( vllm#33572 ). Disabling these features may reduce performance for models with sliding window attention or Mamba attention.
Use this option for: CPU-only offloading, simple setup without additional packages, but less mature than LMCache.

Deploy with LMCache connector

This example demonstrates KVCache offloading with LMCache. LMCache supports  multiple backends  including CPU memory, local disk, S3 and more.

Build custom container with LMCache

First, create a Dockerfile that extends the AWS Deep Learning Container:
# Create Dockerfile
$ cat > Dockerfile <<EOF
FROM public.ecr.aws/deep-learning-containers/vllm:0.15-gpu-py312
RUN pip install --no-cache-dir lmcache
EOF


Build and push to Amazon ECR:

# Set your variables
$ REGION=us-east-1
$ ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
$ REPO_NAME=vllm-lmcache
$ IMAGE_TAG=latest

# Create ECR repository (if needed)
$ aws ecr create-repository --repository-name ${REPO_NAME} --region ${REGION} || true

# Build and push
$ aws ecr get-login-password --region ${REGION} | docker login --username AWS --password-stdin ${ACCOUNT_ID}.dkr.ecr.${REGION}.amazonaws.com
$ docker build -t ${REPO_NAME}:${IMAGE_TAG} -f Dockerfile .
$ docker tag ${REPO_NAME}:${IMAGE_TAG} ${ACCOUNT_ID}.dkr.ecr.${REGION}.amazonaws.com/${REPO_NAME}:${IMAGE_TAG}
$ docker push ${ACCOUNT_ID}.dkr.ecr.${REGION}.amazonaws.com/${REPO_NAME}:${IMAGE_TAG}

Set your HuggingFace Token:

$ export HF_TOKEN="your-huggingface-token"

Deploy to Amazon SageMaker:

This example deploy a CPU RAM KVCache Offloading, similar to the native vLLM implementation. If you want to configure disk based offloading, see Disk section below.
CPU
import boto3
import os

# Initialize SageMaker client
session = boto3.Session()
sm_client = session.client(service_name='sagemaker')
region = session.region_name

# Configuration
# See https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-roles.html for role setup
execution_role = "arn:aws:iam::YOUR_ACCOUNT_ID:role/YOUR_ROLE_NAME"
hugging_face_token = os.environ["HF_TOKEN"]

# Use custom image with LMCache installed
account_id = boto3.client("sts").get_caller_identity()["Account"]
image_uri = f"{account_id}.dkr.ecr.{region}.amazonaws.com/vllm-lmcache:latest"

env = {
"SM_VLLM_MODEL": "meta-llama/Llama-3.1-8B-Instruct",
"HF_TOKEN": hugging_face_token,

# Enable LMCache connector
"SM_VLLM_KV_OFFLOADING_SIZE": "10",
"SM_VLLM_KV_OFFLOADING_BACKEND": "lmcache",
"SM_VLLM_KV_TRANSFER_CONFIG": '{"kv_connector":"LMCacheConnectorV1","kv_role":"kv_both"}',
"SM_VLLM_COMPILATION_CONFIG": '{"cudagraph_mode":"PIECEWISE"}',

# LMCache storage (CPU backend)
"LMCACHE_LOCAL_CPU": "True",
"LMCACHE_MAX_LOCAL_CPU_SIZE": "10",
"LMCACHE_CHUNK_SIZE": "256",
"LMCACHE_USE_LAYERWISE": "True",
}

# Create model
model_name = "vllm-lmcache-model"
create_model_response = sm_client.create_model(
ModelName=model_name,
ExecutionRoleArn=execution_role,
PrimaryContainer={
'Image': image_uri,
'Environment': env
}
)

# Create endpoint configuration
endpoint_config_name = "vllm-lmcache-config"
sm_client.create_endpoint_config(
EndpointConfigName=endpoint_config_name,
ProductionVariants=[{
'VariantName': 'AllTraffic',
'ModelName': model_name,
'InitialInstanceCount': 1,
'InstanceType': 'ml.g6e.2xlarge'
}]
)

# Create endpoint
endpoint_name = "vllm-lmcache-endpoint"
sm_client.create_endpoint(
EndpointName=endpoint_name,
EndpointConfigName=endpoint_config_name
)

print(f"Creating endpoint {endpoint_name}. This will take several minutes...")
Local disk
AWS GPU-enabled instances offer local NVME-based storage. You can leverage this storage to store KV cache with LMCache. Replace the env variable above with this configuration
...

env = {
"SM_VLLM_MODEL": "meta-llama/Llama-3.1-8B-Instruct",
"HF_TOKEN": "your-huggingface-token",

# Enable LMCache connector
"SM_VLLM_KV_OFFLOADING_SIZE": "10",
"SM_VLLM_KV_OFFLOADING_BACKEND": "lmcache",
"SM_VLLM_KV_TRANSFER_CONFIG": '{"kv_connector":"LMCacheConnectorV1","kv_role":"kv_both"}',
"SM_VLLM_COMPILATION_CONFIG": '{"cudagraph_mode":"PIECEWISE"}',

# LMCache storage (Disk backend - uses local NVMe storage)
"LMCACHE_LOCAL_CPU": "False",
"LMCACHE_LOCAL_DISK": "file:///tmp/lmcache",
"LMCACHE_USE_EXPERIMENTAL": "True",
"LMCACHE_MAX_LOCAL_DISK_SIZE": "50",
}

...

Testing your endpoint

Verify that your endpoint is working:
import boto3
import json

runtime = boto3.client("sagemaker-runtime")

payload = {
"model": "meta-llama/Llama-3.1-8B-Instruct",
"messages": [
{"role": "user", "content": "Explain KV cache offloading in one sentence."}
],
"max_tokens": 100
}

response = runtime.invoke_endpoint(
EndpointName="vllm-lmcache-endpoint",
ContentType="application/json",
Body=json.dumps(payload)
)

result = json.loads(response["Body"].read())
print(result["choices"][0]["message"]["content"])
To observe KV cache offloading benefits, test with workloads that demonstrate cache reuse: multi-turn conversations with long context, concurrent requests sharing document prefixes (RAG scenarios), or repeated queries with common system prompts. Monitor Amazon CloudWatch metrics (ModelLatency, Invocations) to validate performance improvements for cache hits. For comprehensive benchmarking methodologies and performance testing scenarios, see the  LMCache benchmarking guide .

Cleanup

Delete your endpoint when you finish testing:
import boto3

sm_client = boto3.client('sagemaker')

# Delete endpoint
endpoint_name = "vllm-lmcache-endpoint" # or "vllm-kvcache-native-endpoint"
sm_client.delete_endpoint(EndpointName=endpoint_name)

# Delete endpoint configuration
endpoint_config_name = "vllm-lmcache-config" # or "vllm-kvcache-native-config"
sm_client.delete_endpoint_config(EndpointConfigName=endpoint_config_name)

# Delete model
model_name = "vllm-lmcache-model" # or "vllm-kvcache-native-model"
sm_client.delete_model(ModelName=model_name)

print(f"Deleted endpoint {endpoint_name}, endpoint config, and model")

Configuration options

To get deeper into the configuration and customize your deployment for your own need, you need to understand the configuration layers.

How Amazon SageMaker configure vLLM

The AWS Deep Learning Container for vLLM uses an entrypoint script that converts environment variables to CLI arguments. Environment variables with the  SM_VLLM_  prefix are automatically converted. The prefix is stripped, converted to lowercase, and underscores become hyphens.
Example:  SM_VLLM_MAX_MODEL_LEN=4096  →  --max-model-len 4096 
This means you configure vLLM entirely through Amazon SageMaker environment variables, which are then passed as CLI arguments to the vLLM server.

Understanding the configuration

  • Activate KV Offloading ( SM_VLLM_KV_OFFLOADING_SIZE ): Set the offloading buffer size to enable KV offloading
  • Example:  "SM_VLLM_KV_OFFLOADING_SIZE": "10"  (10 GiB buffer)
  • Connector Selection ( SM_VLLM_KV_OFFLOADING_BACKEND ): Choose which backend implementation to use
  •  "native"  - vLLM's native connector (built-in since v0.11.0)
  •  "lmcache"  - LMCache connector (requires  pip install lmcache )
  • LMCache Configuration
  •  LMCACHE_*  environment variables for LMCache configuration

Conclusion

KV cache offloading on Amazon SageMaker provides significant performance improvements for LLM inference workloads, particularly those with long contexts.vLLM offers two connector options:
  • Native  OffloadingConnector  (v0.11.0+): Built-in CPU offloading using CUDA DMA with simple setup and no additional dependencies.
  • LMCache  LMCacheConnectorV1 : Extended connector with enhanced optimizations (chunking, layer-wise pipelining) and multiple storage backends (CPU, NVMe, S3).
Both connectors enable cross-request cache reuse. Choose the native connector for simple CPU-only deployments, or LMCache for multi-backend support, distributed caching, and advanced optimizations. Either option can help optimize performance when running LLM workloads on Amazon SageMaker.

Additional resources

Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article