
Optimize LLM Inference with vLLM KV Cache Offloading on Amazon SageMaker AI
Learn how to improve LLM inference performance for long-context workloads like multi-turn conversations and RAG applications by deploying vLLM with KV cache offloading on Amazon SageMaker AI. This guide covers both the native vLLM connector and LMCache, with step-by-step deployment examples and configuration options for CPU and local disk based offloading.
Introduction
When deploying Large Language Models (LLMs) with vLLM on Amazon SageMaker AI, the Key-Value (KV) cache can become a significant bottleneck for long-context scenarios like multi-round conversations or Retrieval-Augmented Generation (RAG).
In this post, you'll learn how to configure KV cache offloading on Amazon SageMaker AI using vLLM. We'll cover both the native vLLM connector and the LMCache connector, show you how to deploy each option, and help you choose the right approach for your workload.
What is KV cache?
During LLM inference, the attention mechanism computes key and value tensors for each token in the sequence. KV caching stores these tensors in GPU memory so they do not need to be recomputed for previously processed tokens. This optimization significantly accelerates generation by avoiding redundant computation.
Why KV cache offloading?
For long contexts (thousands of tokens), the KV cache can consume substantial GPU memory, sometimes more than the model weights themselves. Each active request stores its KV cache in GPU memory, and for long contexts this can be multiple GBs per request. This memory consumption creates critical constraints in production deployments. vLLM uses continuous batching to maximize GPU utilization, processing multiple requests concurrently. However, memory constraints fundamentally limit how many requests can be batched together. Conceptually, the maximum concurrent requests is bounded by available GPU memory divided by the KV cache requirement per request. The actual calculation involves additional factors like PagedAttention blocks, activations, and memory fragmentation. This memory bottleneck directly impacts throughput. Fewer concurrent requests means lower GPU utilization and reduced tokens per second. The performance impact is substantial. Benchmarks demonstrate that KV cache offloading can deliver up to 9× throughput increases by addressing these memory bottlenecks, so you can achieve higher concurrency and better GPU utilization. KV cache offloading solves this by moving KV cache data from GPU memory to alternative storage tiers (CPU RAM, NVMe, S3, Redis), so you can achieve higher concurrency and better resource utilization without requiring additional GPU infrastructure.
Data Flow
GPU → KV Cache → [CPU Memory | Local Storage | Remote Storage] ↑ ↓ └────────── KV Cache Retrieval ──────┘
Solution overview
This guide demonstrates how to deploy vLLM with KV cache offloading on Amazon SageMaker AI. You'll configure one of two connector options:
- Native vLLM connector: CPU offloading using CUDA DMA, included in AWS Deep Learning Containers
- LMCache connector: Advanced connector with multiple storage backends (CPU, Local disk, S3, Redis,...) and enhanced optimizations
Both options reduce GPU memory pressure and increase throughput for long-context workloads. You'll use the AWS Python SDK to deploy endpoints with the appropriate configuration for your use case.
Configure Amazon SageMaker AI vLLM deployment with KV cache offloading
KV cache CPU offloading with native vLLM connector
vLLM introduced a KV Offloading Connector framework with a pluggable architecture that so you can choose different implementations. Starting with vLLM v0.11.0 (released October 2, 2025), vLLM includes a native
OffloadingConnector that provides asynchronous KV cache offloading to CPU memory using CUDA DMA. The native connector is advertised by the vLLM team to achieve 2-22× Time-To-First-Token (TTFT) reduction and up to 9× throughput increases with high cache hit rates under concurrent load. Simple setup with no additional dependencies required.LMCache: Advanced KV cache connector
LMCache provides KV cache offloading for vLLM with additional capabilities beyond the native connector. You can use LMCache's connector implementation (
LMCacheConnectorV1 ) instead of vLLM's native connector. LMCache Connector comes with additional optimizations (chunking, layer-wise pipelining) and multiple storage backends (CPU, NVMe, S3, Redis, ...) LMCache research reports 1.9-8.1× TTFT reduction and 2.3-14× throughput increases for long-context workloads compared to baseline vLLM, with approximately 50% cache hit ratios in production RAG and multi-turn conversation scenarios.Prerequisites
Before you begin, ensure you have the following:
- AWS Account with permissions to:
- Create and manage Amazon SageMaker endpoints (Create models, endpoint configurations, endpoints InvokeEndpoint)
- Create and pass IAM role to SageMaker as execution role
- Create Amazon ECR repositories and push Docker image (only needed for LMCache option)
- AWS CLI configured with appropriate credentials
- Docker installed for building custom container images (only needed for LMCache option)
- Python 3.12+ with the following packages:
-
boto3v1.42.2+
-
sagemakerv3.0.0+ Python SDK
- Hugging Face Token with access to gated models:
- Create a token at huggingface.co/settings/tokens
- Accept the Meta Llama license at huggingface.co/meta-llama/Llama-3.1-8B-Instruct
Note: The resources created in this blog post (Amazon SageMaker endpoints and Amazon ECR repositories) incur charges to your AWS account. This example shows usage of a ml.g6e.2xlarge, which has a hourly cost of $2.80 in us-east-1. Remember to delete your endpoint when done testing (see the Cleanup section).
Deploying to Amazon SageMaker
Deploy with vLLM native connector
The native connector provides asynchronous CPU offloading. No additional setup required.
import boto3
# Initialize SageMaker clientsession = boto3.Session()sm_client = session.client(service_name='sagemaker')region = session.region_name
# Configuration# See https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-roles.html for role setupexecution_role = "arn:aws:iam::YOUR_ACCOUNT_ID:role/YOUR_ROLE_NAME"
# You can find the latest containers at the https://aws.github.io/deep-learning-containers/.image_uri = f'763104351884.dkr.ecr.{region}.amazonaws.com/vllm:0.15.1-gpu-py312-cu129-ubuntu22.04-sagemaker'
# Simple configuration - native offloading enabled by defaultenv = { "SM_VLLM_MODEL": "meta-llama/Llama-3.1-8B-Instruct", # Need to have accepted Meta EULA on HuggingFace "HF_TOKEN": "your-huggingface-token", "SM_VLLM_KV_OFFLOADING_SIZE": "10", # GiB - activates KV offloading "SM_VLLM_DISABLE_HYBRID_KV_CACHE_MANAGER": "True", }
# Create modelmodel_name = "vllm-kvcache-native-model"create_model_response = sm_client.create_model( ModelName=model_name, ExecutionRoleArn=execution_role, PrimaryContainer={ 'Image': image_uri, 'Environment': env })
# Create endpoint configurationendpoint_config_name = "vllm-kvcache-native-config"sm_client.create_endpoint_config( EndpointConfigName=endpoint_config_name, ProductionVariants=[{ 'VariantName': 'AllTraffic', 'ModelName': model_name, 'InitialInstanceCount': 1, 'InstanceType': 'ml.g6e.2xlarge' }])
# Create endpointendpoint_name = "vllm-kvcache-native-endpoint"sm_client.create_endpoint( EndpointName=endpoint_name, EndpointConfigName=endpoint_config_name)
print(f"Creating endpoint {endpoint_name}. This will take several minutes...")
Note on required flags: The native connector requires disabling Hybrid KV Cache Manager. This is due to a known limitation in vLLM v0.15.1 where theOffloadingConnectordoes not yet support the Hybrid Memory Allocator (HMA). A bug report has been filed ( vllm#33572 ). Disabling these features may reduce performance for models with sliding window attention or Mamba attention.
Use this option for: CPU-only offloading, simple setup without additional packages, but less mature than LMCache.
Deploy with LMCache connector
This example demonstrates KVCache offloading with LMCache. LMCache supports multiple backends including CPU memory, local disk, S3 and more.
Build custom container with LMCache
First, create a Dockerfile that extends the AWS Deep Learning Container:
# Create Dockerfile$ cat > Dockerfile <<EOFFROM public.ecr.aws/deep-learning-containers/vllm:0.15-gpu-py312RUN pip install --no-cache-dir lmcacheEOF
Build and push to Amazon ECR:
# Set your variables$ REGION=us-east-1$ ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)$ REPO_NAME=vllm-lmcache$ IMAGE_TAG=latest
# Create ECR repository (if needed)$ aws ecr create-repository --repository-name ${REPO_NAME} --region ${REGION} || true
# Build and push$ aws ecr get-login-password --region ${REGION} | docker login --username AWS --password-stdin ${ACCOUNT_ID}.dkr.ecr.${REGION}.amazonaws.com$ docker build -t ${REPO_NAME}:${IMAGE_TAG} -f Dockerfile .$ docker tag ${REPO_NAME}:${IMAGE_TAG} ${ACCOUNT_ID}.dkr.ecr.${REGION}.amazonaws.com/${REPO_NAME}:${IMAGE_TAG}$ docker push ${ACCOUNT_ID}.dkr.ecr.${REGION}.amazonaws.com/${REPO_NAME}:${IMAGE_TAG}Set your HuggingFace Token:
$ export HF_TOKEN="your-huggingface-token"Deploy to Amazon SageMaker:
This example deploy a CPU RAM KVCache Offloading, similar to the native vLLM implementation. If you want to configure disk based offloading, see Disk section below.
CPU
import boto3import os
# Initialize SageMaker clientsession = boto3.Session()sm_client = session.client(service_name='sagemaker')region = session.region_name
# Configuration# See https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-roles.html for role setupexecution_role = "arn:aws:iam::YOUR_ACCOUNT_ID:role/YOUR_ROLE_NAME"hugging_face_token = os.environ["HF_TOKEN"]
# Use custom image with LMCache installedaccount_id = boto3.client("sts").get_caller_identity()["Account"]image_uri = f"{account_id}.dkr.ecr.{region}.amazonaws.com/vllm-lmcache:latest"
env = { "SM_VLLM_MODEL": "meta-llama/Llama-3.1-8B-Instruct", "HF_TOKEN": hugging_face_token,
# Enable LMCache connector "SM_VLLM_KV_OFFLOADING_SIZE": "10", "SM_VLLM_KV_OFFLOADING_BACKEND": "lmcache", "SM_VLLM_KV_TRANSFER_CONFIG": '{"kv_connector":"LMCacheConnectorV1","kv_role":"kv_both"}', "SM_VLLM_COMPILATION_CONFIG": '{"cudagraph_mode":"PIECEWISE"}',
# LMCache storage (CPU backend) "LMCACHE_LOCAL_CPU": "True", "LMCACHE_MAX_LOCAL_CPU_SIZE": "10", "LMCACHE_CHUNK_SIZE": "256", "LMCACHE_USE_LAYERWISE": "True",}
# Create modelmodel_name = "vllm-lmcache-model"create_model_response = sm_client.create_model( ModelName=model_name, ExecutionRoleArn=execution_role, PrimaryContainer={ 'Image': image_uri, 'Environment': env })
# Create endpoint configurationendpoint_config_name = "vllm-lmcache-config"sm_client.create_endpoint_config( EndpointConfigName=endpoint_config_name, ProductionVariants=[{ 'VariantName': 'AllTraffic', 'ModelName': model_name, 'InitialInstanceCount': 1, 'InstanceType': 'ml.g6e.2xlarge' }])
# Create endpointendpoint_name = "vllm-lmcache-endpoint"sm_client.create_endpoint( EndpointName=endpoint_name, EndpointConfigName=endpoint_config_name)
print(f"Creating endpoint {endpoint_name}. This will take several minutes...")Local disk
AWS GPU-enabled instances offer local NVME-based storage. You can leverage this storage to store KV cache with LMCache. Replace the env variable above with this configuration
...
env = { "SM_VLLM_MODEL": "meta-llama/Llama-3.1-8B-Instruct", "HF_TOKEN": "your-huggingface-token",
# Enable LMCache connector "SM_VLLM_KV_OFFLOADING_SIZE": "10", "SM_VLLM_KV_OFFLOADING_BACKEND": "lmcache", "SM_VLLM_KV_TRANSFER_CONFIG": '{"kv_connector":"LMCacheConnectorV1","kv_role":"kv_both"}', "SM_VLLM_COMPILATION_CONFIG": '{"cudagraph_mode":"PIECEWISE"}',
# LMCache storage (Disk backend - uses local NVMe storage) "LMCACHE_LOCAL_CPU": "False", "LMCACHE_LOCAL_DISK": "file:///tmp/lmcache", "LMCACHE_USE_EXPERIMENTAL": "True", "LMCACHE_MAX_LOCAL_DISK_SIZE": "50",}
...Testing your endpoint
Verify that your endpoint is working:
import boto3import json
runtime = boto3.client("sagemaker-runtime")
payload = { "model": "meta-llama/Llama-3.1-8B-Instruct", "messages": [ {"role": "user", "content": "Explain KV cache offloading in one sentence."} ], "max_tokens": 100}
response = runtime.invoke_endpoint( EndpointName="vllm-lmcache-endpoint", ContentType="application/json", Body=json.dumps(payload))
result = json.loads(response["Body"].read())print(result["choices"][0]["message"]["content"])To observe KV cache offloading benefits, test with workloads that demonstrate cache reuse: multi-turn conversations with long context, concurrent requests sharing document prefixes (RAG scenarios), or repeated queries with common system prompts. Monitor Amazon CloudWatch metrics (ModelLatency, Invocations) to validate performance improvements for cache hits. For comprehensive benchmarking methodologies and performance testing scenarios, see the LMCache benchmarking guide .
Cleanup
Delete your endpoint when you finish testing:
import boto3
sm_client = boto3.client('sagemaker')
# Delete endpointendpoint_name = "vllm-lmcache-endpoint" # or "vllm-kvcache-native-endpoint"sm_client.delete_endpoint(EndpointName=endpoint_name)
# Delete endpoint configurationendpoint_config_name = "vllm-lmcache-config" # or "vllm-kvcache-native-config"sm_client.delete_endpoint_config(EndpointConfigName=endpoint_config_name)
# Delete modelmodel_name = "vllm-lmcache-model" # or "vllm-kvcache-native-model"sm_client.delete_model(ModelName=model_name)
print(f"Deleted endpoint {endpoint_name}, endpoint config, and model")Configuration options
To get deeper into the configuration and customize your deployment for your own need, you need to understand the configuration layers.
How Amazon SageMaker configure vLLM
The AWS Deep Learning Container for vLLM uses an entrypoint script that converts environment variables to CLI arguments. Environment variables with the
SM_VLLM_ prefix are automatically converted. The prefix is stripped, converted to lowercase, and underscores become hyphens.Example:
SM_VLLM_MAX_MODEL_LEN=4096 → --max-model-len 4096 This means you configure vLLM entirely through Amazon SageMaker environment variables, which are then passed as CLI arguments to the vLLM server.
Understanding the configuration
- Activate KV Offloading (
SM_VLLM_KV_OFFLOADING_SIZE): Set the offloading buffer size to enable KV offloading
- Example:
"SM_VLLM_KV_OFFLOADING_SIZE": "10"(10 GiB buffer)
- Connector Selection (
SM_VLLM_KV_OFFLOADING_BACKEND): Choose which backend implementation to use
-
"native"- vLLM's native connector (built-in since v0.11.0)
-
"lmcache"- LMCache connector (requirespip install lmcache)
- LMCache Configuration
-
LMCACHE_*environment variables for LMCache configuration
Conclusion
KV cache offloading on Amazon SageMaker provides significant performance improvements for LLM inference workloads, particularly those with long contexts.vLLM offers two connector options:
- Native
OffloadingConnector(v0.11.0+): Built-in CPU offloading using CUDA DMA with simple setup and no additional dependencies.
- LMCache
LMCacheConnectorV1: Extended connector with enhanced optimizations (chunking, layer-wise pipelining) and multiple storage backends (CPU, NVMe, S3).
Both connectors enable cross-request cache reuse. Choose the native connector for simple CPU-only deployments, or LMCache for multi-backend support, distributed caching, and advanced optimizations. Either option can help optimize performance when running LLM workloads on Amazon SageMaker.
Additional resources
- vLLM KV Offloading Connector - Technical documentation covering the native cLLM connector implementation
- LMCache: Enterprise-scale KV cache management -** Academic research detailing LMCache architecture and optimizations techniques
- LMCache GitHub repository - Configuration options and examples
- AWS Deep Learning Containers - Latest container versions
- Amazon SageMaker Large Model Inference - Performance optimization guide
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article