Exploring Tiered Vector Storage with Amazon S3 Vectors
Assessing the potential for tiered vector storage — hot and cold — using Amazon S3 Vectors.
Vector databases are quickly becoming the backbone of modern AI applications, powering everything from Retrieval-Augmented Generation (RAG) to sophisticated semantic search. The common approach is to load all our embeddings into a single, high-performance "hot" store like OpenSearch, Pinecone, Weaviate, etc. These services offer fantastic low-latency search, but they come with a hidden cost: storing billions of vectors in memory indefinitely can get expensive, fast.
Ever since AWS announced the preview of its S3 Vectors service in mid-July, I've been curious. This new capability, combined with the reality of ever-growing vector storage costs, sparked a question in my mind: what if we could treat vectors just like our other data? We already have hot and cold tiers for object storage and data warehouses. Could a tiered hot + cold model work for vector embeddings, too?
That's what this post is all about—an exploration into a tiered vector storage architecture using the new (and still in preview!) AWS S3 Vectors service.
Why Even Think About Tiered Vector Storage?
In most applications, not all data is created equal, and vectors are no exception. The access patterns are rarely uniform.
● Hot Vectors : These are the embeddings your application queries constantly. Think of active support cases, trending news articles, or documents from the last 30 days. You need sub-second responses for these.
● Cold Vectors : This is the long tail of your data—historical embeddings that are rarely accessed but must be retained. Think of closed case files, archived research papers, or user activity from years ago.
Today, many of us just keep everything in hot storage. But as our applications generate millions of new embeddings each week, this approach doesn't scale cost-effectively. A cold storage layer, like Amazon S3 Vectors, offers a durable and incredibly cheap place to offload these less-frequently used embeddings.
The core idea is simple:
- Keep fresh, active embeddings in a high-speed vector database (like OpenSearch).
- Automate moving older or less relevant embeddings to S3 on a schedule.
- If a query needs to span across historical data, pull the "cold" vectors back into the "hot" tier on-demand.
When Does a Hybrid Approach Make Sense?
Imagine a streaming platform that generates embeddings for every show in its catalog—dialogue, scene descriptions, and metadata. Most user searches will target new releases or trending content. But what if a film student wants to search for a specific theme across all black-and-white films from the 1940s?
This is a great use case for a tiered model.
| Tier | Content | Query Pattern | Storage Example |
|---|---|---|---|
| Hot | Recent or trending titles | High frequency, sub-second response required | OpenSearch, Pinecone |
| Cold | Older catalog, archival content | Occasional, latency-tolerant | S3 Vectors (very low cost) |
This separation brings powerful benefits:
- Cost-Efficiency: You stop paying premium prices to keep your entire history of embeddings in a high-performance database.
- Complete Recall: Archiving to S3 Vectors ensures you never lose search coverage over your historical data.
- Flexibility: You can "reheat" old content by moving it back to the hot tier if it suddenly becomes popular again.
Sketching a Conceptual Architecture
To aid understanding of the data flow, here is a visual representation of the concept before we look at the code snippets. This isn't a rigid blueprint but a starting point for our experiment.
- Vector Querying: An AI application, like a RAG pipeline built on Amazon Bedrock, primarily queries the Hot Tier (OpenSearch) for fast results on recent data.
- Periodic Migration: A "Mover" function (e.g., an AWS Lambda) runs on a schedule. It identifies "cold" vectors in OpenSearch based on age or business rules and moves them to the Cold Tier (S3 Vectors).
- Rehydration: If a query requires searching historical data, a mechanism can be triggered to find the relevant vectors in S3 and load them back into the hot tier for querying.
Putting the Concept to the Test
To test this idea, I set up a small proof-of-concept. The goal wasn't to build a production-ready system but to verify the core mechanics of moving vectors from a hot tier to a cold one.
- Hot Tier: An Amazon OpenSearch cluster storing recent embeddings.
- Cold Tier: An S3 bucket configured with an S3 Vectors index.
- Mover Lambda: A Python function that periodically migrates vectors from OpenSearch to S3 Vectors.
A quick but important reminder: The AWS S3 Vectors service is in preview. The APIs and features are subject to change. This experiment was for learning and exploration, not for production deployment!
The "Mover" Lambda Snippet
Here is the heart of the experiment—the Python code for the Lambda function that moves vectors from OpenSearch (hot) to S3 Vectors (cold). It scans the OpenSearch index, formats the data, and uses the boto3 client to write to S3 Vectors.
As an example I have few vector embedding of a media file stored in AWS OpenSearch within the index media_scenes.
Code snippet:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
def handler(event, context):
"""Handler to fetch vectors from OpenSearch and upload them to S3 Vector Store."""
batch_size = int(os.environ.get("BATCH_SIZE","500"))
vec_field = os.environ["OS_VECTOR_FIELD"]
bucket = os.environ["VECTOR_BUCKET"]
index = os.environ["VECTOR_INDEX"]
moved = 0
last_id = None
while True:
resp = _os_search(after=last_id, size=batch_size)
hits = resp.get("hits", {}).get("hits", [])
if not hits:
print("[INFO] No more documents found in OpenSearch.")
break
print(f"[INFO] Retrieved {len(hits)} documents from OpenSearch.")
vectors = []
for h in hits:
doc_id = h["_id"]
source = h.get("_source", {})
vec = source.get(vec_field)
if not isinstance(vec, list):
continue
meta = {}
if "metadata" in source and isinstance(source["metadata"], dict):
for k, v in source["metadata"].items():
if isinstance(v, (str, int, float, bool, list)):
meta[k] = v
vectors.append({
"key": doc_id,
"data": {
"float32": vec
},
"metadata": meta
})
if vectors:
print(f"[INFO] Uploading {len(vectors)} vectors to S3 bucket '{bucket}', index '{index}'.")
s3v.put_vectors(
vectorBucketName=bucket,
indexName=index,
vectors=vectors
)
moved += len(vectors)
print(f"[SUCCESS] Uploaded {len(vectors)} vectors. Total moved so far: {moved}")
last_id = hits[-1]["_id"]
print(f"[COMPLETE] Migration finished. Total vectors moved: {moved}")
return {"moved": moved}OpenSearch vector embedding for media_scenes index:
Executing the Lambda function:
Success! The vectors have been moved from our expensive hot tier to cost-effective cold storage. Here is a screen print of the S3 vector search for media-scenes index using AWS CLI:
The successful execution of the Lambda, confirmed by the S3 vector search via the AWS CLI, proves that the core plumbing of this hybrid architecture is sound. This small-scale test validates the most critical component of the system: the ability to programmatically migrate embeddings from a high-speed hot tier to a durable, low-cost cold tier.
Looking Ahead: The Native Future Is Coming
Our little experiment confirms that a tiered vector architecture is feasible, and it seems AWS is thinking along the same lines. It's important to highlight that while our conceptual approach could work with any vector database, AWS is building specific native solutions for its own ecosystem.
Two recent announcements, both currently in preview for Amazon OpenSearch Service Serverless, point to a more integrated future for this pattern:
- OpenSearch S3 Vector Data Pipeline Integration (Preview): This feature, detailed in the AWS documentation , Amazon OpenSearch Ingestion (OSI) serves as the data pipeline, automatically transferring vector data from Amazon S3 vector indexes to Amazon OpenSearch Serverless vector collections for efficient search operations.
- OpenSearch S3-backed indexes (Preview): As described in this guide , this proposes an even tighter integration where OpenSearch can manage indexes that are transparently backed by S3, offering durability, scalability, and cost-effectiveness for vector-heavy workloads.
While these powerful native solutions are on the horizon for the OpenSearch Service world, the core concepts we've explored today remain valuable regardless of your tech stack. The pattern of separating hot and cold embeddings applies to any vector database and provides a mental model for building scalable, cost-effective AI systems.
Limitations and Final Thoughts
Since S3 Vectors is in preview, this exploration comes with a few caveats:
- Evolving Features: The APIs and capabilities will likely change as the service matures. What we've tested today might be different tomorrow.
- Latency Trade-offs: Rehydrating vectors from cold storage will always be slower than querying a hot, in-memory database. This architecture is best suited for latency-tolerant use cases.
- Schema Constraints: When you create an S3 Vectors index, you lock in critical schema details like the vector dimension and the distance metric (e.g., cosine, euclidean). This means you cannot mix vectors of different dimensions (for instance, from two different embedding models) within the same index. Managing multiple schemas requires creating and maintaining separate indexes, which adds a layer of operational complexity.
- Simpler Search: The initial S3 Vectors service supports core distance metrics but may not have all the advanced filtering and search capabilities of a mature vector database.
This exploration shows that a tiered vector storage pattern is a promising way to manage the lifecycle and cost of large-scale embeddings. It encourages us to think about vector data not as a static asset to be stored in one place forever, but as dynamic data with a lifecycle that can be optimized.
As AI applications continue to grow, strategies for vector lifecycle management and cost optimization will become essential. While the tools are still evolving, the architectural concepts are here to stay.
Disclaimer: Please note that AWS is constantly evolving, and new features may be available since the release of this blog post. It's recommended to review the latest documentation to determine the most suitable solutions for your specific needs. This blog is a reference guide only. Additionally, ensure that the proposed solutions comply with your organization's security and compliance requirements, as some services may be relatively new and may not be fully compliant with all industry standards.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article