Beyond Keyword Search: Building Multimodal Semantic Search with Amazon Nova Embeddings and S3 Vector
Key insights from AWS Community Day Bengaluru | July 11, 2026
Series: AWS Conferences Tour (42 articles)
- …
- 42Beyond Keyword Search: Building Multimodal Semantic Search with Amazon Nova Embeddings and S3 Vector This article
I had an amazing time at AWS Community Day Bengaluru on 11 July 2026. The energy was like nothing else and while there were many of amazing sessions throughout the day, one specific presentation completely stopped me in my tracks. Naresh Waswani took the stage to discuss creating multimodal semantic search with Amazon Nova Embeddings and S3 Vectors.
Let's be honest, as developers, we often take search for granted. We just hit a database with a query and hope that the right keyword pops up. But Naresh’s session made me realise how much we miss when we depend on words alone.
So here’s the lowdown of what I learned from this awesome session, fully understanding how search is evolving in the cloud.
The Search Evolution: From Exact Words to True Meaning
If we look back at the evolution of search, it can be essentially seen as three separate phases.
We first had Keyword search. This is the classic lexical search, utilising algorithms such as BM25. Matches precise words. If you search for "apple" you'll find the term "apple". It is literal and accurate, but entirely blind to real meaning.
Then semantic search. This uses vector similarity to find meaning even if the words are completely different. Think of it as ordering food on Swiggy. If you write in “cheesy Italian comfort food” keyword search will fail if the menu only lists “Margherita Pizza”. Semantic search knows the relation between such concepts and serves you exactly what you are desiring.
And last, we’re entering the era of Answers, when Generative AI writes a smart answer instead of just a page of blue links. Both semantic search and these new answer engines are vector based.
The Problem with Traditional Search
Naresh used a scenario that many may relate to in an effort to explain why keyword search is not suitable for modern use cases. Imagine you had to filter through thousands of hours of support conversations to uncover an audio file of “an angry customer complaining about billing.”
This just won’t appear in a normal keyword search. The recording is not text, so there are no words to match. Even with a perfect speech-to-text transcript, it might never include the exact phrases "angry" or "billing". The customer may just be screaming about “a ridiculous charge on my account.”
Why Transcription Loses the Signal
You could be asking yourself, why not just transcribe everything? Transcription provides you words and names and themes. But the signal is completely lost.
When you reduce audio to text, you lose the tone, focus, and passion. Angry is a tone, not a word stated. You miss the context of who sounded unsure or the overall atmosphere of the room. transcripts can not answer a text query : how something is said.
Embeddings 101: Turning Meaning into Geometry
Now it gets spicy. We fix this by using embeddings. An embedding is just a list of integers (a vector) that captures the fundamental meaning of a piece of content.
A machine learning model examines your material and puts comparable things close and irrelevant things far. Geometry develops meaning. Searching is no longer “look for these letters in this order.” No. That's not what it says. It means discover the nearest mathematical points.
The point to bear in mind here is that direction, not distance, carries meaning. These models learn from seeing what content co-occurs at vast size, thus related concepts end up heading in the same direction. The two vectors can be very long or very short. What we compare is the angle between them. Same direction signifies same meaning.
This constitutes a mathematical idea, hence it is completely modality-free. Train one spot to contain them all . The same method that works for text works for pixels and audio samples .
Amazon Nova: One Model, Every Modality
This leads us to Amazon Nova Multimodal Embeddings. Developers have had two options to tackle multimodal data.
The first approach was to flatten everything to text. You can create captions for photos, extract tables to CSVs and transcribe audio. All that text you put into one vector space. It’s cheap and accurate for names and numbers, but again it lacks visual appeal, arrangement and tone.
The second approach is the native embedding path. This is exactly what Nova does. Nova integrates text, images, music, video and documents into a single common embedding area. Since all things fall into the same space, a query in one modality can retrieve hits in another modality.
Enter text and receive matched audio snippets. You may upload a photo to find similar products. Describe a scene and jump to the exact part in the video. Nova embeds the modality itself, which is why a tone like “angry” finally becomes searchable.
The Knobs That Matter in Nova
If you are working with Nova, there are four main parameters you need to design around:
- Output dimension: You can select sizes including 3072, 1024, 384 or 256. In Matryoshka training, the smaller vectors still hold most of the signal, so you can sacrifice a little bit of precision for better storage and speed.
- Sync vs Async: Use synchronous calls for near real-time searches and async jobs for processing large audio, video or text chunks.
- Limits: Text inputs are limited to 8,192 characters while audio and video are processed in 30 second intervals.
- Embedding Purpose: You have to set your intention. Use
GENERIC_INDEXfor data storage and query for*_RETRIEVALpurpose. They need to communicate the same embedding language.
The Storage Problem and S3 Vectors
Now we have a new challenge : Vectors are exploding and it’s not inexpensive to store them.
Semantic search vectors all your documents, images and recordings. In a genuine enterprise corpus there are billions of these and they have to live somewhere. Dedicated vector databases typically require always-on computing. You are paying for clusters that are running 24/7 even if you query them once an hour or never. The corpus will grow into the billions and memory resident indexes get extremely expensive.
To help overcome this challenge, AWS has released S3 Vectors (Previewed in July 2025, General Availability in December 2025). “This is the first native vector support in a cloud object store.”
How S3 Vectors Change the Game
S3 Vectors is completely serverless and requires no cluster provisioning. It is up to 90% cheaper than purpose-built vector databases.
The size is enormous. You can have up to 10k vector indexes per bucket, and 2 billion vectors per index (potentially 20 trillion vectors per bucket). It has the same elasticity, durability, and 11 nines availability as regular Amazon S3 and warm query latencies of about 100 milliseconds.
The conceptual model is simple: you have a Vector Bucket and you create a Vector Index inside it (specified by dimension, a metric such as cosine, and a datatype such as float32). Inside that index, you keep the vector itself and its metadata.
Cost-First, Not Latency-First
It’s crucial to recognise the tradeoffs here. S3 Vectors is a change in pricing. You just pay for what you store and query – nothing sits idle.
But the real truth is, it’s not some single-digit-millisecond, high-QPS (Queries Per Second) engine. It does deliver sub-second performance (as low as ~100 ms when warm).
If you are designing an architecture Think of it as data tiering. You put your hot, often visited vectors into OpenSearch to get low, predictable latencies. You shove your cold, infrequently queried, bulk vectors into S3 Vectors for huge cost reductions. Then, as demand increases, you can smoothly promote vectors up into OpenSearch.
The ideal practice for multi-tenant SaaS applications is one index per tenant in your bucket. This provides you faster scoped queries, clearer IAM separation, and makes reasoning at scale a lot easier.
Show Me the Code!
Naresh demonstrated how easy this is to do with the standard Python
boto3 . It comes down to 3 easy steps.1. Generate an Embedding
With Bedrock you pass your text (or substitute it for picture, audio, or video) to the Nova model. The purpose is defined as
GENERIC_INDEX .Python
import boto3, jsonbedrock = boto3.client("bedrock-runtime", region_name="us-east-1")resp = bedrock.invoke_model(modelId="amazon.nova-2-multimodal-embeddings-v1:0",body=json.dumps({"taskType": "SINGLE_EMBEDDING","singleEmbeddingParams": {"embeddingPurpose": "GENERIC_INDEX", # indexing"embeddingDimension": 1024,"text": {"truncationMode": "END", "value": text}}}))vec = json.loads(resp["body"].read())["embeddings"][0]["embedding"]
2. Create an Index and Store Metadata
Then you build the index in S3 Vectors and put your newly produced vector into it, providing extremely important metadata (such the agent's name and topic) that goes with the vector.
Python
s3v = boto3.client("s3vectors")s3v.create_index(vectorBucketName="support-calls", indexName="calls",dimension=1024, distanceMetric="cosine",dataType="float32")s3v.put_vectors(vectorBucketName="support-calls", indexName="calls",vectors=[{"key": "call_8842","data": {"float32": vec},"metadata": {"agent": "R. Mehta", "topic": "billing"}}])
3. Run a k-NN Query
Finally, you integrate your user's search query (with the
GENERIC_RETRIEVAL purpose this time), and ask S3 Vectors to discover the closest matches, filtered by your metadata.Python
# embed the query - note the RETRIEVAL purposeq = embed("an angry customer complaining about billing",purpose="GENERIC_RETRIEVAL")res = s3v.query_vectors(vectorBucketName="support-calls", indexName="calls",queryVector={"float32": q},topK=5,filter={"topic": "billing"}, # metadata filterreturnDistance=True)for m in res["vectors"]:print(m["key"], m["distance"])
The dividend? You get the best actual audio call matches to your text inquiry by meaning, ranked, filtered and cited by key.
A Quick Decision Guide and Production Gotchas
So when should you really use S3 Vectors? When you have a vast or ever-growing corpus. Where cost and operational simplicity are paramount. Where sub-second latency is sufficient. Go for it.
If you really require single-digit-ms latency, or thousands of concurrent queries per second, or if you care more about consistent performance than about infrastructure cost, go elsewhere (OpenSearch, or a dedicated vector DB).
To be fair, Naresh also gave me some honest production “gotchas” to look out for:
- Storage constraints: Top-K results are limited to 30 and do not support pagination, which may break standard scrolling UIs. Filterable metadata is limited to 2 Kbytes.
- Embeddings constraints: Async embedding maxes out at 30 jobs per account, thus you have to queue and poll. Index and retrieval purposes mixed silently damage your search recollection.
- Generation constraints: New accounts usually have 0 TPM quota (Tokens Per Minute), so you need to seek a limit raise before you start. Throttling is basically an architectural fact, so put backoff and jitter logic in.
Seeing it in Action: The Multimodal Demo
The presentation concluded off with a great live demo of the Nova Multimodal Search UI.
As the search query, Naresh posted a photograph of a crying baby. All living in one shared vector space, the search results were an incredible mix of modalities. It identified visually similar images (such a created image of a baby, and shockingly, a lion), but more significantly, it provided a relevant audio file:
freesound-community-baby-crying.mp3 .
We searched in pixels and the engine returned us sound. That is the magic of genuine cross-modal semantic search.
Key Takeaways
- Keywords miss the vibe: Traditional search looks for precise words, but native multimodal embeddings capture the true tone, emotion and context – thus a query for “angry” can effectively identify a customer shouting in an audio file.
- One shared space for everything: You no longer need to transcribe audio and caption photos to make them searchable. Amazon Nova enables full cross-modal search by putting text, video, audio and images into a common shared vector space.
- Vector storage is now serverless: With Amazon S3 Vectors, you don’t need to pay for always-on database clusters. It cuts costs by up to 90 percent for large datasets that are rarely queried.
- Tiering is the winning strategy: Store your latency-sensitive “hot” data in OpenSearch and your huge “cold” vector archives in S3 Vectors for the optimal balance of cost and speed.
Conclusion
Ultimately, keyword search is not going away, but it is no longer sufficient for the rich complex data we generate today.
If you’re starting a startup or managing a huge data lake, Amazon Nova plus S3 Vectors changes the equation significantly. Now you can deliver next-gen AI search capabilities without breaking your infrastructure budget. Long story short, the future of search isn’t matching letters on a screen, it’s comprehending human meaning, no matter what form it takes.
About the Author
As an AWS Community Builder, I enjoy sharing the things I've learned through my own experiences and events, and I like to help others on their path. If you found this helpful or have any questions, don't hesitate to get in touch! 🚀
References
Event: AWS Community Day Bengaluru
Topic: Beyond Keyword Search: Building Multimodal Semantic Search with Amazon Nova Embeddings and S3 Vector
Date: July 11, 2026
Series: AWS Conferences Tour (42 articles)
- …
- 42Beyond Keyword Search: Building Multimodal Semantic Search with Amazon Nova Embeddings and S3 Vector This article
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article