AWS Builder Center

Builder Center

The Tech Under the Hood: ANN, Hybrid Search, and RAG - Part 3 of 4

The Tech Under the Hood: ANN, Hybrid Search, and RAG - Part 3 of 4

Dive into the engineering of AI search. Learn how ANN scales vector retrieval, why Hybrid search maximizes accuracy, and how RAG grounds LLMs in enterprise data.

Sohaib Katariwala

So you've got your text converted into vectors. Great. Now what?
If you have 10 documents, finding the closest vector to your query is simple, just compare them all. But what about 10 million documents? Or a billion? You can't brute-force that in real time. This is where the engineering gets clever.

Approximate Nearest Neighbor (ANN) Search

The problem of finding the most similar vectors at scale is called nearest neighbor search. The simple approach of comparing your query vector to every single document vector, doesn't scale.
This is where Approximate Nearest Neighbor (ANN) algorithms come in. The most widely used one is called HNSW (Hierarchical Navigable Small World), and it's used by pretty much every major vector database out there: OpenSearch, Chroma, Qdrant, Redis, Weaviate, pgvector, the list goes on.
A technical diagram comparing two search methods. The left side, labeled 'Linear Search (Brute Force),' shows a system slowly checking every single document node one by one. The right side, labeled 'HNSW Graph Search (ANN),' displays a multi-layered network graph where a search query rapidly jumps across high-level connected nodes to drill down instantly to the nearest neighbor.

HNSW builds a graph-based index that lets you navigate to the nearest neighbors in logarithmic time instead of linear time. The trade-off? It finds approximately the nearest neighbors, not guaranteed exact ones. In practice, that's totally fine fro most use cases. ANN algorithms routinely hit 95%+ recall rates while searching billions of vectors in milliseconds.
That's the kind of performance that makes real-time semantic search possible at scale.

Hybrid Search: Best of Both Worlds

Here's something that might seem counterintuitive: the best search systems don't replace keyword search with semantic search, they use both.
Keyword search is still great for:
  • Exact matches (product codes, error messages, model numbers)
  • Rare or highly technical terms
  • Cases where precision matters more than recall
Semantic search is better for:
  • Conceptual queries and natural language questions
  • Handling synonyms and paraphrasing
  • Understanding intent and context
  • Cross-language search
Hybrid search runs both in parallel and combines the scores. According to AWS research on Amazon OpenSearch Service, hybrid search delivers 8–12% better results than keyword search alone and 15% better than semantic search alone. That's a meaningful improvement, and it's why hybrid is the recommended approach for most production systems including on Amazon OpenSearch Service.
A flow diagram showing a single user query splitting into two parallel engines: a 'Keyword Engine (BM25)' and a 'Semantic Engine (Vector).' The keyword engine pulls exact matches (like product IDs), while the semantic engine pulls conceptual matches. Both result sets flow into a 'Score Normalization' block, which outputs a single, highly accurate ranked list.


RAG: Connecting Search to AI

The last piece of the puzzle is Retrieval Augmented Generation (RAG). This is where search meets generative AI.
The problem with large language models (LLMs) is that they're trained on general knowledge. They don't know about your company's internal docs, your product catalog, or anything that happened after their training cutoff. RAG fixes that.
Here's how it works:
  1. User asks a question in natural language
  1. The question gets converted to a vector embedding
  1. Vector search retrieves the most relevant documents from your knowledge base (Vector Database)
  1. Those documents get passed to the LLM along with the original question
  1. The LLM generates an answer grounded in your actual data
A step-by-step diagram of the RAG process. 1. A user asks a question. 2. An embedding model converts it to a vector. 3. The vector is used to search a Vector Database. 4. The database returns relevant internal company documents. 5. Both the original question and the retrieved documents are injected into an LLM prompt. 6. The LLM generates a grounded, accurate answer based on the retrieved data.

The result? An AI that can answer questions about your specific data, not just general knowledge. This is the pattern behind most enterprise AI assistants and chatbots you see today.
On AWS, a typical RAG setup combines Amazon OpenSearch Service (for vector storage and retrieval), Amazon Bedrock (for the LLM and embedding generation), and AWS Lambda (for orchestration). Bedrock's Knowledge Bases feature can automate most of this pipeline for you.

Putting It All Together

At this point, you've got the full picture of how AI-powered search works under the hood:
  • Embeddings turn meaning into math
  • ANN algorithms make searching at scale fast
  • Hybrid search combines semantic and keyword approaches for the best results
  • RAG connects search to generative AI for intelligent, grounded answers

Resources

# ai# vector-search# semantic-search# opensearch# amazon-opensearch
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article