The Problem with Dense-Only Retrieval
Bi-encoders encode queries and documents into isolated vector representations, relying entirely on cosine similarity across latent semantic space. While effective for thematic matching, dense embeddings suffer from:
Vocabulary Mismatch & Specificity Loss: Specialized alphanumeric tokens (e.g., ERR_HTTP2_PROTOCOL_ERROR_404) often map poorly across general-purpose embedding spaces.
Bi-Encoder Compression Bottlenecks: Compressing an entire document chunk into a single fixed-length vector (e.g., 768 or 1536 dimensions) strips out fine-grained term occurrences.
False Positives in Domain Nuance: Semantically similar yet practically irrelevant passages score high, polluting the LLM's context window.
User Query ──► BM25 Sparse Retriever (Lucene / OpenSearch / SQLite FTS5) │ ▼ Top-50 Raw Candidate Chunks Cross-Encoder Reranker (e.g., BAAI/bge-reranker-large) │ ▼ Top-5 Precision Reranked Chunks LLM Generation Context Window
Stage 1: High-Recall Sparse Retrieval (BM25)The query evaluates against an inverted index using the BM25 scoring algorithm:$$\text{Score}(D, Q) = \sum_{i=1}^{N} \text{IDF}(q_i) \cdot \frac{f(q_i, D) \cdot (k_1 + 1)}{f(q_i, D) + k_1 \cdot \left(1 - b + b \cdot \frac{\vert{}D\vert{}}{\text{avgdl}}\right)}$$Fast execution over millions of chunks in under 10ms.Guarantees that exact terms, IDs, and domain-specific acronyms are captured in the initial candidate pool ($k = 50$).Stage 2: Cross-Encoder Full Attention RerankingUnlike bi-encoders, cross-encoders process the query and each candidate chunk simultaneously through cross-attention layers:$$\text{Input} = [\text{CLS}] \circ \text{Query} \circ [\text{SEP}] \circ \text{Candidate Chunk} \circ [\text{SEP}]$$Evaluates token-level interactions between query terms and candidate text directly.Eliminates semantic drift and filters out low-signal BM25 keyword matches.Truncates the pool to the top 3–5 high-relevance chunks.Implementation Example (Python)Pythonfrom rank_bm25 import BM25Okapi
from sentence_transformers import CrossEncoder
1. Corpus Preparation
documents = [
"PostgreSQL supports JSONB indexing via GIN indexes for fast key-value lookups.",
"BM25 is a bag-of-words retrieval function that ranks a set of documents based on query terms.",
"Cross-encoders compute cross-attention over query-document pairs, providing high-precision scoring.",
"To resolve ERR_CONN_REFUSED, verify that the backend daemon is actively listening on target port 8080."
]
tokenized_corpus = [doc.lower().split(" ") for doc in documents]
bm25 = BM25Okapi(tokenized_corpus)
reranker = CrossEncoder("BAAI/bge-reranker-base")
2. Query Execution
query = "How to fix ERR_CONN_REFUSED on port 8080?"
tokenized_query = query.lower().split(" ")
Step 1: Initial sparse retrieval (BM25)
bm25_scores = bm25.get_scores(tokenized_query)
top_k_indices = sorted(range(len(bm25_scores)), key=lambda i: bm25_scores[i], reverse=True)[:3]
candidates = [documents[i] for i in top_k_indices]
Step 2: Cross-Encoder Reranking
pairs = [[query, candidate] for candidate in candidates]
rerank_scores = reranker.predict(pairs)
Step 3: Final Ordered Context
ranked_results = sorted(zip(candidates, rerank_scores), key=lambda x: x[1], reverse=True)
for doc, score in ranked_results:
print(f"[{score:.4f}] {doc}")
Comparative EvaluationArchitectureExact Term MatchLatency (p95)Infrastructure CostHallucination RateDense Only (HNSW / Cosine)Moderate25msHigh (RAM/VRAM)HighSparse Only (BM25)High8msVery Low (CPU/Disk)ModerateBM25 + Cross-Encoder RerankerVery High35msLow (Standard CPU/GPU)LowestPractical TakeawaysTune $k_1$ and $b$ Parameters: For technical documentation with short chunk lengths (256–512 tokens), setting $b = 0.4$ prevents short, concise code blocks from being penalized by length normalization.Keep Candidate Size Controlled: Restrict Stage 1 retrieval to 30–50 candidates to keep reranker inference times under 30ms on standard CPUs.Zero Vector DB Footprint: A pure BM25 + Reranker pipeline runs entirely on traditional full-text search engines (Elasticsearch, PostgreSQL FTS, SQLite), eliminating specialized vector database maintenance.
Loading article