When Semantic Search Fails: Building Hybrid Retrieval for Production RAG

Semantic search returned survey number 45/2 when we asked for 45/3, and a deed from "Sharma to Patil" when we asked for "Sharma to Patel." The embedding model couldn't distinguish "conceptually similar" from "exactly this one" - and in title validation, that distinction is the entire point.

This post is a deep dive from our production RAG case study, where we built a document intelligence system that validates property title cleanliness across over a thousand properties in six Indian metros. Hybrid search was the difference between a demo that looked impressive and a system that lawyers actually trusted.

Semantic Similarity Is Not Relevance

Cosine similarity measures how close two things are in embedding space, not whether one answers the other.

We had a working RAG pipeline: solid chunking (after the iterations in the chunking deep dive), stabilized extraction schemas, handled OCR. We deployed semantic search and started routing real queries from the legal review team. Within the first week, we had a category of failures that no amount of prompt tuning could fix.

Query: "Encumbrance certificate for survey number 45/3?" Retrieved: ECs for survey numbers 45/2, 45/4, and 46/3. Should have retrieved: The EC for 45/3 specifically. Why: "45/3" and "45/2" embed into nearly identical vectors. The model can't distinguish "conceptually similar" from "exactly this one."

Query: "Sale deed from Sharma to Patel dated March 2015?" Retrieved: A 2017 deed between a different Sharma and Patel, and a 2015 deed from a Sharma to a Patil. Should have retrieved: The specific March 2015 deed between those two parties. Why: Proper nouns don't embed meaningfully. "Sharma" and "Patel" are common surnames treated as generic tokens. The model ranked by similarity to the concept of "sale deed involving parties with common surnames" rather than matching specific names.

Query: "Bank NOC for HDFC loan account ending 4521?" Retrieved: A different HDFC loan document (account ending 4528) and a mortgage deed from ICICI Bank discussing loan closure conditions. Should have retrieved: The specific NOC for that specific account. Why: Account numbers are arbitrary identifiers with zero semantic content. "4521" and "4528" are indistinguishable to the embedding model. It retrieved based on surrounding context ("HDFC," "loan," "NOC") which matched several documents equally well.

The pattern: every failure involves exact identifiers: survey numbers, proper nouns, account numbers, assessment years. These are the queries that matter most in title validation, and they're precisely where semantic search is weakest.

This isn't an Indian property documents problem. Any domain with precise identifiers (case numbers in legal research, ticker symbols in finance, part numbers in manufacturing, patient IDs in healthcare) will hit the same wall.

---

BM25 + Vector Search: The Hybrid Approach

The fix wasn't replacing semantic search. It was adding keyword search alongside it.

Semantic search excels at conceptual matching: "what are the obligations of the buyer?" retrieves relevant clauses even if they say "the Purchaser shall" or "it shall be the responsibility of the Second Party." You need semantic search for recall.

BM25 keyword search excels at exact matching: "survey number 45/3" returns documents containing exactly that string. "Sharma to Patel" matches those specific names. You need keyword search for precision on identifiers.

We already had pgvector in PostgreSQL for semantic search. Adding Elasticsearch for BM25 was straightforward: every chunk that went into pgvector also went into Elasticsearch. Same content, same metadata, different index.

Fusion: Reciprocal Rank Fusion (RRF). We chose RRF over score normalization for a practical reason: pgvector cosine similarity scores and Elasticsearch BM25 scores are on completely different scales. A cosine similarity of 0.82 and a BM25 score of 12.4 don't mean the same thing, and normalizing them requires assumptions about score distributions that shift with query patterns.

RRF sidesteps this by using rank position rather than raw scores. For each document in either result set, its RRF score is the sum of 1 / (k + rank) across both lists. A document ranked #1 in both lists scores highest. A document ranked #1 in keyword but absent from semantic still gets credit.

We experimented with weighted fusion (70/30, 50/50, various splits) but the optimal balance depends on query type, which you don't know until you've seen the query. Unweighted RRF with re-ranking downstream consistently outperformed tuned weights while being simpler to maintain.

---

Re-ranking: The Quality Multiplier Most Teams Skip

Hybrid search gets the right document into the top 15-20 results. Re-ranking gets it into the top 3-5, which is what the LLM actually sees.

RRF assigns scores based on rank position, but the difference between rank #1 and #2 in BM25 might be trivial (one extra keyword mention) or enormous (exact match vs. partial match). RRF can't distinguish these cases.

Cross-encoder re-ranking evaluates each candidate against the original query using a model that sees both simultaneously. Unlike bi-encoder embeddings (which encode query and document separately), a cross-encoder processes the pair together, modeling fine-grained interactions. We re-scored the top 20 RRF results and took the top 5 for the LLM context window.

The latency cost: 80-150ms per query. For legal review, where a lawyer spends 10 minutes on the output, this was invisible. For high-throughput, low-latency applications, it's a real tradeoff.

Re-ranking made the biggest difference on queries where both retrieval paths returned partially relevant results but neither had the best result at rank #1. For example, BM25 found the right deed (matched on party names) but ranked a different clause higher; semantic search found the right clause type but from a different deed. The cross-encoder surfaced the right-deed-right-clause combination.

The limitation: if neither retrieval path surfaces the right document in the top 20, re-ranking can't fix it. It re-orders candidates; it doesn't add new ones. That's a data quality problem, not a retrieval problem.

---

Latency Budget

The entire retrieval pipeline -- embedding, parallel pgvector and Elasticsearch queries, RRF fusion, and cross-encoder re-ranking -- runs under 200ms at P95. LLM generation dominates total latency by an order of magnitude (multiple seconds). The hybrid search + re-ranking overhead is invisible to the end user.

pgvector and Elasticsearch run in parallel, which keeps the retrieval addition modest over a semantic-only pipeline.

In our system, queries came from lawyers reviewing properties, not end users waiting for autocomplete. The latency budget was generous. For a consumer-facing application, you'd want to profile whether cross-encoder re-ranking is justified by the precision improvement for your query patterns.

---

Evaluation: Measuring the Improvement

We compared three configurations on an internal test set of representative queries from the legal review team, manually labeled with correct document references. We measured MRR, precision, and recall across semantic-only, hybrid, and hybrid+reranking.

The jump from semantic-only to hybrid was larger than from hybrid to hybrid+reranking. Hybrid search was the bigger win; re-ranking was the refinement that pushed precision into the range lawyers trusted.

Recall barely changed with re-ranking, which makes sense: re-ranking reorders existing results, it doesn't improve recall. The recall improvement came entirely from adding BM25.

By query type:

• Exact identifier queries (survey numbers, account numbers): semantic-only precision was poor. Hybrid brought it to a level lawyers could rely on. This was the category that justified the entire hybrid architecture.
• Proper noun queries (party names): improved substantially with hybrid search.
• Conceptual queries ("what are the conditions precedent?"): semantic search was already good here; hybrid added marginal value.

---

Adding Hybrid Search to an Existing System

Adding hybrid search to an existing pgvector setup is straightforward. Every chunk that goes into your vector store also goes into Elasticsearch (or OpenSearch, or any BM25-capable engine) -- same content, same metadata. On each query, run both searches in parallel, merge the result sets using RRF (pure computation, no model calls), and optionally re-score the top merged results with a cross-encoder. Each layer is independently valuable and independently reversible -- you can deploy hybrid search without re-ranking and add re-ranking later. You can A/B test hybrid against semantic-only on a percentage of traffic before committing. The key is to measure: compare MRR, precision, and recall between the old and new pipelines on a representative query set from your actual usage.