Hybrid Retrieval and Cross-Encoder Reranking for RAG

When engineering a Retrieval-Augmented Generation (RAG) system, the default starting point for most developers is simple vector search (dense retrieval). You split your corpus into chunks and generate embeddings with a model like OpenAI's text-embedding-3-small. You then index them in a vector database and query with cosine similarity.

While this approach works for basic demo applications, it frequently fails in production environments. Dense embeddings capture semantic meaning, but they struggle with exact matching. They miss specific product codes, serial numbers, acronyms, and rare technical keywords.

To build a search system that handles both conceptual meaning and exact keyword precision, you need a hybrid search architecture combined with a reranking stage. This guide outlines how to build this pattern.

Why vector embeddings fail alone

Dense embeddings miss exact matches. A 2024 documentation search pilot surfaced three recurring failure modes in documentation and e-commerce search.

• Query: "Error code 403-B"
  • Dense Embeddings: Might retrieve articles about general authentication errors or HTTP 403 Forbidden. They share a semantic neighborhood but miss the specific article on the rare "403-B" variant.
• Query: "Python 3.12 syntax"
  • Dense Embeddings: Often retrieves articles about Python 3.10, Python 3.11, or general syntax. The vector distance between 3.11 and 3.12 is nearly identical in the embedding space.
• Query: "SKU-99218"
  • Dense Embeddings: Struggles entirely because alphanumeric product identifiers do not carry inherent semantic meaning. They map to near-random points in the high-dimensional space.

To solve this, we pair dense vector search with sparse retrieval (BM25). BM25 (Best Matching 25) is a term frequency-based algorithm that rewards exact matches, matches of rare terms, and penalizes documents that repeat keywords excessively.

The hybrid search pipeline architecture

A production-grade retrieval pipeline runs dense and sparse retrieval in parallel, merges the results, and then filters them using a deep learning model.

The pipeline consists of three sequential phases.

1. Parallel Retrieval: The pipeline sends the user's query to a BM25 engine (like Elasticsearch or a local inverted index) and a Vector Search engine together.
2. Rank Fusion: The results from both lists merge through Reciprocal Rank Fusion (RRF) to produce a single, unified candidate list.
3. Cross-Encoder Reranking: The top candidates go to a Cross-Encoder model, which scores the exact relevance of each document against the query and outputs a high-precision list.

Merging search tracks with Reciprocal Rank Fusion (RRF)

You cannot combine dense and sparse search by simply adding their raw similarity scores together. BM25 scores have no upper bound (e.g., 12.5). Vector similarity scores stay within a fixed range (e.g., 0.78 cosine similarity). The two scales differ entirely.

Instead of normalized scores, we use Reciprocal Rank Fusion (RRF). RRF evaluates the relative rank of a document within each search result list.

For each document $d$, the score sums $1 / (k + r_m(d))$ across every retrieval system $m$. Here $M$ is BM25 plus vector search. The rank $r_m(d)$ is the 1-indexed position in system $m$. The constant $k$ is typically $60$ and softens the weight of top ranks.

$$\text{RRF_Score}(d \in D) = \sum_{m \in M} \frac{1}{k + r_m(d)}$$.

RRF guarantees that a document appearing in the top 5 of both systems will rank higher than a document that is #1 in only one system and missing from the other.

The precision layer: cross-encoder reranking

Retrieval models (both sparse and dense) are Bi-Encoders. They process the query and the documents independently. Documents get pre-embedded while queries embed on the fly, so the model cannot perform deep token-level cross-attention between the query and the document text.

A Cross-Encoder processes the query and a candidate document together as a single input pair (e.g., [CLS] Query [SEP] Document [SEP]). Full self-attention layers compare every word in the query with every word in the document.

Cross-Encoders are accurate, but computationally heavy and slow. You cannot run a Cross-Encoder over 100,000 documents in real-time.

Instead, we use a two-stage approach.

1. Stage 1 Retrieval (Fast): Use BM25 and Vector search to filter 100,000 documents down to the top 50 candidates.
2. Stage 2 Reranking (Accurate): Use a Cross-Encoder (like bge-reranker-large or Cohere Rerank) to score just those 50 candidates, reorganizing them to place the most relevant items at the top.

The performance gains of this hybrid and reranking architecture over standalone methods are large.

Hybrid search captures exact keyword and semantic hits. Adding a Cross-Encoder Reranker shifts the curve upwards and provides high recall accuracy even at lower values of $K$.

Implementing a hybrid and rerank pipeline in TypeScript

This TypeScript coordinator merges hits from an inverted index and a vector database. It applies RRF and calls a reranking service.

interface SearchHit {
  documentId: string;
  text: string;
  score: number;
}
 
interface RankedDocument {
  documentId: string;
  text: string;
  rank: number;
}
 
// Reciprocal Rank Fusion calculation
function computeRRF(
  vectorHits: SearchHit[],
  bm25Hits: SearchHit[],
  k = 60,
): RankedDocument[] {
  const rrfScores: Record<string, { text: string; score: number }> = {};
 
  const processHits = (hits: SearchHit[]) => {
    hits.forEach((hit, index) => {
      const docId = hit.documentId;
      const rank = index + 1; // 1-indexed rank
      const scoreContribution = 1 / (k + rank);
 
      if (!rrfScores[docId]) {
        rrfScores[docId] = { text: hit.text, score: 0 };
      }
      rrfScores[docId].score += scoreContribution;
    });
  };
 
  processHits(vectorHits);
  processHits(bm25Hits);
 
  // Sort documents by ascending combined score rank
  return Object.entries(rrfScores)
    .map(([documentId, data]) => ({
      documentId,
      text: data.text,
      score: data.score,
    }))
    .sort((a, b) => b.score - a.score)
    .map((item, index) => ({
      documentId: item.documentId,
      text: item.text,
      rank: index + 1,
    }));
}
 
// Coordinator function
export async function retrieveHybrid(
  query: string,
  limit = 5,
): Promise<RankedDocument[]> {
  // 1. Run Stage 1 Retrieval in parallel
  const [vectorHits, bm25Hits] = await Promise.all([
    queryVectorDatabase(query, 50), // Retrieve top 50 semantic candidates
    queryBM25Index(query, 50), // Retrieve top 50 keyword candidates
  ]);
 
  // 2. Perform Rank Fusion
  const fusedCandidates = computeRRF(vectorHits, bm25Hits, 60);
 
  // Slice to the top 20 candidates for the expensive reranking stage
  const topCandidates = fusedCandidates.slice(0, 20);
 
  // 3. Stage 2: Cross-Encoder Reranking
  const reranked = await queryRerankerModel(
    query,
    topCandidates.map((doc) => ({ id: doc.documentId, text: doc.text })),
  );
 
  // Return the top-k results
  return reranked.slice(0, limit);
}

Engineering tradeoffs

When deploying a hybrid search architecture, evaluate these considerations.

• Latency Overhead: Running three operations (vector search, BM25 search, and Cross-Encoder evaluation) increases query latency. Optimize this by querying the indices in parallel, caching frequent queries, and keeping the candidate count for the reranker low (e.g., $N = 25$).
• Infrastructure Complexity: You must maintain both an inverted index for keyword search and a vector index. Modern databases (like Pgvector, Elasticsearch, Qdrant, and Pinecone) offer built-in hybrid search and automatic RRF, letting you manage one datastore instead of two.
• Cost of Compute: Cross-Encoder models are larger and require GPU acceleration to maintain sub-100ms response times. If hosting a model locally is too expensive, consider managed APIs like Cohere Rerank or Jina Reranker.

By building hybrid search and reranking, you move away from RAG demonstrations that only work on simple queries, towards search systems that reliably locate documents under complex real-world conditions.