# Beyond Simple Vector Search: Implementing Hybrid Retrieval and Cross-Encoder Reranking

URL: https://mepritam.dev/references/beyond-vector-search-hybrid-retrieval-reranking/

An engineering deep dive into combining BM25 keyword matching with dense vector embeddings and Cross-Encoder reranking models to build robust, production-grade RAG systems.

---

When engineering a Retrieval-Augmented Generation (RAG) system, the default starting point for most developers is simple vector search (dense retrieval). You split your corpus into chunks, generate embeddings using a model like OpenAI's `text-embedding-3-small`, index them in a vector database, and query it using cosine similarity.

While this approach works for basic demo applications, it frequently fails in production environments. Dense embeddings represent overall semantic meaning, but they struggle with exact matching. They are blind to specific product codes, serial numbers, acronyms, and rare technical keywords.

To build a search system that handles both conceptual meaning and exact keyword precision, you need a **hybrid search architecture** combined with a **reranking stage**. This guide outlines how to implement this pattern.

---

## 1. The Retrospective: Why Vector Embeddings Fail Alone

To see where dense embeddings fail, consider these query scenarios in a documentation or e-commerce search:

- **Query: "Error code 403-B"**
  - _Dense Embeddings:_ Might retrieve articles about general authentication errors or HTTP 403 Forbidden because they share a semantic neighborhood. It misses the specific article addressing the rare "403-B" variant.
- **Query: "Python 3.12 syntax"**
  - _Dense Embeddings:_ Often retrieves articles about Python 3.10, Python 3.11, or general syntax because the vector distance between `3.11` and `3.12` is extremely small in the embedding space.
- **Query: "SKU-99218"**
  - _Dense Embeddings:_ Struggles entirely because alphanumeric product identifiers do not carry inherent semantic meaning. They map to near-random points in the high-dimensional space.

To solve this, we pair dense vector search with **sparse retrieval (BM25)**. BM25 (Best Matching 25) is a term frequency-based algorithm that rewards exact matches, matches of rare terms, and penalizes documents that repeat keywords excessively.

---

## 2. The Hybrid Search Pipeline Architecture

A production-grade retrieval pipeline runs dense and sparse retrieval in parallel, merges the results, and then filters them using a deep learning model.

![RAG Hybrid Search Pipeline](/images/references/hybrid-search-pipeline.jpg)

The pipeline consists of three sequential phases:

1.  **Parallel Retrieval:** The user's query is dispatched to a BM25 engine (like Elasticsearch or a local inverted index) and a Vector Search engine simultaneously.
2.  **Rank Fusion:** The results from both lists are combined using Reciprocal Rank Fusion (RRF) to produce a single, unified candidate list.
3.  **Cross-Encoder Reranking:** The top candidates are sent to a Cross-Encoder model, which evaluates the exact relevance of each document against the query, outputting a high-precision list.

---

## 3. Merging Search Tracks: Reciprocal Rank Fusion (RRF)

You cannot combine dense and sparse search by simply adding their raw similarity scores together. BM25 scores are unbounded positive numbers (e.g., `12.5`), while vector similarity scores are bounded decimals (e.g., `0.78` cosine similarity). They are completely different scales.

Instead of normalized scores, we use **Reciprocal Rank Fusion (RRF)**. RRF evaluates the _relative rank_ of a document within each search result list.

$$\text{RRF\_Score}(d \in D) = \sum_{m \in M} \frac{1}{k + r_m(d)}$$

Where:

- $M$ is the set of retrieval systems (in this case, BM25 and Vector Search).
- $r_m(d)$ is the 1-indexed rank of document $d$ in system $m$.
- $k$ is a constant smoothing parameter (typically set to $60$) that prevents high-ranking documents from dominating the score too much.

RRF guarantees that a document appearing in the top 5 of _both_ systems will rank higher than a document that is #1 in only one system and missing from the other.

---

## 4. The Precision Layer: Cross-Encoder Reranking

Retrieval models (both sparse and dense) are **Bi-Encoders**. They process the query and the documents independently. Because documents are pre-embedded and queries are embedded on the fly, the model cannot perform deep token-level cross-attention between the query and the document text.

A **Cross-Encoder**, on the other hand, processes the query and a candidate document _together_ as a single input pair (e.g., `[CLS] Query [SEP] Document [SEP]`). This allows full self-attention layers to compare every word in the query with every word in the document.

While Cross-Encoders are incredibly accurate, they are computationally heavy and slow. You cannot run a Cross-Encoder over 100,000 documents in real-time.

Instead, we use a two-stage approach:

1.  **Stage 1 Retrieval (Fast):** Use BM25 and Vector search to filter 100,000 documents down to the top 50 candidates.
2.  **Stage 2 Reranking (Accurate):** Use a Cross-Encoder (like `bge-reranker-large` or Cohere Rerank) to evaluate just those 50 candidates, reorganizing them to place the most relevant items at the top.

The performance gains of this hybrid and reranking architecture over standalone methods are substantial:

![Search Retrieval Accuracy Comparison](/images/references/search-accuracy-comparison.jpg)

As shown above, Hybrid search captures exact keyword and semantic hits, while the addition of a Cross-Encoder Reranker shifts the curve upwards, providing high recall accuracy even at lower values of $K$.

---

## 5. Implementing a Hybrid & Rerank Pipeline in TypeScript

Here is a TypeScript implementation of a retrieval coordinator that merges hits from an inverted index and a vector database, applies RRF, and contacts a reranking service.

```typescript
interface SearchHit {
  documentId: string;
  text: string;
  score: number;
}

interface RankedDocument {
  documentId: string;
  text: string;
  rank: number;
}

// Reciprocal Rank Fusion calculation
function computeRRF(
  vectorHits: SearchHit[],
  bm25Hits: SearchHit[],
  k = 60,
): RankedDocument[] {
  const rrfScores: Record<string, { text: string; score: number }> = {};

  const processHits = (hits: SearchHit[]) => {
    hits.forEach((hit, index) => {
      const docId = hit.documentId;
      const rank = index + 1; // 1-indexed rank
      const scoreContribution = 1 / (k + rank);

      if (!rrfScores[docId]) {
        rrfScores[docId] = { text: hit.text, score: 0 };
      }
      rrfScores[docId].score += scoreContribution;
    });
  };

  processHits(vectorHits);
  processHits(bm25Hits);

  // Sort documents by ascending combined score rank
  return Object.entries(rrfScores)
    .map(([documentId, data]) => ({
      documentId,
      text: data.text,
      score: data.score,
    }))
    .sort((a, b) => b.score - a.score)
    .map((item, index) => ({
      documentId: item.documentId,
      text: item.text,
      rank: index + 1,
    }));
}

// Coordinator function
export async function retrieveHybrid(
  query: string,
  limit = 5,
): Promise<RankedDocument[]> {
  // 1. Run Stage 1 Retrieval in parallel
  const [vectorHits, bm25Hits] = await Promise.all([
    queryVectorDatabase(query, 50), // Retrieve top 50 semantic candidates
    queryBM25Index(query, 50), // Retrieve top 50 keyword candidates
  ]);

  // 2. Perform Rank Fusion
  const fusedCandidates = computeRRF(vectorHits, bm25Hits, 60);

  // Slice to the top 20 candidates for the expensive reranking stage
  const topCandidates = fusedCandidates.slice(0, 20);

  // 3. Stage 2: Cross-Encoder Reranking
  const reranked = await queryRerankerModel(
    query,
    topCandidates.map((doc) => ({ id: doc.documentId, text: doc.text })),
  );

  // Return the top-k results
  return reranked.slice(0, limit);
}
```

---

## 6. Engineering Tradeoffs

When deploying a hybrid search architecture, evaluate these considerations:

- **Latency Overhead:** Running three operations (vector search, BM25 search, and Cross-Encoder evaluation) increases query latency. Optimize this by querying the indices in parallel, caching frequent queries, and keeping the candidate count for the reranker low (e.g., $N = 25$).
- **Infrastructure Complexity:** You must maintain both an inverted index for keyword search and a vector index. Modern databases (like Pgvector, Elasticsearch, Qdrant, and Pinecone) offer built-in hybrid search and automatic RRF, letting you manage one datastore instead of two.
- **Cost of Compute:** Cross-Encoder models are larger and require GPU acceleration to maintain sub-100ms response times. If hosting a model locally is too expensive, consider managed APIs like Cohere Rerank or Jina Reranker.

By implementing hybrid search and reranking, you move away from RAG demonstrations that only work on simple queries, towards search systems that reliably locate documents under complex real-world conditions.