RRF and Reranking: Why My RAG Pipeline Needed Both

Notes from building the retrieval layer for AnvoQ AI · rag, retrieval, qdrant, nvidia, search

When I was building the retrieval layer for my RAG backend, I ran into a question that trips up a lot of people the first time they build hybrid search: if you already have a reranker scoring your results, why bother with Reciprocal Rank Fusion at all? Aren't you just fusing things twice?

I asked myself the same thing halfway through building this, so I want to write down the answer properly — partly for anyone else hitting the same wall, and partly so future-me remembers why the pipeline looks the way it does.

RRF and a reranker solve different problems, so running both isn't redundant:

RRF is cheap, rank-based fusion of first-stage retrievers whose scores can't be compared (e.g. dense vs. sparse search).

Reranking raw hits without fusing first roughly doubles reranker calls per knowledge base — fusing keeps it to one merged set.

Reranker scores are raw logits, not calibrated probabilities. A 0.5-sigmoid cutoff silently drops genuinely relevant chunks — thresholds have to be derived empirically per model.

What RRF actually is

Reciprocal Rank Fusion is a way to combine multiple ranked lists into one list, without needing the underlying scores to mean the same thing. The formula is almost embarrassingly simple:

score(doc) = sum( 1 / (k + rank_in_list) )  across every list the doc appears in

k is usually a small constant like 60, mostly there to keep the top ranks from dominating too aggressively. That's it. No score normalization, no weighting scheme, no training. A document that shows up near the top of two different lists gets a high fused score. A document that only shows up in one list, and near the bottom, doesn't.

The part that matters most is what RRF ignores: it never looks at the actual scores, only the position. That sounds like it's throwing away information, and it is — but that's the whole point.

When you actually need it

RRF earns its place the moment you're combining two retrieval methods whose scores aren't comparable.

The classic case, and the one I ran into, is dense vector search vs. sparse/keyword search. A dense retriever gives you cosine similarities that live somewhere in a tight, roughly-bounded range. A sparse retriever (BM25-style) gives you scores that depend on term frequency and document length and can swing wildly depending on the query. If you try to just add those two numbers together, or pick some weighting between them, you're comparing apples to oranges and the result is unstable — the weighting that works for one query type quietly breaks for another.

RRF sidesteps that entire problem. It doesn't care that a 0.82 cosine score and a 14.3 BM25 score aren't on the same scale, because it never touches either number directly. It only asks "where did this doc rank in each list," which is always comparable no matter what scoring function produced it.

So the short version: you need RRF when you have multiple retrievers producing ranked lists over the same candidate pool, and their raw scores can't be meaningfully compared or weighted against each other.

Why I needed it in my project

My retrieval pipeline is multi-KB — a query can fan out across several knowledge bases, and within each one I run both a dense search and a sparse search against the same Qdrant collection. That's exactly the situation above: two retrievers, two incompatible score distributions, same candidate pool.

But I'm also running an NVIDIA cross-encoder reranker (nv-rerank-qa-mistral-4b) after retrieval, across the pooled results from every KB. So the natural question was: if the reranker is going to look at everything anyway and produce a real relevance judgment, why not skip RRF and just throw the raw dense + sparse hits straight at the reranker?

Two reasons that came out clearly once I actually built it end to end:

Cost. The reranker is a cross-encoder — it reads the query and chunk together and does real inference per pair, which is expensive and slow. Skipping fusion and handing it every raw dense hit and every raw sparse hit roughly doubles reranker calls per KB, multiplied across however many KBs got selected. Fusing first means the reranker only sees one merged, deduplicated set per KB instead of two overlapping ones.
Different jobs. RRF and the cross-encoder aren't solving the same problem, so having both isn't redundant. RRF is cheap first-stage triage: merge two heterogeneous signals into one ordering, fast, with zero calibration needed. The reranker is expensive, high-precision final judgment — it actually reads the text and decides real relevance, and in my case also does double duty as a relevance filter, not just a re-orderer.

That second point turned out to matter more than I expected.

How I actually wired it up

The fusion step happens entirely inside Qdrant, in a single round trip per KB, using Prefetch + FusionQuery:

result = await self.qdrant.query_points(
    collection_name=collection,
    prefetch=[
        qmodels.Prefetch(query=dense_vector, using="dense", limit=top_k),
        qmodels.Prefetch(query=sparse_vector, using="sparse", limit=top_k),
    ],
    query=qmodels.FusionQuery(fusion=qmodels.Fusion.RRF),
    limit=top_k,
    with_payload=True,
)

Qdrant runs both searches server-side and fuses them with RRF before it ever comes back over the wire. So each KB costs one round trip instead of two searches plus a client-side merge — which matters when a query can fan out across several KBs in parallel.

From there, the pipeline looks like this:

Fan out to every selected KB, run dense + sparse + RRF fusion inside Qdrant, in parallel across KBs.
Pool every KB's fused top-k into one candidate list.
Send the whole pool through the NVIDIA cross-encoder reranker, once.
Sort by rerank score, drop anything under a threshold, keep the final top-N.

The part that took actual trial and error was step 4. My first instinct was to normalize the reranker's raw logits with a sigmoid and cut off at 0.5 — "probability above half means relevant," which sounds reasonable. It was wrong. The model returns a raw cross-encoder logit, not a calibrated probability, and genuinely relevant chunks routinely scored negative depending on how the query was phrased. One real example: the query "How many leaves are allowed?" against the chunk that literally contained the leave-entitlement numbers scored -5.07. A 0.5-probability cutoff would have silently thrown that away.

So instead I looked at actual score distributions across real queries against real KBs:

Relevant (even loosely phrased, not just near-verbatim matches) — landed roughly -6 to +4.96 (min observed: -5.07)
Off-topic / wrong KB — landed roughly -9 to -22 (typically -11 to -22)

There's a real gap between those two bands, and -6.0 sits in it — that's the threshold the pipeline uses now. It's explicitly a property of this model's score distribution: if the reranker model ever changes, that number has to be re-derived, not assumed.

One other thing I learned building this: don't skip reranking just because a KB's candidate pool already fits inside the final top-N. It's tempting — if you only pulled 6 chunks and you need 10, why rerank at all? But the reranker isn't only picking order, it's the thing deciding relevance. Skipping it for small pools used to let every candidate through unfiltered, including weak, off-topic matches from a KB that simply didn't have the answer. Now reranking always runs, regardless of pool size.

The short version

RRF and a cross-encoder reranker aren't competing for the same job — they're stacked, each doing the part it's actually good at:

RRF: cheap, rank-based, no calibration needed. Fuses heterogeneous first-stage retrievers (dense + sparse) into one ordering, per KB, server-side.
Cross-encoder reranker: expensive, high-precision. Reads query and chunk together, produces a real relevance judgment across the pooled multi-KB candidates, and doubles as the relevance filter.

If you're only running one retrieval method, you don't need RRF — there's nothing to fuse. If you're running two or more retrievers whose scores can't be honestly compared, that's exactly the situation RRF exists for. And if precision on the final answer matters more than raw recall, a reranker on top of that fused set is still worth the extra latency — RRF just makes sure it isn't scoring twice the work it needs to.