RAG Reranking: Fix the Wrong-Chunk Problem

When your RAG system pulls up the wrong passage, the fix is usually not smaller chunks or a fancier embedding model. It is a reranker: a second model that re-reads the top candidates from your vector search against the full query and reorders them. Vector search is built for recall, finding roughly relevant text fast across a whole corpus. A reranker is built for precision, deciding which of those candidates actually answers the question. Add one when retrieval finds the right document but ranks the wrong chunk on top, which is the single most common failure operators misread as a chunking problem.

Here is the mistake I see most often. A support bot answers "how do I cancel" with the refund policy instead of the cancellation steps, and the team's first move is to re-chunk the whole knowledge base or swap the embedding model. Both are expensive, both take a week, and neither is the right lever. The refund chunk and the cancellation chunk are lexically close, both came back in the top results, and the embedding model simply ranked them in the wrong order. That is a precision failure, and reranking is the tool for it.

How a reranker reads the whole question

Your vector search uses a bi-encoder. It turns the query into a vector and compares it against document vectors that were computed ahead of time and stored in the index. That precomputation is what makes it fast enough to search millions of chunks in milliseconds, but it comes at a cost: the query and the document are never looked at together. Each is compressed into its own vector in isolation, and the match is a distance between two summaries.

A reranker uses a cross-encoder. It takes the query and one candidate document and runs them through a model in the same pass, so the model can weigh the query's words directly against the document's words and output a single relevance score. That joint attention is why it is more accurate, and why it cannot scale to your whole corpus: scoring every query against every document, one pair at a time, would be far too slow. So you use both, in sequence. The Sentence-Transformers documentation describes the standard shape plainly: retrieve about 100 candidates with the bi-encoder, then rerank with the cross-encoder and keep the top few. In production I usually retrieve 30 to 50 and rerank down to 5, which trims latency without giving up much recall.

The order matters. Retrieval decides which documents even get a chance. Reranking decides which of those wins. If the right document is not in the retrieved set at all, no reranker can save you, because it only reorders what it is given. That constraint is the whole diagnostic.

Recall or precision: which problem do you actually have?

This is the part every "how cross-encoders work" tutorial skips, and it is the only part that tells you whether to add a reranker at all. Before you touch anything, ask one question: when the answer is wrong, is the right chunk anywhere in the retrieved results, or is it missing entirely?

Pull the top 20 retrieved chunks for a handful of failing queries and look. You are sorting every failure into one of two buckets.

A recall failure means the right chunk is not in the retrieved set. The vector search never surfaced it. A precision failure means the right chunk is in the set, just ranked below a wrong one that got returned first. These have completely different fixes, and reaching for the wrong one is how teams burn a week.

Symptom	What is actually wrong	The lever
Right chunk is nowhere in the top 20	Retrieval recall: wrong embedding model, bad chunking, or the fact is split across chunks	Better embeddings, revisit chunking, or hybrid keyword plus vector search
Right chunk is in the top 20 but ranked below a wrong one	Retrieval precision: the bi-encoder ordered noisy candidates poorly	Add a reranker
Right chunk is retrieved and ranked first, but the answer is still wrong	Generation, not retrieval: prompt, or the model ignoring context	Fix the prompt, not the pipeline

Most teams that think they have a chunking problem have a precision problem. The evidence is right there in the retrieved set: the answer came back, it just came back at position six. A reranker moves it to position one. Re-chunking, by contrast, gambles the whole index on a change that may not touch the actual failure. Diagnose before you rebuild.

When a reranker earns its place, and when it does not

A reranker is worth adding when the candidate pool is noisy. That means long documents chunked into passages that read almost the same, corpora with many near-duplicate entries, ambiguous or short queries where the intent is easy to misread, and any case where you already retrieve the answer but it lands too far down the list to make the prompt. The noisier the retrieval, the more a precision pass buys you.

It is not worth adding in a few clear cases. If your knowledge base is small and a query reliably returns the one obviously correct chunk in the top two, a reranker is pure overhead. If your latency budget is brutal, such as a real-time voice agent where every hundred milliseconds shows up in the conversation, the extra model pass may cost more than it returns. And if your real problem is recall, the right chunk never being retrieved, a reranker cannot help, because it only reorders what retrieval already found. We walk through whether you even need a vector store in the first place in do you need a vector database for AI on your docs; reranking is a question you only reach once you have committed to retrieval and it is returning too much noise.

What it costs: a few hundred milliseconds and a fraction of a cent

Reranking adds one model call per query and a bounded amount of latency. A hosted reranker typically adds 100 to 300 milliseconds on a normal candidate set, since it runs one pass per candidate. You control that directly by controlling how many candidates you send: rerank 25, not 200. Retrieve wide, rerank narrow.

On price, a hosted reranker is cheap relative to what it replaces. Cohere prices Rerank per search unit, defined as one query against up to 100 documents, so a single query with a modest candidate set is one unit. As of July 2026, Cohere's current models are rerank-v4.0-fast for low latency and rerank-v4.0-pro for maximum quality. If you would rather not add a line item, open cross-encoder rerankers like the BAAI bge-reranker family run locally for free; you trade the API bill for operating the model yourself.

Here is the comparison that actually matters. The alternative to reranking is often stuffing more chunks into the prompt and hoping the model sorts them out. That is the expensive path: every extra chunk is more input tokens on every call, forever, and a longer context that the model can still get lost in. We covered why more context is not the same as better answers in AI on a long document, do not just stuff the context. A reranker lets you send fewer, better chunks, which usually costs less per query overall and produces a cleaner answer. Precision up front is cheaper than volume downstream.

How to bolt one onto a pipeline you already have

You do not need a machine learning platform to add reranking. It is one HTTP call slotted between your vector search and your model, and it fits inside an existing n8n, Zapier, or Make flow.

1. Retrieve wider than you used to. If you were fetching the top 5 chunks from your vector database and passing them straight to the model, fetch the top 30 or 40 instead. You are casting a wider net on purpose.
2. Send the query and those candidates to the rerank endpoint. It returns the same candidates with a relevance score and a new order.
3. Keep the top 5 by rerank score and pass only those to the model. The rest are dropped.
4. Everything downstream stays the same. Your prompt, your model, your output format do not change.

The step you cannot skip is measurement. Build a small evaluation set, 20 to 30 real queries with the chunk you know should win for each, and run it twice: retrieval only, then retrieval plus rerank. Compare how often the correct chunk lands in the top result, or the top 3, in each run. If reranking moves that hit rate up meaningfully, keep it. If it does not, you had a recall or generation problem and just proved it cheaply. Keeping that index honest over time is its own discipline, which we cover in stale RAG data, when AI answers from deleted docs.

How to start

Do not add a reranker on faith. Start with the diagnostic, because it is free and it decides everything. Take five queries where your RAG gives the wrong answer, pull the top 20 retrieved chunks for each, and check whether the correct chunk is in that set. If it is there but ranked low, you have found a precision problem and a reranker will fix it. If it is missing entirely, stop and fix retrieval first, because no amount of reranking recovers a chunk that was never fetched.

Once you know it is precision, the change is an afternoon: widen your retrieval, add the rerank call, keep the top few, and measure the hit rate before and after. If you are building an AI assistant on your own documents and want the retrieval layer to actually return the right passage the first time, that is the kind of work we do at custom software platforms. Tell us what the corpus and the failing queries look like at contact, and we will tell you whether the fix is a reranker, better chunking, or a different architecture entirely.