Anthillo /blog
← All posts

After Search Comes Selection

Mikołaj Wojciuk

Software and Data Engineer ·

We have all probably heard the enigmatic-sounding acronym – RAG. Retrieval-Augmented Generation has taken the world of AI systems built to work with our documents by storm. There are several good reasons for that, the most important being that an approach based on the semantic similarity of texts is very effective at finding passages related to users’ questions.

The problem is that “related to the question” does not mean “the best answer to it”. In a real system, finding good candidates is only the first step – after that you have to decide which of them are actually worth trusting. So let’s look at where the problem lies and, more importantly, how to solve it.

The first ranking is only a proposal

Suppose a user asks a question against a knowledge base containing a few thousand pages of documentation. The vector search engine does its job and a moment later returns ten passages. They all look reasonable: they concern the right product, use similar terminology and revolve around exactly the same topic.

At first glance, everything works.

In semantic search, every document passage is represented by a vector that tries to capture its meaning in a high-dimensional space. The user’s query goes through exactly the same process, and the algorithm compares the vectors using a chosen metric – most often cosine similarity. Put very simply: the closer two texts are in meaning, the higher the score they get and the higher the passage lands in the ranking. The catch is that this number primarily describes semantic similarity, not whether the document contains the best answer to the question asked.

Only when we look at the results up close does it turn out that the first passage describes an older version of the system, the second merely mentions the parameter we care about, and only the third contains the exact answer.

Order matters fundamentally here. In a practical system we won’t pass every retrieved passage to the language model. That would be slow, expensive, and would exhaust the available context window in no time. Out of dozens or hundreds of candidates we pick only a handful of the best – say the Top 5 – and only those make it into the prompt.

This means that, from the model’s point of view, a correct answer sitting at position 30 might as well not exist. The retriever found it, but the system will never use it. This reveals one of the less obvious properties of search: finding the right information is not enough – it also has to end up high enough on the list.

The first stage of search has to be fast. Its goal is to sift through a huge collection and produce a sensible list of candidates. The ranking it returns is therefore better treated as a first proposal than a final verdict. Especially when the difference between a correct and an incorrect answer lies not in the main topic but in a small detail: a product version, a date, a condition, an exception, or a single parameter.

This is exactly what creates the need for a second look at the results.

Semantic similarity versus actual relevance

Vector search is great at answering the question: “Which passages talk about similar things?”. It is much worse at answering: “Which of these passages best answers exactly what the user asked?”. It’s a subtle but crucial difference.

This is where we need to introduce the notion of relevance – how well a result actually matches a specific query. Two documents can be very similar semantically yet differ dramatically in terms of relevance. If we ask about configuring version 2.3 of a system, the documentation for version 2.1 may look like its twin topically, but it remains a useless answer.

We run into cases like this in practice all the time. In technical documentation, procedures often differ only by device model. In e-commerce, two products can belong to the same category and have similar descriptions, but only one of them meets the user’s specific requirement. To an embedding model, both results sit right next to each other. To a human, the difference is fundamental.

Negations and exceptions can be even more treacherous:

“Feature X is supported in version 3.2."
"Feature X is not supported in version 3.2.”

Semantically, these sentences are almost identical. In terms of the answer, they are polar opposites.

That is why a similarity score is not a measure of correctness. A score of 0.86 does not mean a document is “86% right”. It only describes a mathematical relationship between texts. Since the first stage of search cannot assess relevance precisely enough, we need a mechanism that orders the retrieved candidates by a different criterion.

That mechanism is called a reranker.

The retriever searches broadly, the reranker judges precisely

The division of responsibilities here is very natural. The retriever (the search module) has to be lightning fast and, above all, not miss anything useful. Its job is not perfect ordering, but fishing out a good set of candidates from among millions of passages.

It’s worth recalling two classic metrics here: recall and precision. Recall tells us what fraction of all the relevant results in the collection we managed to find. Precision, in turn, answers the question of what fraction of the returned results actually hit the mark.

If the collection contains five passages that could answer the question and the retriever finds four of them, we achieve high recall. But if it throws in several dozen completely irrelevant ones alongside those four, the precision of that list will be terrible.

At the first stage we can afford that. What matters most is getting the right information into the net – if it’s missing now, the reranker will have nothing to pull to the top of the list later.

The reranker’s job is different. It no longer searches the whole collection. It receives a ready-made shortlist and precisely evaluates the relationship between the question and each candidate individually. Its goal is to boost precision at the very top of the ranking.

In short. The retriever searches a large collection and builds a broad pool of candidates, caring above all about high recall. The reranker works on that limited pool and improves precision at its top before the selected passages reach the LLM’s context.

A language model has no magical power to reconstruct information it wasn’t given. If five excellent passages make it into the prompt, the model has a solid basis for reasoning. If it gets outdated or wrong material, it will produce an equally poor answer. The age-old rule of data engineering applies: garbage in, garbage out.

So reranking is not just “cosmetics”. It directly affects the quality of the data the LLM sees, and thus the usefulness of the entire system. Because the candidate pool has been narrowed down, at this stage we can use a slower but much more accurate model. Forcing that level of precision across the whole collection would simply paralyse the system.

It’s a classic architectural trade-off: the first stage fights for recall, the second raises precision where it matters most.

How to turn Top 100 into Top 5

Implementing reranking starts with a key decision: how many results do we let through to the second stage? The retriever might return, say, top_k = 100, creating a candidate pool – a set of promising passages.

The top_k parameter at the search stage is not the same as the final number of documents sent to the LLM. The former can be 100 or 200, while only 5 end up in the prompt.

The flow looks like this:

Query – goes to the search engine.
Retrieval – returns, for example, 100 candidates.
Reranking – computes a new relevance score for each candidate.
Sorting – reorders the list by the new scores.
Top 5 – goes to the LLM as context.

At the reranking stage, each candidate is re-scored independently against the query:

Example relevance scores
fragment_1    0.91
fragment_2    0.37
fragment_3    0.84

After this step, the previous order goes in the bin. A passage that was in 34th place for the retriever can jump to second place if the more accurate model decides it perfectly matches the user’s intent.

This brings us to the rerank window – the number of candidates passed through the second model. The wider the window, the better the chance of finding a hidden gem. Keep in mind, though, that every additional pair has a computational cost.

Setting top_k = 5 directly on the retriever is a beginner’s mistake. If the right answer had been in 17th place, we would cut it off before the reranker ever got a chance to look at it. It’s much safer to cast a wide net and only then carefully sort through the catch.

How to choose: rules, models and LLMs

Before we look at rerankers themselves, we need to clearly separate them from hybrid search. They are two distinct things, even though they often work side by side.

Hybrid search is still part of the retrieval stage. It runs lexical search (e.g. BM25, ideal for catching specific IDs or proper names) and vector search (which captures context) in parallel. The problem arises when merging the two lists. The raw scores of the two algorithms live on completely different scales. This is where RRF (Reciprocal Rank Fusion) helps – an algorithm that ignores the score values themselves and focuses on the documents’ positions in both lists. The higher a result ranks in both lists, the higher it lands in the merged set.

Only at this point, with a curated set of candidates, does the reranker step in. What technologies can we use?

Cross-encoders. In classic vector search, the question and the document are encoded separately. A cross-encoder receives both texts at once and returns a relevance score. This lets the model see dependencies: specific numbers, negations, or matching names. Examples include the ms-marco models and BGE Reranker. They are computationally expensive, but excellent at the shortlist stage.

Learning to Rank (LtR). Instead of analysing the text itself, the model evaluates a set of concrete features: the BM25 score, vector similarity, publication date, exact match, source authority. Algorithms such as LambdaMART learn to combine these signals from training data (e.g. user clicks).

LLM-based scoring. A language model can receive the list of candidates and sort them itself. This offers the greatest flexibility – we can instruct it in the prompt: “Prefer the current version of the procedure and reject documents about older devices”. The price, however, is time, token usage and inference cost.

There is no single universal reranker. The choice depends on the requirements of a specific project.

Quality versus latency and cost

Reranking improves quality, but it takes a toll on every query. The retriever searches the collection in fractions of a second, relying on optimised indexes. The reranker forces the system to do heavy work live.

The trade-off is obvious: a rerank window that’s too small limits the system’s potential, while one that’s too large drastically increases latency. A lightweight cross-encoder can be fast (especially with batching), but a powerful LLM analysing the results will add noticeable seconds to the response time.

That’s why it pays to think in terms of a latency budget – the maximum time within which the system must respond before the user loses patience. Part of the budget goes to retrieval, part to ranking, and the rest to text generation. In e-commerce search, an extra 300 milliseconds is a huge overhead; in an enterprise analytics assistant, it’s a perfectly acceptable price for confidence in the data.

Advanced solutions use cascading approaches: first a broad retriever, then a lightweight, fast reranker that cuts the pool down to, say, 10 results, with a heavy LLM invoked only for the most complex queries.

Reranking is not a free “upgrade” – it’s an investment that has to make architectural sense.

Good ranking is part of the architecture

A reranker is not a magic wand. If the right file isn’t in the candidate pool, the best algorithm in the world won’t conjure it out of thin air.

The quality of a RAG system has to be assessed holistically. The retriever takes care of recall, the reranker boosts precision, and the LLM neatly synthesises whatever we hand it. This is also why a benchmark of a single embedding model says so little about the final system. The best LLM and an outstanding cross-encoder will fail if we neglect good chunking of documents and data cleaning.

In real-world projects, the key to success is not finding the “ultimate model”, but orchestrating the entire processing pipeline. Ranking is a full-fledged part of the architecture that decides what fuel the AI will ultimately run on. In a well-designed system, no single component has to be perfect – it’s enough that each does its job reliably enough to make the next one’s work easier.

← All posts