Searching for information seems fairly straightforward at first. A user asks a question, the system searches the available data, and returns the results that match it best. The better the knowledge base is prepared, the easier the whole task should be.
In practice, however, different questions may require completely different approaches. Sometimes we are looking for an exact name, a device identifier, or a specific parameter. In other cases, the user describes a problem in their own words and expects the system to find a document about the same thing, even if it uses completely different terminology.
Both cases may involve the same knowledge base, but not necessarily the same search method. Instead of asking which retrieval method is “the best”, it makes much more sense to ask how several of them can be used together so that they complement one another.
BM25 and vector search see different things
The difference is easiest to see with a simple shopping example. Suppose a user searches for a specific phone model, such as iPhone 17 Pro. In this case, we do not need particularly deep language understanding — what matters most is finding exactly that model.
This is where traditional lexical search works very well. One of the best-known algorithms of this type is BM25, which builds on the mechanism of the older TF-IDF method.
TF-IDF scores text relevance based on two pieces of information: how often a word appears in a specific document and how rare it is across the entire collection. A word that appears in almost every document carries little information, while a rare term — such as an exact model name — is a very strong signal.
We can build a vector representation of documents based on TF-IDF in just a few lines:
from sklearn.feature_extraction.text import TfidfVectorizer
documents = [
"iPhone 17 Pro 256 GB night camera",
"iPhone 17 128 GB long battery life",
"Galaxy S26 Ultra camera and image stabilization",
]
vectorizer = TfidfVectorizer()
document_vectors = vectorizer.fit_transform(documents)As a result, each document is represented by a sparse vector whose dimensions correspond to terms appearing in the collection. We can then compare a query against those documents and select the ones that share the most relevant terms with it.
BM25 builds on a similar intuition. However, the score does not increase indefinitely with every additional occurrence of the same word. If a term appears ten times in a document, it does not automatically make that document ten times better than one where it appears only once. BM25 introduces term-frequency saturation and also takes document length into account. This prevents long documents from winning simply because they had more opportunities to contain the queried words.
TF-IDF and BM25 are obviously not the only approaches to traditional retrieval. Other statistical ranking models exist as well. The important part is their shared assumption: the presence of specific terms remains one of the primary relevance signals.
Now consider a different question: “Which phone is good for taking photos at night?”. The phrase “night photography” may never appear in the product description. Instead, the text might mention a wide aperture, image stabilization, a large sensor, or a dedicated low-light mode.
For a search system based purely on words, such a document may not look like a perfect match. Vector search, on the other hand, looks at the relationship between the meaning of the query and the meaning of the document. Both are transformed into numerical representations by an embedding model and then compared using a chosen similarity metric.
This means that a query and a document do not need to use exactly the same wording to end up close to each other in embedding space.
That does not mean vector search automatically replaces traditional retrieval. Product catalogs contain many things we do not want to interpret too loosely: model names, storage capacities, version numbers, or exact identifiers. iPhone 17 and iPhone 17 Pro may be semantically very close, but to the user they can represent two completely different choices.
In practice, we therefore get two different kinds of signals. BM25 is very good at answering “where do these exact words occur?”, while vector search answers “where is there content with a similar meaning?”. Combining both perspectives is where hybrid search begins.
Hybrid search — two signals, one ranking
Since lexical and vector search perform well on different kinds of queries, the natural next step is to run both at the same time. This is the core idea behind hybrid search.
For a single query, we can therefore get two rankings. The first is based on lexical matching, while the second is based on semantic similarity. The same document may rank very highly on one list and much lower on the other.
Suppose we get the following results:
Lexical search:
1. Document A
2. Document B
3. Document C
Vector search:
1. Document C
2. Document A
3. Document DDocument A performs very well in both methods, even though neither of them necessarily considers it the clear winner. C is the strongest semantic result, but only ranks third lexically. B appears high only in lexical search, while D appears only in semantic search.
Now we need to turn these two lists into a single ranking.
The first idea might be to simply combine the scores returned by both algorithms. Suppose BM25 returns a score of 12.7 for a document, while vector similarity is 0.84. This does not mean the first signal is fifteen times more important. These numbers represent completely different things and operate on different scales.
We can try to normalize them first and then assign weights to individual methods. For example, 40% of the final score could come from lexical search and 60% from semantic search. This gives us a lot of control, but it also introduces additional parameters that need to be tuned on real data and maintained over time.
Another approach is Reciprocal Rank Fusion, or RRF. Instead of looking at raw scores, it focuses on the position of a document in each ranking.
In simplified terms, the higher a document appears on a list, the larger its contribution. If it ranks highly in several rankings at once, those contributions are added together.
A simple version of this mechanism can be implemented directly:
def reciprocal_rank_fusion(*rankings, k=60):
scores = {}
for ranking in rankings:
for position, document_id in enumerate(ranking, start=1):
score = 1 / (k + position)
scores[document_id] = scores.get(document_id, 0) + score
return sorted(
scores.items(),
key=lambda item: item[1],
reverse=True,
)
lexical = ["A", "B", "C"]
semantic = ["C", "A", "D"]
print(reciprocal_rank_fusion(lexical, semantic))For our example, the result looks approximately like this:
A 0.03252
C 0.03227
B 0.01613
D 0.01587Document A wins because it appears very high on both lists. C comes right behind it — it ranked first semantically and third lexically. B and D receive contributions from only one ranking.
The value 0.03252 does not mean that the document is “3.2% relevant”. It is simply a technical score used to order the results. With RRF, what matters most is the relative order of documents.
There is also an important distinction between hybrid search and reranking. RRF is still part of retrieval. It does not re-evaluate the full document and query using a more precise model. It combines rankings that have already been produced by different retrieval methods.
Only afterwards can we take the resulting pool of candidates and evaluate it more precisely — for example with a cross-encoder. That is already reranking, which we discussed in a separate article.
In practice, hybrid search can therefore be represented quite simply:
lexical_results = lexical_search(query, top_k=50)
semantic_results = vector_search(query, top_k=50)
results = reciprocal_rank_fusion(
lexical_results,
semantic_results,
)This can still be one search engine, one database, and one user query. “Hybrid” does not necessarily mean several separate systems — it means using several independent signals when building the final ranking.
Not everything should be searched in the same way
So far, we have assumed that we are searching through a single collection of text. In real systems, however, knowledge is rarely that uniform. Alongside documentation, we may have a product catalog, parameter tables, support-ticket history, system configuration, or data retrieved in real time from an external API.
Each of these sources answers slightly different questions.
If a user wants to know how a return procedure works, the natural place to search is a document describing the process. If they ask for the current status of an order, searching documentation chunks with embeddings makes little sense — the information we need exists in a specific record and is best retrieved directly from the system that stores it.
The same applies to structured data. A question about a product that costs less than 3000 PLN, has 256 GB of storage, and is currently in stock could theoretically be interpreted semantically. A much more reasonable approach is to translate some of those requirements into ordinary conditions:
filters = {
"price": {"lt": 3000},
"storage_gb": 256,
"in_stock": True,
}Semantic search can then handle the less precise part of the query — for example, “a good phone for photography” — while the explicit requirements are handled by filters.
This distinction matters. Embeddings are very useful for representing the meaning of text, but there is no reason to ask them to resolve every logical constraint. If a user explicitly says that the price must be below 3000 PLN, the system can simply treat that as a hard condition.
This is why metadata is such an important part of a well-designed knowledge base. Information about the source of a document, its date, version, product type, or category allows us to narrow the search space before we even start comparing textual meaning.
The same piece of knowledge can therefore be retrieved on several levels. Content provides semantic similarity, lexical search finds exact terminology, and metadata narrows the results to records that are actually relevant in a given context.
Not every piece of information should be copied into a vector database either. Frequently changing data — inventory levels, current process status, or the latest price — is often better retrieved from the source system at query time.
A knowledge base may store a product description or technical documentation, but it does not need to become another copy of every system in the organization.
As a result, it starts to look less like one large collection of text chunks and more like an access layer over different kinds of information, where the retrieval method depends both on the user’s question and on the nature of the data itself.
An agent needs to choose the right retrieval method
If different kinds of information require different access methods, another decision appears: how does the system know which one to use?
The user should not have to specify that their question needs to be sent to vector search, a SQL database, or an external API. From their perspective, it is still just one question.
The agent can make that decision. Before performing the actual retrieval, it analyzes the query and chooses an appropriate route. A question about a procedure can be sent to the documentation index, a request for a specific product to hybrid search with filters, and a question about the current status of an order directly to the transactional system.
Importantly, this kind of query routing does not mean that a large language model has to make every decision.
If some cases are unambiguous, ordinary rules may be enough:
def route_query(query):
if contains_order_number(query):
return "orders_api"
if contains_product_filters(query):
return "product_search"
return "knowledge_base"There is little reason to ask an LLM whether a string that clearly looks like an order number should be sent to the order system if we can recognize it deterministically, more cheaply, and more predictably.
A language model becomes more useful when the intent can no longer be derived from a few simple features of the query. A user might write:
“Is the phone I looked at yesterday still available, and would it be good for taking photos at concerts?”
Such a question may require several steps. First, the system has to determine which product the user means. Then it needs to retrieve current availability from the product system, while camera specifications or capability descriptions may come from the knowledge base.
The choice therefore does not always end with a single route. An agent can use several sources within one task and combine the information only at the end.
At that point, it becomes more than just an interface to a search engine. It needs to determine what knowledge it needs, where to find it, and which results should be combined. Retrieval itself becomes one of the tools available to the agent rather than the entire system logic.
This also has an important architectural consequence. We do not need to force every type of data into one representation just so that the agent can use it. Documentation can remain in an index designed for lexical and vector search, structured data can remain in a database, and frequently changing information can stay in the system that is its actual source of truth.
In such an architecture, the most important capability is no longer one particular retrieval method, but the ability to choose the right method for the question. An agent does not need to know everything. It needs to know how to reach the information it currently needs.
A knowledge base is a living system
Even the best-designed retrieval system becomes useless if it works on outdated data. Documents get new versions, products change their parameters, procedures are revised, and some information simply stops being valid.
A knowledge base is therefore not something we build once and leave untouched.
This becomes particularly visible when versioned documents are indexed. If we store both version 2.1 and version 2.3 of the same manual, the two may be semantically very similar and contain almost identical terminology. From the search engine’s perspective, both are excellent candidates. From the user’s perspective, one of them may already be completely obsolete.
For this reason, while processing data it is worth preserving information that allows us to distinguish not only what a chunk contains, but also where it came from, when it was created, and what exactly it applies to.
A version number, update date, source document identifier, or validity status may be just as important as the embedding itself.
Updating such knowledge should not simply mean adding new chunks to the index. If a document has been replaced by a newer version, we need to know what to do with its previous representation. Sometimes it is enough to mark it as archived and exclude it from default retrieval. In other cases, the old material should be removed entirely.
Embeddings behave in the same way. If the contents of a chunk change, its previous vector representation no longer describes the current text. Updating a document may therefore mean repeating part of the entire process: content extraction, chunking, metadata updates, and embedding generation.
In practice, it is useful to be able to unambiguously connect indexed records to their original source. If we know that chunks A1, A2, and A3 were generated from a particular version of a document, we can update only that part of the index after the document changes instead of rebuilding the entire knowledge base.
The more data sources a system uses, the more important it becomes to define the source of truth. If the same information appears in documentation, a database, and an operational system, the agent should know which source to trust when they disagree.
Documentation may be the source of truth for how something is supposed to work, while the transactional system is the source of truth for the current status of an order.
Managing a knowledge base therefore starts to resemble traditional data engineering much more than a one-time process of “putting documents into a vector store”. We need update processes, version control, source identification, and a way to remove information that should no longer appear in search results.
A retrieval system can only find what we make available to it. If the knowledge base does not reflect the actual state of the system, even the best algorithm will be very effective at finding… an outdated answer.
Retrieval is an architectural layer, not a single algorithm
When we look at the whole process from a broader perspective, retrieval stops looking like a single query to a single database. It increasingly becomes a separate system layer responsible for deciding where to search, how to search, and which information should be passed further.
In one case, a simple lexical match is enough. In another, semantic search works better. Sometimes we run both mechanisms in parallel, add metadata filtering, and merge two rankings. For another question, the best solution may be to bypass document search entirely and query an API or database directly.
This does not mean that every system needs all of these elements from the beginning. Quite the opposite — the architecture should grow together with real needs. If simple vector search produces good results, adding routing, multiple indexes, and additional algorithms simply because they are available does not provide much value.
As the number of sources and variety of questions grow, however, the limits of a simple approach become more visible. At that point, quality no longer depends only on the embedding model or search-engine parameters.
What matters is how the data is prepared, what metadata is available, how fresh the information is, which retrieval method is selected, how filtering is applied, how rankings are combined, and which source is chosen in the first place.
This complements the idea of a knowledge base as something more than a collection of documents stored in a new format. Representing knowledge is only half of the task. The other half is building a mechanism capable of reaching that knowledge in a way that matches the question.
That is why a well-designed RAG or agentic system does not have one “magic” retrieval method. Instead, it has a set of simpler mechanisms, each suited to a particular kind of information, together with logic that allows them to be used at the right moment.
And perhaps that is the most important conclusion: a good knowledge base is not only about what we store. How effectively we can retrieve that knowledge matters just as much.