Every serious retrieval stack ends up running three engines at once, and every team arrives there by the same unhappy road. You start with vectors, because vectors are the thing the demos use. Then a user pastes an error code and the vector store hands back six documents that discuss the general feeling of that error code and none that contain it, so you bolt on BM25. Then someone asks a question whose answer is spread across four linked records that share no vocabulary, and neither the vectors nor the keyword index can compose them, so you bolt on a graph. Now you have three retrievers, three score formats, and a fusion step that someone wrote on a Friday and nobody has touched since because it is held together with normalization constants that work until they don't.
The engineered version is the one worth building, and it differs from the accreted one in two specific places. Three retrieval modes is correct. The duct tape is not. The difference between the two is entirely in how you fuse the results and where you put the reranker, and both of those have right answers that have been sitting in the information-retrieval literature for years while application teams reinvented worse versions. The earlier essays built the substrate: the metagraph from essay 01, the bi-temporal model from essay 02, the contradiction engine from essay 03. This one is about reading from that substrate at query time, when an agent needs an answer and the latency budget is measured in tens of milliseconds.
Why three, and not one
The reason no single retrieval mode wins is that the three modes fail in places that do not overlap. A dense embedding model is built to collapse surface form into meaning, which is exactly what you want when the query says "roll credentials if my signing key leaks" and the document says "procedure for rotating compromised JWT private keys." Those two sentences share almost no tokens. The vector model puts them next to each other because it has learned that they mean the same thing. That is the dense retriever doing the one thing it is uniquely good at.
Hand that same model a query like "ORA-28009 after upgrading to Oracle 19c" and watch it underperform a keyword index from 1994. The embedding does not treat ORA-28009 as a meaningful token. It treats it as a slightly unusual string near some Oracle vocabulary, and it will happily rank a general "Oracle upgrade gotchas" page above the one document that contains the exact error code and its fix. BM25 ranks that document first without trying, because BM25 rewards rare exact terms and an error code is the rarest exact term there is. Lexical retrieval owns identifiers, code, version strings, and any query where the literal characters are the point.
The graph owns a third territory that neither of the others can reach. Ask "which incidents were caused by the rollout of feature flag ff_new_router" and the answer is a traversal: start at the flag node, walk the deployed_in edges to the deployments, walk the caused edges to the incidents, collect the linked root-cause documents. No amount of semantic similarity finds that, because the answer was never a similarity question. It was a structural one, and structure is what a graph stores. The metagraph from essay 01 makes this sharper still, because the edges carry timestamps and provenance, so the same traversal can be filtered to the last ninety days or weighted by source credibility without leaving the query.
The trap is treating this as a contest. Teams spend weeks tuning the single retriever they bet on, trying to make vectors handle error codes or making BM25 understand paraphrase, when the move is to stop choosing. Run all three. Each one is allowed to be mediocre outside its territory because the other two cover for it. The engineering problem is no longer "which retriever," it is "how do I merge three ranked lists without the merge becoming the new weakest link."
The score-normalization swamp
Here is where most hybrid systems quietly break. The obvious way to combine three retrievers is to take each document's score from each engine, weight the scores, and add them up. The instinct is reasonable. The execution is a swamp, because the three engines produce scores that do not live on the same scale and were never meant to be compared.
A BM25 score is unbounded above. It grows with term rarity and document length normalization, and a strong lexical match can score 24 while a weak one scores 3. A cosine similarity is bounded to the range from negative one to positive one, and in practice the top hits cluster tightly between 0.7 and 0.9, because good embedding models make most relevant things look fairly similar. A graph score from personalized PageRank is a probability mass that sums to one across all nodes, so any individual document scores something like 0.01, with most of the distribution piled up near zero. Three numbers: 24, 0.86, 0.012. Add them with any naive weighting and the BM25 term swallows the other two whole. You can set its weight to a tenth and it still dominates, because a tenth of 24 is larger than all of 0.86.
So you reach for normalization, and the swamp deepens. Min-max scaling maps each engine's scores into a zero-to-one band per query, which sounds clean until a single outlier hit stretches the band and flattens everything else, or until a query returns three candidates and the normalization has nothing to work with. Z-score normalization handles tails better by centering on the mean and dividing by the standard deviation, but it produces negative values, assumes something about the shape of the distribution, and still needs per-engine weights that you now have to tune by hand. And every one of these schemes is fragile in the same way: the score distributions shift whenever you change anything upstream. Re-tune the ANN index for higher recall and the cosine distribution moves. Expand the corpus and the BM25 statistics move. Each shift silently re-weights your fusion, and you find out when the quality regresses in production and nobody changed the fusion code.
| Normalization scheme | What it does | Where it gives way |
|---|---|---|
| Min-max | Maps each engine's scores into a zero-to-one band, per query. | One outlier hit stretches the band and flattens everything else. A query returning three candidates leaves it nothing to work with. |
| Z-score | Centers on the mean and divides by the standard deviation, so tails behave. | Produces negative values, assumes a shape for the distribution, and still needs per-engine weights tuned by hand. |
| Any of them, over time | Holds while the upstream score distributions hold. | Re-tune the ANN index for higher recall and the cosine distribution moves. Expand the corpus and the BM25 statistics move. Each shift silently re-weights the fusion. |
The lesson the IR community learned and most application teams had to relearn is that raw scores are the wrong thing to fuse. The information you actually want is already sitting in a cleaner form, and you are throwing it away.
Reciprocal Rank Fusion, and why ranks beat scores
The cleaner form is rank. Each engine, whatever its score scale, produces an ordered list. Document D is 4th in the graph results, 6th in the vectors, 5th in BM25. Those positions carry the engine's judgment without dragging along its arbitrary units. Reciprocal Rank Fusion takes only the positions, and it is almost insultingly simple:
Walk the arithmetic, because the behavior falls out of it directly. With k at 60, a rank-1 finish contributes 1/61, about 0.0164. Rank 10 contributes 1/70, about 0.0143. Rank 100 contributes 1/160, about 0.00625. Two properties matter. The function is monotonic in rank, so a better position always helps. And it has diminishing returns with depth, so the gap between rank 1 and rank 2 is much larger than the gap between rank 100 and rank 101. The k constant sets how steep that falloff is and how much credit a document gets just for appearing at all.
The consequence is the thing that makes RRF the right default. A document that is solidly in the top ten of all three engines, without ever winning any of them, beats a document that ranked first in BM25 and then fell off the other two lists entirely. Consensus across modalities outranks a single loud match. That is precisely the behavior you want from a hybrid system, because a result that three independent retrievers all liked is more trustworthy than one that only the keyword index loved. And a strong unique hit from one modality still surfaces, because its single 1/(k+rank) term is enough to carry it into the fused top results even with nothing from the others.
The deep reason RRF sidesteps the normalization swamp is that ranks are invariant to monotonic transformations of scores. Double every BM25 score and the order within the list does not change, so the fusion does not change. Re-tune your ANN index and shift the whole cosine distribution and, as long as the relative order of the top results holds, RRF is unmoved. The fragility that made score fusion a re-tuning treadmill disappears. That is why Elasticsearch and OpenSearch adopted RRF as their hybrid fusion primitive rather than building elaborate score-normalization machinery: it holds up across heterogeneous backends and it needs exactly one parameter, which you can leave at its default.
There is a more principled score-based method worth knowing for the cases where you genuinely need weighted control, for instance when a root-cause-analysis flow should trust the graph more than the keyword index. Distribution-Based Score Fusion calibrates each engine's raw score into a probability or a quantile using historical score distributions, often through Platt scaling or isotonic regression, and only then combines them with weights. It is more work, because it needs offline calibration and ongoing monitoring, and it earns its keep only when uniform rank fusion is too blunt for the weighting you need. For the overwhelming majority of stacks, RRF at k=60 is the answer, and DBSF is the thing you reach for when you have measured a specific reason to.
| Reciprocal Rank Fusion | Distribution-Based Score Fusion | |
|---|---|---|
| What it reads | Each engine's rank positions. | Each engine's raw scores, calibrated into a probability or a quantile. |
| How it calibrates | One smoothing constant, k, defaulting to 60. | Historical score distributions, often through Platt scaling or isotonic regression. |
| What it costs | Nothing to maintain. Ranks survive any monotonic change to the scores. | Offline calibration and ongoing monitoring. |
| When to reach for it | The overwhelming majority of stacks. | When uniform rank fusion is too blunt for the weighting you need, such as a root-cause flow that should trust the graph over the keyword index. |
Where the graph earns its place in the pipeline
A mistake worth naming, because it is common and it quietly caps the quality of graph-augmented systems: treating the graph as a first-stage search engine. The graph cannot do first-stage search well, because a traversal needs somewhere to start. Given the raw natural-language query "why is inference latency spiking at night," the graph has no entry point. It does not know which node that question is about. Any document freshly ingested but not yet entity-linked is invisible to a pure traversal. The graph is brilliant at expansion and useless at cold entry, and a system that asks it to do entry gets the worst of it.
The pattern that works is seed-then-expand. The dense and lexical retrievers run first and cheaply, and their top hits become seeds: the documents and the entities they mention are mapped onto graph nodes. Then the graph does the thing only it can do. From those seed nodes it traverses out along typed edges, filtered by time and by edge type, scoring the neighborhood with personalized PageRank from the seed set with a restart probability around 0.15. It surfaces the runbook attached to the incident, the architectural decision record three hops from the service, the postmortem that shares no vocabulary with the query but sits two edges from the document the vector model found. These are the results that make graph retrieval feel like reasoning, and they are reachable only because something else supplied the entry point.
The graph then produces its own ranked list, and that list joins the dense list and the BM25 list as a third equal input to RRF. This is the part that separates the engineered version from the duct-taped one. The graph enters as a retrieval mode with its own ranking, fused on the same footing as the other two. Three lists in, one fused list out, and the graph's structural reach is now a first-class contributor rather than a bag of bonus points sprinkled on after the real retrieval finished.
Recall first, precision last
Fusion gives you a single ranked candidate list, but it does not give you the final answer, because the candidate list is tuned for recall and an agent needs precision. The two are different jobs and they want different machines. The first stage casts a wide net cheaply: BM25 returns its top 200, the dense index returns its top 200, the graph expansion returns up to 200, and RRF fuses and deduplicates them down to a candidate pool of two or three hundred. The goal of this stage is to make sure the right answer is somewhere in the pool, not to know where. Recall is the metric. The retrievers are cheap enough to run wide.
The second stage is where precision is bought, and it is bought with a model that is far too expensive to run over the whole corpus. A cross-encoder takes the query and a single candidate document, concatenates them, and runs them jointly through a transformer that attends across both at once, producing one relevance score that reflects a genuine reading of how well the document answers the query. This is dramatically more accurate than the bi-encoder cosine similarity from the first stage, and it is dramatically more expensive, because its cost is quadratic in sequence length and it has to run once per candidate. You cannot run a cross-encoder over a million documents. You can run it over the two hundred that fusion already promoted, and that is the entire point of the two-stage shape.
Between the cheap bi-encoder and the expensive cross-encoder sits a middle option that is worth understanding, because it changes the economics. ColBERT and other late-interaction models encode the query and the document into per-token embeddings and score them with a MaxSim operation: for each query token, take its best match against any document token, then sum those maxima. This preserves fine-grained token-level matching that a single pooled vector throws away, while still allowing the document embeddings to be precomputed and stored. It costs more than a plain cosine and less than a full cross-encoder, which makes it the right tool for reranking a larger pool, say the top three to five hundred, before an optional cross-encoder takes the final stretch down to the twenty results the agent actually reads.
| Stage | Mechanism | Pool size | Role |
|---|---|---|---|
| BM25 + dense + graph | sparse, ANN, PPR traversal | ~600 in | wide recall, run in parallel |
| RRF fusion | 1/(k+rank), k = 60 | ~300 | merge ranks, dedup, no score plumbing |
| ColBERT rerank | token-level MaxSim | ~60 | cheap-ish precision on a mid pool |
| Cross-encoder | joint query-doc attention | ~18 | top precision on a tiny pool |
Where the milliseconds go
An agent loop that issues several retrievals per turn cannot spend half a second on each one, so the latency budget is a design constraint rather than an afterthought. The good news is that the first stage parallelizes cleanly. BM25, the dense ANN lookup, and the graph traversal share no state and can run at the same time, which means the first-stage cost is the maximum of the three rather than the sum. A BM25 query against a tuned index lands in single-digit milliseconds. An HNSW lookup tuned for 0.95 recall at ten results lands in a handful of milliseconds. A two-hop personalized PageRank from a small seed set is the slowest of the three and still finishes in the low tens of milliseconds. Run them together and first-stage recall plus fusion costs you something like fifteen to twenty milliseconds on the critical path.
The reranker is the tail, and it is serial because it consumes the fused list the first stage produced. A ColBERT pass over a few hundred candidates runs in ten to thirty milliseconds with vector-optimized kernels. A cross-encoder over a hundred candidates, batched, on a single mid-tier GPU, adds another twenty to fifty depending on the model size and how aggressively you truncate documents. So the whole critical path, recall plus fusion plus a two-tier rerank, lands in roughly sixty to ninety milliseconds, which is inside the budget for an interactive agent and leaves room for the generation call that follows. The knobs are all visible: truncate documents to keep the cross-encoder's quadratic cost down, use a smaller reranker online and a larger one only for offline evaluation, or drop the cross-encoder entirely on latency-sensitive paths and let ColBERT carry the precision alone.
| On the critical path | What it runs | Cost |
|---|---|---|
| BM25 | A query against a tuned index. | single-digit ms |
| Dense ANN | An HNSW lookup tuned for 0.95 recall at ten results. | a handful of ms |
| Graph | A two-hop personalized PageRank from a small seed set. | low tens of ms |
| First stage, in parallel | All three at once, plus fusion. The cost is the slowest channel. | 15 to 20 ms |
| ColBERT rerank | A pass over a few hundred candidates with vector-optimized kernels. | 10 to 30 ms |
| Cross-encoder | A hundred candidates, batched, on a single mid-tier GPU. | 20 to 50 ms |
| Whole path | Recall, fusion, and a two-tier rerank. | 60 to 90 ms |
None of these numbers are exotic. They are what falls out of running three indexes you already have in parallel and putting a bounded reranker behind a bounded candidate pool. The expense is controlled by the pool size at every stage, which is why the funnel shape matters: the reranker's per-document cost is high, but it only ever sees a small number of documents, so the total stays inside budget no matter how large the corpus grows underneath it.
Without the duct tape
Assemble the pieces and the system has a clean shape that survives contact with a growing corpus. At ingestion, each document is chunked and written three ways: into the BM25 index with its fields and timestamps, into the dense index as embeddings, and into the graph as entities and typed edges. At query time, BM25 and dense retrieval run in parallel and produce two ranked lists; their top hits seed a constrained graph traversal that produces a third; RRF at k=60 fuses the three on rank alone, with no score normalization to drift out from under you; and a ColBERT-then-cross-encoder rerank stage buys the top-of-list precision the agent depends on. Three retrieval modes, one fusion primitive with one parameter, one reranking funnel with bounded cost.
What makes this engineered rather than accreted is that every join in it has a reason that holds up. RRF replaces the brittle per-engine score scaling that turned fusion into a re-tuning treadmill. The graph is treated as a retrieval mode with its own ranking rather than a heuristic bolted onto the end. The rerankers are cleanly separated second-stage components with controlled pool sizes and stated latency budgets. There is no Friday-afternoon normalization constant holding the thing together, because there is nothing for it to hold together. The score scales never meet, so they never fight.
The next essay turns from reading the substrate to the thing that accumulates inside it over time. Hybrid retrieval is how an agent reads the world model on any given turn. Episodic memory is why the world model is worth reading at all, and why a system that has been running and accumulating structured episodes for a year is a moat that a competitor starting today cannot simply buy their way past.
END OF ESSAY 10 · CONTINUE TO ESSAY 11 →