RAG is easy to set up, but hard to make accurate. Many knowledge-base Q&A systems answer off the mark, and the root isn’t the LLM — it’s that retrieval only used one method. Hybrid retrieval plus re-ranking are the two puzzle pieces that push accuracy up.
The weakness of vector-only retrieval
Vector retrieval finds by “similar meaning,” but for proper nouns, model numbers, and IDs — words that need exact matching — it turns out inaccurate. A user searches “model XT-200,” and the vector search may hand you a pile of content that’s roughly related in meaning but not that model at all. Keyword matching is more reliable there. Pure-vector solutions are strong on conversational long-tail queries and weak on documents dense with technical terms. That’s a structural weakness, not something tuning fixes.
Hybrid retrieval and re-ranking
Figure: key takeaways of hybrid retrieval and re-ranking
| Step | What it does | Parameter suggestion |
|---|---|---|
| Retrieval layer | Run vector and keyword together | Fuse around top 30 |
| Re-ranking | Rerank model re-scores | Select top 3 to 5 chunks |
| Recall count | Pull more first, then fine-rank | 20 to 50 candidates |
| Evaluation | Check whether top chunks answer the question | Retest with real questions |
How to roll it out
At the retrieval layer, run vector and keyword in parallel and fuse the top N; use a rerank model to reorder those N, and feed only the top few chunks to the LLM. Make the recall count larger, say pulling 30 first, then let rerank select — steadier than only pulling 5. Before launch, look at real questions to check “whether the top retrieved chunks actually address the question.” If the material is wrong, an even stronger model is pointless. If some query type stays lopsided after fusion, tune that path’s weight rather than redoing everything. Fusion weights have no fixed value: in term-heavy scenarios raise the keyword weight; in conversational-long-tail-heavy scenarios raise the vector weight. Before launch, check recall with real queries, and adjust whichever side is off.
How to choose a rerank model
A re-ranking model isn’t better just because it’s pricier; it depends on how well it matches your language. In Chinese scenarios, specifically test re-ranking quality on Chinese queries; don’t just reuse the English default. Many open-source rerank models already do well in Chinese, and a small team can start from open source — move to heavier options only when volume grows. Small teams should use open-source rerank to run the chain through and establish a baseline before discussing upgrades. Re-ranking calls cost money and time per call, so don’t re-rank the whole base — only the few dozen recalled candidates. For a small internal knowledge base, hybrid plus a light rerank is enough; consider heavier re-ranking only at a million-document scale.
Evaluation runs through everything
Every step of hybrid retrieval plus re-ranking is quantifiable: recall rate, top-N hit rate, final answer accuracy. Retest with the same question set every time you change a parameter, and you know which step actually helped. Offline, use a fixed question set to watch recall; online, watch whether users are actually satisfied, like whether they keep asking follow-ups. Only combining both is complete: if offline passes but online still answers off the mark, the problem is usually in generation or the material itself, not retrieval. Feed newly added real questions back into the set weekly, or evaluation gets steered by its own historical questions. RAG accuracy is about 70% retrieval and 30% generation — get this step right and answer quality often visibly improves.
Online and offline evaluation
Offline, use a fixed question set to watch recall; online, watch whether users are actually satisfied, like whether they keep asking follow-ups or down-vote. Only combining both is complete: if offline passes but online still answers off the mark, the problem is usually in generation or the material itself, not retrieval. Pull a batch of real user questions into the offline set weekly; the closer the set matches production traffic, the more accurate the evaluation.
How to set the candidate count
How many to recall before reranking has trade-offs: too few misses the right one, too many slows things down and dilutes. Generally pull 20 to 50 candidates, and let rerank select the top 3 to 5 to feed the model. Tune this number with your real queries instead of copying someone else’s config — small document bases take the lower bound, million-document bases take the upper bound with heavier re-ranking.
A minimal viable pipeline
Don’t jump straight to the heaviest setup. A minimal viable pipeline: vector store plus BM25, take the top 20 from each, fuse into 30 to 40, fine-rank with a light rerank model to the top 5, then feed the LLM. On tens of thousands to a few hundred thousand documents this pipeline already shows clear gains and costs little. After it runs and you have evaluation numbers, decide whether to swap in heavier re-ranking or add a fine-ranking layer. Have a measurable baseline first; then you know whether each optimization is worth it.
Common pitfalls
Pit one: tuning only the model, not retrieval, blaming a weak model when answers are off. Pit two: too small a recall count, so the right material never enters the candidates, and no amount of rerank strength can save it. Pit three: hard-coding fusion weights, letting vectors dominate in a term-heavy base. Pit four: evaluating with someone else’s question set that doesn’t match your production traffic distribution, exposed only after launch. The fix for every pit is the same: “first check with real questions whether the top retrieved chunks address the question.” Pass that gate before talking about anything else.
