Your RAG App Retrieves the Wrong Chunk

When a retrieval-augmented answer is wrong, the model is usually innocent. Build a small evaluation set, measure hit rate and MRR, then fix chunking, hybrid search and re-ranking in that order.

A user asks your assistant a question it should be able to answer from your own documents, and gets something confidently wrong. The first instinct is to change the prompt or the model. Nine times out of ten the problem is upstream: the right passage was never retrieved, so the model was asked to answer from the wrong text.

You cannot fix what you do not measure, and retrieval is measurable on its own, without the model in the way.

Build a small evaluation set first

Thirty to fifty questions is enough to find real problems. For each one, record the question and which source passage should answer it. Take the questions from real users where you can; invented questions are too tidy and hide the messiness you are trying to catch.

[
  { "question": "How many days before a session can I cancel?",
    "expected_source": "policies/bookings.md#cancellations" },
  { "question": "Do you refund after the first class?",
    "expected_source": "policies/refunds.md#after-start" }
]

Note those two questions. They use different words from each other and probably different words from your documents, which is the whole difficulty.

Measure retrieval, not the answer

Run each question through retrieval alone and record where the expected source lands:

Hit rate at k: the share of questions where the right passage appears in the top k results. Measure at k = 1, 3 and 10.
MRR (mean reciprocal rank): the average of 1 divided by the position of the right passage. It rewards being first, not merely present.

If hit rate at 10 is poor, no prompt will save you: the text was never in the context. If hit rate at 10 is good but hit rate at 1 is poor, your ranking is the problem, not your index.

Fix one: chunking that follows the document

Fixed-size chunking at 500 characters cuts sentences in half and splits a heading from the paragraph it introduces. Two chunks that each hold half a rule will each look half-relevant, and neither will win.

Chunk on structure instead: split on headings, keep a heading with its body, and keep list items together. Then set a size range rather than a size, and overlap neighbouring chunks by a sentence or two so a rule that straddles a boundary survives in one piece.

Write the source path and heading into each chunk's metadata, and prepend the heading to the chunk text before embedding. A chunk that begins "Cancellations: you may cancel up to 24 hours before" embeds far closer to a question about cancelling than one that begins "You may cancel up to".

Fix two: hybrid search

Vector search is good at meaning and bad at exact tokens. Ask about an error code, a product name or a number, and a pure embedding search will cheerfully return something about the same topic without the code in it. Keyword search has the opposite strengths.

Run both and combine them. Reciprocal rank fusion is the simplest method that works: for each document, add up 1 / (60 + rank) over each result list, and sort by that sum. No training, no tuning, and it usually moves hit rate at 1 more than any amount of prompt editing.

Fix three: re-rank the shortlist

Retrieve twenty candidates cheaply, then score the top twenty properly with a cross-encoder re-ranker, which reads the question and the passage together rather than comparing two vectors made separately. Keep the best three or five. This is the single biggest improvement to MRR in most systems, and it costs one extra model call on a short list.

Fix four: rewrite the query before you search

Conversation breaks retrieval. "What about the Flutter one?" contains nothing to search for. Before retrieving, rewrite the question into a standalone one using the last few turns:

History: user asked about the Android course fee.
Follow-up: "What about the Flutter one?"
Rewritten: "What is the fee for the Flutter course?"

This is a small, cheap model call and it fixes a whole category of failures that look like the model losing the plot.

Only then, the generation side

With the right passage reliably in context, turn to the answer. Ask the model to quote the sentence it used, and to say it does not know when the context does not contain the answer. Show the citation in your interface: it makes a wrong answer obvious to the user instead of plausible, and it gives you the evidence to debug the next failure.

Keep the evaluation running

Put the evaluation set in your repository and run it whenever you change chunking, embeddings or the index. Retrieval quality drifts silently as documents change, and a number that was 0.9 six months ago is worth checking before you promise anyone an accurate assistant.