“The answer is wrong” is an observation. Diagnosis begins upstream.
In a retrieval-augmented system, the final answer sits at the end of several decisions: what entered the corpus, how it was parsed, which candidates were retrieved, how they were ranked, what context reached the model, and what the system decided to do when the evidence was weak.
1. Corpus and parsing failures
The correct answer cannot be retrieved if the source is missing, stale, inaccessible, or badly extracted. Common issues include broken tables, lost headings, scanned pages treated as empty text, duplicated versions, and metadata that no longer points to the original document.
Before tuning retrieval, verify that the expected evidence exists in the indexed representation and still carries the structure required to interpret it.
2. Candidate retrieval failures
The relevant passage may exist but never enter the candidate set. Vocabulary mismatch, entity ambiguity, weak query rewriting, poor chunk boundaries, and over-reliance on a single retrieval method are frequent causes.
This is where recall-oriented tests help: for a representative question, does the useful evidence appear anywhere in the top candidates?
3. Ranking and context failures
Useful evidence can be retrieved and then pushed out by re-ranking, diversity rules, context assembly, or an aggressive token budget. The system may also include too much loosely related material, making the right passage harder for the model to use.
A supported answer requires the right evidence to survive every stage. Its presence somewhere in the index is only the start.
4. Answer-support failures
The model may receive good evidence and still make an unsupported claim, combine two sources incorrectly, omit a condition, or attach a citation to the wrong statement. Evaluation should compare individual claims with the evidence shown to the model and then assess the final prose against a reference answer.
5. Decision-behavior failures
Sometimes the best answer is a clarification, an escalation, or “the supplied material lacks support for this claim.” A system forced to answer every question will convert evidence gaps into confident language.
I treat refusal quality and evidence sufficiency as first-class behavior. The system should know when it lacks the authority, context, or source support to proceed.
A small evaluation set can still be useful
A practical baseline can begin with twenty to fifty questions chosen across important workflows and failure slices. For each one, retain the expected source, acceptable answer conditions, critical claims, and expected abstention behavior. Then measure retrieval and answer behavior separately.
The value of the set comes from representative coverage and useful diagnosis. A small sample cannot prove universal quality.