The RAG agent that passed every demo and failed in week three

We have seen this project many times. A team builds a RAG agent over its SOPs, contracts or support history. The demo goes well. Ten hand-picked questions get good answers and the agent goes live.

Three weeks later the complaints start. A policy was updated and the agent still quotes the old one. A question about one supplier returns the terms of another. Nobody can say whether the last prompt change made things better or worse, because nobody measured before it.

“The model is fine. So where is it going wrong?”

The cause is rarely the model. It is that the system was judged by impressions instead of by a test set. A RAG agent has at least two parts that fail in different ways: the retriever that finds the passages, and the generator that writes the answer. A single score called accuracy hides which one broke.

“Did it find the right passage, or just write a good answer?”

So the first thing we do is split the evaluation. For retrieval we measure whether the right passages were found (context recall), whether the passages found were mostly relevant (context precision), and how high the right passage ranked. For generation we measure groundedness, meaning every claim in the answer is supported by the retrieved text, and answer relevance, meaning the answer addresses what was asked. Each dimension gets its own rubric. A judge prompt that scores everything at once produces muddy numbers. Separate metrics give separate diagnoses.

“Whose questions are we actually testing?”

The second thing is a gold set built from the client's real material. We collect 100 to 300 actual questions from emails, tickets and operator interviews, including the awkward ones: questions the documents cannot answer, questions with outdated sources, and questions that span two documents. Domain experts write or verify the expected answer and the source passage. This set is small enough to review by hand and stays under version control.

“Can we trust the judge?”

Third, we use an LLM as a judge for scale, but we do not trust it blindly. Judge models have known biases. They tend to favor longer answers and their own outputs, and different judges disagree. We calibrate the judge against a sample that humans have scored, and we send every failure and every borderline case to a person. The judge covers volume. People cover edge cases.

“Did this change help, or does it only feel better?”

Fourth, the gold set becomes a release gate. Any change to a prompt, chunking rule, embedding model, index or base model runs the full set in CI. If groundedness or context recall drops below the agreed threshold, the change does not ship. This is the step that ends the argument about whether a change helped.

“What happens when the documents change?”

Fifth, we keep measuring after launch. We sample live traces and score them on the same metrics, segmented by use case, and alert on the trend against a rolling baseline. Source documents change, so we also test for it directly: update a document, then check that the answer changes with it and that the old version is no longer cited. Failures found in production go back into the gold set, so the next release is tested against them.

“Now we can argue with numbers instead of opinions”

In the engagements where we set this up, the pattern is the same. The first run of the gold set is uncomfortable, because it shows problems the demo hid. Most of them turn out to be retrieval problems: chunks cut in the wrong place, tables flattened into noise, missing metadata filters. Those are fixed without touching the model. Once the numbers are visible, the conversation with the operators changes from opinions to evidence, and autonomy can be widened step by step.