RAG evaluation that tells you which layer broke

A RAG pipeline can return an answer that is short, confident, and wrong. (RAG, retrieval-augmented generation, means the model answers from documents you supply as context, not from memory alone.)

One way to catch a bad answer is to score it. A separate model, acting as a judge, grades the answer against criteria you set and returns a number, high for a good answer, low for a bad one.

That number tells you the answer is bad. It does not tell you which part of the pipeline made it bad.

The model, the prompt, the system prompt, and the retrieved documents all shape the answer, and one score grades them together. Fixing the right part means knowing which one broke.

The pipeline, before any judgement

Take a small RAG pipeline of the kind any team might build, an assistant that answers employee questions from a company’s expense policy. It builds each answer from three things, the employee’s question, the policy text it retrieves as context, and a system prompt that sets how it should respond. All three are below, exactly as the pipeline received them.

Reading the three inputs, it is reasonable to suspect the prompt, which is vague and open-ended, naming no flight, no class, no duration. It is also the one input you cannot control. Employees ask what they ask, and a pipeline has to hold up against a vague question rather than wish for a better one. That suspicion is worth holding, because it points at the one thing that turns out not to be the problem.

What the pipeline actually returned

Two models answered through the same pipeline:

• GPT-4o Mini (a cheap, fast model)
• Claude Sonnet 4.5 (a stronger, more expensive one)

To score their answers, the judge needs a rubric (the set of weighted criteria that define a good answer). This one rewards answers that are correct about the policy, honest about gaps, and complete, with the weights shown here.

GPT-4o Mini answered, in full, “No.” It scored 0 out of 100. The policy does permit economy reimbursement, so a flat no is not just terse, it is incorrect. Claude Sonnet 4.5 declined to answer without more detail and scored 43.8 out of 100, higher, because at least it did not assert a false rule.

A score of 0 tells you the answer is bad. It does not tell you why, and the why is not the vague question. Alongside the score, a per-layer diagnostic (a check that attributes a low score to a specific part of the pipeline) named two causes, neither of them the prompt.

The first cause sits in the retrieved context. Two passages of the policy disagree:

• Travel Policy v2 allows business class on long flights with approval.
• The general guidelines ban business class entirely.

A reader scanning the policy for the word “flight” slides straight past the conflict.

The second cause sits in the system prompt. An instruction to always give a one-sentence yes or no leaves the model no room to surface a conflict it may have noticed. Told to be definitive, it was definitive, and it was wrong.

Changing one thing at a time

With two suspected causes, the way to know which one matters is to change one input, hold the rest constant, and score again. Attributing a score move to a single edit only works if only one thing moved.

The first edit replaces the terse system prompt with one that permits the assistant to state conditions and to flag a conflict when the policy contains one. The context keeps its contradiction. The question stays exactly as vague as before.

With that single change, the two models split in a way that is worth sitting with.

Only the system prompt changed. The vague question and the contradictory policy stayed as they were.

Claude Sonnet 4.5 climbed from 43.8 to 100. Freed to describe the conflict, it named both passages, explained that economy is reimbursable under either reading, and flagged that business class is where the policy disagrees with itself. GPT-4o Mini climbed too, from 0 to 50, but no further. It gave a fluent, confident answer drawn from one side of the contradiction and never mentioned the other. The diagnostic on the GPT-4o Mini result moved with the evidence: it stopped citing the system prompt, which was now fixed, and pointed instead at the model’s own choice to answer from a single passage while a conflicting one sat in context.

The vague question never changed, and both models improved anyway. The input everyone suspected was not holding the score down. A fix to a different layer was.

The fix the pipeline actually needed

The remaining fault was in the source. A strong model will often work around a contradiction in the context and still produce something useful, while a weaker one fails on it outright. The outright failure is the safer of the two, because it is visible. The strong model’s quiet save is the more dangerous case. It depends on the phrasing of each question, so it passes in testing and fails in production, intermittently, on inputs no one thought to try. The bad answers still reach users, and they are hard to reproduce and harder to diagnose. A broken pipeline that a good model usually hides is still a broken pipeline.

The second edit reconciles the two passages into one consistent rule, and leaves the system prompt and the question alone.

With the contradiction gone, both models scored 100, and the diagnostic reported the pipeline clean.

Each bar is one change. GPT-4o Mini only cleared the contradiction once the context itself was fixed.

GPT-4o Mini, stuck at 50 while the conflict remained, now matched Claude Sonnet 4.5. Its visible ceiling was the useful signal, marking the contradiction that Claude had quietly absorbed by scoring 100 a run earlier. The layer capping the score was never the model or the prompt. It was a contradiction sitting in the retrieved text, in plain sight, that a quality score alone would have left for a human to find by rereading the policy line by line.

Keeping the answer honest after the fix

Fixing a pipeline once is debugging. The same measurement, run on a schedule across a few comparable models, becomes something more useful during active development. It tells your own changes apart from the model’s.

A scheduled benchmark re-scores the pipeline on every run. When a score drops, the question is always the same. Did something in the pipeline change, or did the model underneath it change. Running several comparable models at once answers it.

Illustrative. At run 4 all three drop together, a change in the pipeline. At run 6 one drops alone while the others hold, a change in that model.

If every model drops together on a run where nothing in the pipeline was touched, the cause is shared, a prompt edit, a context change, a retrieval tweak that reached all of them. If one model drops while the others hold steady and nothing in the pipeline moved, the cause is that model. A provider updated it, and the update cost quality on this task.

That difference decides what to do next. A shared drop sends you to your own recent changes. A single-model drop sends you to the model, and to the decision of whether to pin a version, switch, or re-tune for it. Without the comparison, both failures look identical, a number that went down.

A quality score answers whether the output is good. The layer-level diagnostic answers which part of the pipeline to fix. A scheduled, multi-model benchmark answers whether a change was yours or the model’s. The three together turn a RAG pipeline from something you hope is holding into something you can watch.