We ran a benchmark. The scores were wrong (but the models were fine).
Seven models, three runs each. The task: summarise a customer support ticket into three bullet points. The prompt and ticket are shown below.
Scores came back ranging from 0 to 83.3. GPT-4.1 Nano scored 83.3 on run 2 and 0 on runs 1 and 3. Claude Opus 5 scored 83.3, then 62.5, then 45.8, declining across three consecutive runs on a simple task.
The data looked like a model reliability problem. It was not.
Before changing any model, we ran the diagnostic. It reads every layer of the pipeline: prompt, system prompt, context, parameters, criteria, scoring. It returns specific findings with specific fixes.
20 findings across 3 runs.
| Stage | Findings |
|---|---|
| Prompt | 8 |
| Criterion | 7 |
| Scoring | 2 |
| Parameters | 1 |
| Model | 1 |
19 of 20 findings pointed to the pipeline. One pointed to a model.
The conflict the diagnostic found
The rubric had a structural problem. Here is the actual response from Claude Opus 5, run 2, scored 62.5:
What caused the problem: Not stated in the ticket. David reports only the symptoms: since last Tuesday the suite scheduled three weeks ago has not run, with no email, no result, and no error message in the dashboard. Usage shows zero runs for that day despite a daily schedule. Re-saving the schedule twice has not resolved it. No root cause is identified in the information provided.
What the next step should be: Investigate why the daily schedule is not triggering (silent failure with no error surfaced to the user or dashboard), confirm how many scheduled days have been missed since last Tuesday, and respond to David with either a fix or a decision on the refund he requested.
The next step bullet invented specific internal actions not present in the ticket: investigate why the schedule is not triggering, confirm how many days were missed.
- The Accuracy criterion penalised it for that
- The Actionability criterion rewarded it for being specific and concrete
The same bullet, two criteria, contradictory scores. The response was not wrong. The rubric was.
Two findings from the run:
Accuracy judge penalizes the next-step bullet for inventing details while Actionability judge awards it for the same details, producing contradictory scores on one response.
Suggestion: Revise the Accuracy criterion to explicitly exclude next-step content from its scope.
The Actionability criterion requires a concrete next step even when the prompt and ticket provide none, causing a 0.00 score for a response that correctly reports absence.
Suggestion: Revise the Actionability criterion to allow “not stated” when the ticket contains no next-step information.
The models were not inconsistent. The rubric was contradictory. Every model was being scored against criteria that could not both be satisfied at the same time.
The fix
Three changes to the prompt and rubric:
- Prompt: Added explicit instructions for the two ambiguous cases. “State ‘unknown’ if the ticket does not identify a cause” and “state ’not specified’ if the ticket does not state a next step.”
- Accuracy criterion: Scoped to factual claims in the first two bullets only. The next-step bullet explicitly excluded.
- Actionability criterion: Updated to allow “not specified” as a passing response when the ticket contains no next-step information.
No model changes. No parameter tuning. No prompt rewriting beyond two clarifying clauses.
Run the diagnostic on your pipeline
Find out which layer is causing your score variance before you swap the model.
Start with ProThe result
Every model scored 100 on the first run with the fixed rubric. Zero diagnostic findings.
Score before and after rubric fix -- same models, same task
GPT-4.1 Nano had scored 0 on two of three runs with the original rubric. Claude Opus 5 had been declining run over run. Both scored 100 on the first run after the fix. The 300x cost difference between the cheapest and most expensive model in the set produced identical scores once the measurement was clean.
What this means for your pipeline
When scores are inconsistent, the natural response is to rewrite the prompt. It is the most visible lever and the easiest to reach. But prompt engineering against a broken rubric is optimising the wrong thing – and the more you tune, the more you overfit to a measurement that was never correct.
The diagnostic improvement loop runs in the other direction:
- Measure first
- Diagnose before changing anything
- Fix the broken layer
In this case that was the rubric. In other pipelines it is the context store, the system prompt, or the prompt itself. The full framework covers all 13 layers and how they interact.
The finding that surfaces most often: scores are measuring how well the model navigates a contradictory specification, not how well it performs the task. Fix the specification first. The model question becomes much clearer once the measurement is clean.
This post is part of the ROI Maxxed AI framework: 13 levers for building an LLM stack where every task gets the cheapest model that delivers the quality that task needs.