We ran a benchmark. The scores were wrong (but the models were fine).
We ran the same structured summarisation task across seven models, three times each. The prompt and the support ticket it was run against are shown below.
Prompt
Context
The results looked like a model reliability problem. Scores ranged from 0 to 83.3 across the three runs, with no model producing consistent results:
- GPT-4.1 Nano: 83.3 on run 2, then 0 on both other runs
- Claude Opus 5: 83.3, 62.5, 45.8, declining on every run
The diagnostic runs on every evaluation. It reads every layer of the pipeline: prompt, system prompt, context, parameters, criteria, scoring. It surfaces findings you can inspect at any time. We looked at what it had flagged.
20 findings across 3 runs.
| Stage | Findings |
|---|---|
| Prompt | 8 |
| Criterion | 7 |
| Scoring | 2 |
| Parameters | 1 |
| Model | 1 |
19 of 20 findings pointed to the pipeline. One pointed to a model.
The conflict the diagnostic found
The rubric had a structural problem. Here is the actual response from Claude Opus 5, run 2, scored 62.5:
Response
What the customer wants: David, a Pro subscriber of about six months, wants his scheduled daily benchmark to actually run as paid for. Either a fix to the scheduling problem, or a refund covering the days the benchmarks did not run.
What caused the problem: Not stated in the ticket. David reports only the symptoms: since last Tuesday the benchmark scheduled three weeks ago has not run, with no email, no result, and no error message in the dashboard. Usage shows zero runs for that day despite a daily schedule. Re-saving the schedule twice has not resolved it. No root cause is identified in the information provided.
What the next step should be: Investigate why the daily schedule is not triggering (silent failure with no error surfaced to the user or dashboard), confirm how many scheduled days have been missed since last Tuesday, and respond to David with either a fix or a decision on the refund he requested.
The next step bullet invented specific internal actions not present in the ticket: investigate why the schedule is not triggering, confirm how many days were missed.
- The Accuracy criterion penalised it for that
- The Actionability criterion rewarded it for being specific and concrete
The same bullet, two criteria, contradictory scores. The response was not wrong. The rubric was.
Two findings from the run:
Accuracy judge penalizes the next-step bullet for inventing details while Actionability judge awards it for the same details, producing contradictory scores on one response.
Suggestion: Revise the Accuracy criterion to explicitly exclude next-step content from its scope.
The Actionability criterion requires a concrete next step even when the prompt and ticket provide none, causing a 0.00 score for a response that correctly reports absence.
Suggestion: Revise the Actionability criterion to allow “not stated” when the ticket contains no next-step information.
The models were not inconsistent. The rubric was contradictory. Every model was being scored against criteria that could not both be satisfied at the same time.
The fix
Three changes to the prompt and rubric:
- Prompt: Added explicit instructions for the two ambiguous cases. “State ‘unknown’ if the ticket does not identify a cause” and “state ’not specified’ if the ticket does not state a next step.”
- Accuracy criterion: Scoped to factual claims in the first two bullets only. The next-step bullet explicitly excluded.
- Actionability criterion: Updated to allow “not specified” as a passing response when the ticket contains no next-step information.
No model changes. No parameter tuning. No prompt rewriting beyond two clarifying clauses.
Run the diagnostic on your pipeline
Find out which layer is causing your score variance before you swap the model.
Start with ProThe result
Every model scored 100 on the first run with the fixed rubric. Zero diagnostic findings.
These results come from specific, dated runs on defined tasks, using real models through a real pipeline. They show how each model performed on those tasks, not a general ranking of model quality, which depends entirely on your task and data. Run your own to see your numbers.
Score before and after rubric fix · same models, same task
GPT-4.1 Nano had scored 0 on two of three runs with the original rubric. Claude Opus 5 had been declining run over run. Both scored 100 on the first run after the fix. The 300x cost difference between the cheapest and most expensive model in the set produced identical scores once the measurement was clean. On a harder task that requires multi-step reasoning across ambiguous evidence, that cost difference buys something real.
What this means for your pipeline
When scores are inconsistent, the natural response is to rewrite the prompt. It is the most visible lever and the easiest to reach. But prompt engineering against a broken rubric is optimising the wrong thing. The more you tune, the more you overfit to a measurement that was never correct.
The diagnostic improvement loop runs in the other direction:
- Measure first
- Diagnose before changing anything
- Fix the broken layer
In this case that was the rubric. In other pipelines it is the context store, the system prompt, or the prompt itself.
This post is part of the ROI Maxxed AI framework: 14 levers for building an AI stack where every task gets the cheapest model that delivers the quality that task needs.