Your rubric is your definition of quality
A rubric (the set of weighted criteria an AI judge scores each answer against; see writing rubrics) looks like a formality, the box you tick before an evaluation runs. It is actually the most consequential choice in the whole process. The rubric is the definition of “good” that every score is measured against, and two reasonable people can write two reasonable rubrics that disagree about which answer wins.
That disagreement is not hypothetical. The run below gives the same six answers to two different rubrics and watches the ranking turn over.
These results come from specific, dated runs on defined tasks, using real models through a real pipeline. They show how each model performed on those tasks, not a general ranking of model quality, which depends entirely on your task and data. Run your own to see your numbers.
The task, and the two ways to grade it
The answers being scored all come from one task: a chain of conditional shift swaps, shown in full below. It is deliberately fiddly, with two traps that reward careful state-tracking and punish skimming.
The task
A grader can care about two different things here, and they do not have to agree. One is whether the model got the answer right: the chained conditionals are easy to misread, so a correct final schedule means the model tracked state through every step and caught the traps. The other is whether the model followed the instructions: start with the schedule, no preamble, stay under 15 lines. A model can nail the format while getting the logic wrong, or get the logic right while blowing the line limit with its working.
So the same set of answers was scored with two rubrics. The first puts almost all the weight on the correct final schedule and reasoning. The second puts almost all the weight on the formatting constraints, with logic as a minor factor.
Same answers, two rankings
Same answers, two rubrics: the score depends on what you reward
Each model’s answer was scored against both rubrics. The two bars rarely match, and in two cases they point in opposite directions.
Two results tell the whole story.
One model produced a tidy, perfectly formatted answer that happened to get the puzzle wrong. The logic-weighted rubric scored it near the bottom. The format-weighted rubric scored it near the top. Same answer, a 60-point swing, driven entirely by which rubric graded it.
Another model did the opposite: it worked the puzzle correctly but showed all its steps, running well past the 15-line limit. The logic rubric rewarded it. The format rubric, seeing a correct answer wrapped in too many lines, pushed it below the tidy-but-wrong answer above.
Neither rubric is broken. Each measures something real. For a customer-facing reply, format and brevity genuinely matter. For a calculation that feeds a downstream system, the right answer matters. The two rubrics simply encode two different definitions of quality, and a model that serves one definition well can serve the other poorly.
Decide what good means, then measure for it
Write a rubric that rewards what your task actually needs, and score every model against it. Try Pro for a week, $10.
Why this is easy to get wrong by accident
Most rubrics are not written to reward the wrong thing on purpose. It happens quietly. A criterion that rewards a step-by-step explanation also, without anyone intending it, penalizes a model that answers correctly in one line. A length limit meant to keep answers tight ends up outscoring a correct answer in favour of a shorter wrong one. A criterion written to value thoroughness quietly rewards the model that writes the most, not the one that is the most right.
The scores still come back clean and confident. Nothing flags that the rubric is measuring presentation when the intent was correctness. The ranking looks authoritative, and the wrong model gets promoted on the strength of it.
Getting the definition right
The fix is not a better model or a stricter judge. It is a rubric that rewards what the task actually needs, built with a few habits that keep it honest.
Write the criteria from the job, not the output. Start from what a good answer has to do in production (be correct, be safe, be on-brand, fit a length budget) and weight accordingly, so that if correctness is what matters, correctness carries the weight rather than presentation. Then read the per-criterion scores, not just the total, because a model scoring 90 overall can be scoring 100 on format and 40 on correctness, and the breakdown shows what the total hides. When a result looks surprising, inspect the rubric first: often the model did exactly what the rubric rewarded, which was not what you meant to ask for. And on harder tasks, score the same answers more than once, since the scoring itself carries some variance and repeated runs separate a stable signal from noise before you act on it.
The models in this test never got better or worse. The only thing that moved the ranking was the definition of quality behind it. That definition is the rubric, and it is worth as much care as the choice of model it is meant to judge.
What’s next
Rubric Guide
How to write criteria that produce reliable, consistent judge scores, and how to fix them when they don't.
Comparisons Guide
Fire a prompt at multiple models simultaneously and compare cost, latency, and output side by side.
Benchmark Guide
Save a run configuration, schedule it on a cadence, and track quality and cost drift over time.