Rubrics

A rubric (the set of weighted criteria an AI judge scores each answer against; see writing rubrics) looks like a formality, the box you tick before an evaluation runs. It is actually the most consequential choice in the whole process. The rubric is the definition of “good” that every score is measured against, and two reasonable people can write two reasonable rubrics that disagree about which answer wins.

That disagreement is not hypothetical. The run below gives the same six answers to two different rubrics and watches the ranking turn over.