Rubric Writing Guide

A rubric is a named, reusable scoring template: a list of criteria, each with a description and an optional weight. You create it once and attach it to any evaluation or benchmark. The judge scores each model response against each criterion independently, producing a per-criterion score and reasoning you can inspect in the Score tab.

The quality of your scores depends almost entirely on the quality of your criteria. This guide covers how to write criteria that produce reliable results, the four patterns that cause problems, and how to test a rubric before you rely on it.

Descriptions are prompts to the judge

The judge scores each response against the exact text of your criterion description. It does not infer intent from the criterion name. It does not apply common sense about what you probably meant. It reads the description and scores accordingly.

This means a criterion description is a prompt. The same rules that make a comparison prompt produce consistent results apply here: specificity produces reliability, vagueness produces noise.

A criterion named “Accuracy” with no description leaves the judge with one word to work from. It will score something, but what it scores will vary across runs and across models in ways you cannot predict or control. A criterion named “Accuracy” with a description that says “All product names, version numbers, and pricing figures cited in the response must match the current documentation exactly” gives the judge a clear standard. Scores become consistent and inspectable.

The four patterns that produce unreliable scores

1. Requirement type not stated

Every criterion is one of four types: a presence requirement (“X must be stated”), an accuracy requirement (“X must be accurate if stated”), a prohibition (“Y must not appear”), or a conditional (“if A is present, B must also be present”). The judge infers the type from your description. Vague descriptions produce inconsistent inferences.

The vague version reads as either a presence requirement or an accuracy requirement depending on how the judge interprets it. One run scores a response 0 because pricing was not mentioned. The next run scores it 100 because pricing was not mentioned and the criterion only applies when it is. Both inferences are defensible from the text. Neither is what you wanted.

2. Absence traps

A criterion that implies required content will score 0 if that content is absent, even if absence was not your intent. “Response includes a summary of next steps” will penalise any response that does not include next steps, regardless of whether the prompt asked for them.

If something must be present, say so explicitly. If it only needs to be accurate when present, say that instead. The distinction matters and the judge will not assume it.

3. No description

The judge scores against the description field, not the criterion name. A criterion with no description leaves the judge with only the name. Always add a description, even for criteria that seem self-explanatory. “Concise” means different things to different readers. “Response must be 150 words or fewer” does not.

4. Compliance criteria written as topic labels

Compliance criteria need to be written as specific questions or statements, not topic labels. The judge retrieves a clause from your compliance store and scores the response against it. A topic label gives the retrieval step nothing to work with.

“Pricing” does not retrieve a useful clause. “Does the response make pricing claims that are not supported by the uploaded pricing documentation?” retrieves the relevant section and gives the judge a clear scoring standard.

Good and bad examples

The table below covers five common criterion types. Each shows the vague version, the problem it causes, and a specific rewrite.

Requirement type – presence, accuracy, prohibition, or conditional – should be explicit in every description.

Compliance criteria and the compliance store

Compliance rubrics work differently from quality rubrics. Instead of scoring against the criterion description alone, the judge retrieves the most relevant clause from your compliance store and scores the response against that clause. The judge never relies on its training knowledge of regulations. It reads the actual document you uploaded.

This has two implications for how you write compliance criteria.

First, use the same terminology as your regulation document. If your criterion says “personal data” but the regulation says “personally identifiable information”, retrieval may not find the right clause. The criterion will be marked unscored rather than scored as zero – but an unscored criterion is not a passing criterion. Match your language to the document.

Second, write criteria as specific questions, not topic labels. The retrieval step uses your criterion description to find the relevant clause. A specific question produces a better retrieval result than a category name.

When a matching clause is found, the score is traceable: the Score tab shows which clause was retrieved and how the response was scored against it. If the regulation updates, re-upload the document and re-run the benchmark. Scores reflect the new version automatically.

See the Compliance Store Guide for how to structure your compliance documents to produce reliable retrieval.

Testing your rubric

Run a single evaluation with a response you know is correct before using a rubric in a benchmark suite. If the scores do not reflect that the response is good, the criteria are the problem, not the model.

Work through the criteria one at a time. For each criterion that scored unexpectedly:

1. Read the description as if you are the judge – what type of requirement does it imply?
2. Check whether the response actually satisfies the criterion as written, not as intended
3. Rewrite the description to make the requirement type explicit
4. Re-run the evaluation

The Diagnostics tab accelerates this process. When a criterion produces an unexpected score, the diagnostic judge flags the specific ambiguity and suggests a tighter description. Check Diagnostics before rewriting criteria manually – it often identifies the exact phrase causing the inconsistency.

What’s next