Scoring Troubleshooting Guide

Judge scores not what you expected?

The scoring engine has several moving parts — prompt, context, rubric criteria, compliance store, and the judge model itself. Most unexpected scores trace back to one of them.

1. Rubric criteria quality

The criterion judge scores each response against the exact text of your criterion. Vague, ambiguous, or poorly scoped criteria produce unreliable scores.

Common problems:

• Too vague — “Be accurate” gives the judge nothing to measure. Specify what accuracy means in your context: “All product names and version numbers cited must match the current documentation.”
• Presence vs accuracy confusion — “Pricing figures accurate” reads as an accuracy requirement (only applies if pricing is mentioned), but you may intend it as a presence requirement (pricing must be mentioned). Write what you mean: “Response must include current pricing for all mentioned tiers, and figures must match published pricing.”
• Prohibition not stated as prohibition — “No superlatives” is clearer than “Superlatives accurate.” If something must be absent, say so explicitly.
• No description — The judge is instructed to ignore criterion names and score only against the description. If your criterion has no description, the name is all the judge has. Always add a description.

Fix: Edit your rubric criteria to be specific, unambiguous, and explicit about whether the criterion is a presence requirement, accuracy requirement, or prohibition.

2. Prompt quality

The judge evaluates the response in the context of the original prompt. An ambiguous prompt produces ambiguous responses, which produce inconsistent scores.

Common problems:

• Underspecified prompt — “Describe our product” leaves the model free to describe anything. The judge has no clear standard to score against.
• Prompt and rubric misaligned — If your prompt asks for a two-sentence summary but your rubric scores for completeness, the judge will penalise responses for being brief — which is what you asked for.

Fix: Make your prompt specific about format, length, and scope. Align your rubric criteria to what the prompt actually asks for.

3. System prompt

If you have a system prompt attached, it shapes how the model responds — and the judge evaluates the response in that context. A system prompt that contradicts your rubric criteria will produce low scores even for correct responses.

Fix: Review your system prompt alongside your rubric. They should be consistent.

4. Context store quality (RAG)

If you’re using a context store, the model answers from the retrieved chunks. Poor chunking means the model may not have the information it needs — and the judge will score the response accordingly.

Common problems:

• Best-effort chunking — PDF and plain text files are chunked by paragraph or token boundary, not by document structure. Section headings and numbered clauses may be split across chunks, reducing retrieval accuracy.
• Too few chunks — A file that produces 2 chunks instead of 20 means most of the document is unretrievable.
• Terminology mismatch — If your prompt uses different terms than the document, retrieval may miss the relevant section.

Fix: Upload context documents as DOCX or Markdown where possible — these are chunked by heading structure, which preserves section boundaries. For PDFs, consider converting to DOCX first (Google Docs: File → Download → Microsoft Word).

5. Compliance store quality

The compliance store is queried per criterion — the judge retrieves the most relevant regulation clause for each criterion, then scores the response against it. If retrieval fails or returns a low-confidence match, the criterion is marked as unscored.

Common problems:

• Unstructured compliance document — A plain text file with no headings or numbered clauses produces poor retrieval. The judge cannot identify which part of the document is relevant to a given criterion.
• Criterion text doesn’t match document terminology — If your criterion says “personal data” but your regulation document says “personally identifiable information”, retrieval may not find the right clause.

Fix: Format compliance documents with clear numbered clauses (e.g. “1.1 Responses must not contain personal data.”). Upload as DOCX or Markdown. Write criterion descriptions using the same terminology as the regulation document.

6. Score coherence

The coherence score reflects how consistent the criterion judge’s reasoning is — not whether the scores are high or low. A low coherence score means the auditor found logical errors in the criterion reasoning (e.g. penalising a response for not mentioning something the clause doesn’t require).

Low coherence usually means:

• A criterion is being applied as a presence requirement when it’s actually an accuracy requirement (see section 1 above).
• The criterion judge drifted from the clause and scored against something else.

Fix: Review the criterion reasoning shown in the Score tab. If the reasoning is wrong, the criterion description is likely the cause — tighten it.

7. Judge model

The default judge is the cheapest eligible model. For compliance rubrics or complex criteria, a more capable model may produce more consistent results.

Fix: In your benchmark suite or evaluation, select a specific judge model (Pro+ feature). Mid-to-high tier models produce more reliable criterion scoring than smaller models.

Still not right?

If you’ve worked through the above and scores still don’t reflect your expectations, the most likely remaining cause is a mismatch between what your rubric is measuring and what the prompt is asking for. Try running the same prompt with a gold standard answer instead of a rubric — this removes criterion ambiguity entirely and gives the judge a clear reference to score against.