Scoring Guide

LLM Prover produces two types of output when you run a comparison or evaluation: scores and diagnostics. Scores tell you how well each model performed. Diagnostics tell you what in your pipeline might be affecting those scores.

This guide covers both.

How scores are calculated

Every result includes a quality score. What produces that score depends on your tier and whether you have a rubric or gold standard attached.

Heuristic score (all tiers, no rubric or gold standard)

When you run a comparison without a rubric or gold standard, the score is a heuristic – a composite of measurable signals: response length relative to prompt complexity, format compliance, and output consistency across runs. It is a useful signal for comparing models on the same prompt, but it does not measure whether the response is correct or meets your criteria.

Gold standard score (all tiers, gold standard attached)

When you provide a gold standard answer, the judge scores each response against it. The score reflects semantic similarity and factual alignment with your reference answer. This is the most reliable score type for tasks where there is a known correct output.

Rubric score (Pro and above)

When you attach a rubric, the judge scores each response against each criterion individually. The overall score is an aggregate of the per-criterion scores. Each criterion gets its own score and reasoning, visible in the Score tab of the result drawer.

Compliance rubric score (Pro+ and above)

Same as rubric scoring, but each criterion is scored against the most relevant clause retrieved from your compliance store, not against the response in isolation. If no matching clause is found, the criterion is marked unscored rather than scored as zero.

Rubric criteria quality

The most common reason for unexpected scores is criteria that are ambiguous, underspecified, or misaligned with what the prompt actually asks for.

The judge scores each response against the exact text of your criterion description. Vague descriptions give the judge room to interpret – and it will, consistently, in ways you may not expect.

Common problems and fixes:

Too vague. “Be accurate” gives the judge nothing to measure. Specify what accuracy means: “All product names and version numbers cited must match the current documentation.”

Presence vs accuracy confusion. “Pricing figures accurate” reads as an accuracy requirement – it only applies if pricing is mentioned. If you mean pricing must be present and accurate, write that: “Response must include current pricing for all mentioned tiers, and figures must match published pricing.”

Absence traps. A criterion that implies required content will score 0 if that content is absent, even if absence was not what you intended. If something must be present, say so explicitly. If it only needs to be accurate when present, say that instead.

No description. The judge scores against the description, not the criterion name. A criterion with no description leaves the judge with only the name to work from. Always add a description.

Compliance criteria. Write compliance criteria as specific questions or statements, not topic labels. “Does the response make unsupported pricing claims?” works. “Pricing” does not.

A real example of criterion ambiguity. This criterion name and description were used in a compliance evaluation:

Availability – “Feature availability stated per tier (Clause 1.1)”

The criterion scored 100. The diagnostic judge flagged it as high severity:

“The criterion description is ambiguous about whether tiers must be explicitly stated in all descriptions or if only avoiding explicit general-availability claims is required.”

The response described a feature without stating which tier it belongs to. The criterion scored it as passing because the description did not clearly say that omitting tier information is a failure. A tighter description fixes this:

“Any feature described in a response must either state which tier it is available on, or must not imply it is available to all users. A response that describes a feature without stating tier availability fails this criterion.”

Prompt quality

The judge evaluates each response in the context of the original prompt. An ambiguous prompt produces ambiguous responses, which produce inconsistent scores – not because the judge is unreliable, but because the responses are genuinely different answers to different interpretations of the same question.

Common problems:

Underspecified prompt. “Describe our product” leaves the model free to describe anything. The judge has no clear standard to score against.

Prompt and rubric misaligned. If your prompt asks for a two-sentence summary but your rubric scores for completeness, the judge will penalise responses for being brief – which is what you asked for.

Fix: Make your prompt specific about format, length, and scope. Align your rubric criteria to what the prompt actually asks for. See the Prompting Guide for how to write prompts that produce consistent, comparable results.

System prompt effects

A system prompt shapes how the model responds before your user prompt runs. The judge evaluates the response in that context. A system prompt that contradicts your rubric criteria will produce low scores even for responses that are correct given the system prompt’s instructions.

Fix: Review your system prompt alongside your rubric. They should be consistent. If you want to test the effect of a system prompt, run two evaluations – one with it, one without – and compare the scores.

Context store quality

If you are using a context store, the model answers from the retrieved chunks. The judge scores the response against your criteria, but the response can only be as good as what was retrieved.

Common problems:

Best-effort chunking. PDF and plain text files are chunked by paragraph or token boundary, not by document structure. Section headings and numbered clauses may be split across chunks, reducing retrieval accuracy.

Too few chunks. A file that produces 2 chunks instead of 20 means most of the document is unretrievable.

Terminology mismatch. If your prompt uses different terms than the document, retrieval may miss the relevant section.

Fix: Upload context documents as DOCX or Markdown where possible – these are chunked by heading structure, which preserves section boundaries. For PDFs, consider converting to DOCX first. See the RAG Store Guide for chunking strategy detail.

Compliance store quality

The compliance store is queried per criterion. For each criterion, the judge retrieves the most relevant clause from your compliance store, then scores the response against it. If retrieval fails or returns a low-confidence match, the criterion is marked as unscored – not scored as zero.

Common problems:

Unstructured compliance document. A plain text file with no headings or numbered clauses produces poor retrieval. The judge cannot identify which part of the document is relevant to a given criterion.

Criterion text does not match document terminology. If your criterion says “personal data” but your regulation document says “personally identifiable information”, retrieval may not find the right clause.

Fix: Format compliance documents with clear numbered clauses. Upload as DOCX or Markdown. Write criterion descriptions using the same terminology as the regulation document. See the Compliance Store Guide for setup detail.

A real example of a chunking problem. A compliance document contained both a compliant and a non-compliant example in the same section:

“Non-compliant: ‘LLM Prover lets you upload files and use them as RAG context.’”

Because both examples were in the same chunk, the retrieval returned that chunk for the Availability criterion. The criterion judge saw a non-compliant example in the retrieved context that closely matched the evaluated response – but still scored the response as 100. The diagnostic judge caught this:

“The retrieved context chunk shows a non-compliant example that closely matches the evaluated response yet the score was 1.00. Improve document content to separate the compliant and non-compliant examples into distinct retrievable chunks.”

The fix: split compliant and non-compliant examples into separate sections in your compliance document so retrieval returns only the relevant clause, not a mixed chunk.

Judge model selection

The default judge is the most cost-efficient eligible model. For most rubric evaluations this is sufficient. For compliance rubrics or criteria that require nuanced reasoning, a more capable judge model produces more consistent results.

Fix: In your benchmark suite or evaluation settings, select a specific judge model (Pro+ feature). Mid-to-high tier models produce more reliable criterion scoring than smaller models, particularly for compliance use cases where clause interpretation matters.

Diagnostic judge

The diagnostic judge is a secondary pass that runs automatically alongside scoring. It does not produce scores – it produces pipeline health findings: observations about what in your prompt, context, system prompt, parameters, or model behaviour might be affecting result quality.

What it flags:

Each finding has a pipeline stage (context, prompt, system_prompt, parameters, model, criterion, or scoring), a severity (critical, high, medium, or low), an observation, and a suggestion.

Here are two real findings from a compliance evaluation run:

The first finding is a criterion problem – the description was not specific enough. The second is a context problem – the compliance document structure caused the wrong chunk to be retrieved. Both produced a score of 100 that did not reflect the actual compliance posture. Neither would have been visible without the diagnostic judge.

Where to find it:

• Benchmarks: the Diagnostics tab (Findings, Heatmap, and Trends sub-tabs). Also visible as a severity badge in the Run History table and per-model row in the Table tab.
• Compare and Evaluate: in the result row drawer, inline in the breakdown section.

How it relates to scores:

The diagnostic judge is completely separate from rubric scoring. Criteria scores tell you whether the response met your standards. Diagnostics tell you why scores might be lower than expected – or why they are inconsistent across runs. Diagnostics do not change scores. They explain them.

What to do next