Diagnostics Guide

Every run in LLM Prover includes a diagnostic pass. The diagnostic judge reads your full pipeline setup (prompt, context, system prompt, parameters, and rubric criteria) and flags anything that might be affecting result quality. It does not produce scores. It explains them. When scores are lower than expected, inconsistent across runs, or just hard to interpret, diagnostics is where you look first.

What diagnostics is for

Scores tell you how well each model performed. Diagnostics tells you why. The two are separate. A run can score 100 and still have diagnostic findings – findings reflect pipeline health, not whether the response was good. The most common causes of unexpected scores are not model quality. They are prompt ambiguity, context retrieval problems, criterion descriptions that give the judge inconsistent instructions, and parameter settings that constrain output in ways you did not intend. The diagnostic judge surfaces all of these before you spend time debugging manually.

Where to find diagnostics

Diagnostics appears in three places depending on where you are in the dashboard.

Comparisons and evaluations. The results table has a Diagnostics column. Each row shows either a green tick (clean) or a severity badge (low, medium, high, critical). Click any row to open the result drawer, then switch to the Table tab for the full finding – observation, evidence, and suggestion – per model.

Benchmarks. Open a benchmark and select the Diagnostics tab. Three sub-tabs give you different views of the accumulated findings across all runs: Findings, Heatmap, and Trends. This is where you work with diagnostics at scale – across multiple runs, multiple models, and multiple pipeline stages at once.

Individual run drawer. Inside any result drawer, the Table tab shows a per-model breakdown. Each finding appears as an inline box with severity, stage, observation, evidence, and suggestion.

Severity levels

Every finding has a severity: critical, high, medium, or low. Severity reflects how likely the finding is to be affecting your scores, not how urgent it is to fix.

• Critical. Almost certainly causing incorrect or unreliable scores. Act before relying on results.
• High. Likely affecting scores in a meaningful way. Fix before using results to make decisions.
• Medium. May be introducing noise or inconsistency. Worth reviewing, especially if scores vary across runs.
• Low. Minor signal. May not be affecting scores at all.
• Clean (green tick). No findings. Pipeline setup looks healthy for that model on that run.

Pipeline stages

Every finding is tagged with the pipeline stage it came from. Understanding which stage a finding belongs to tells you immediately where to look.

Prompt. The instruction you sent to the model. Findings here typically mean the prompt is ambiguous, underspecified, or misaligned with your rubric criteria. The most common fix is adding explicit format or scope constraints – or replacing test data that contains real personal identifiers with fictional ones.

Context. Content retrieved from your context store. Findings here typically mean retrieval returned the wrong chunk, a chunk contained mixed content, or document structure is making it hard to retrieve the right section. The fix is usually restructuring the document so each chunk covers one topic cleanly.

System prompt. The standing instruction applied before your user prompt. Findings here typically mean the system prompt contradicts your rubric criteria or constrains the model in ways that conflict with what you are measuring. A system prompt that does not include a required compliance clause will produce findings on every run until it does.

Parameters. Temperature, max tokens, seed, and other inference settings. Findings here typically mean a parameter is constraining output in a way that affects scoring – most commonly max_tokens truncating responses before they can satisfy a completeness criterion.

Criterion. Your rubric criteria descriptions. Findings here typically mean a criterion is ambiguous, the requirement type is not stated, or the description gives the judge inconsistent instructions. This is the most common source of high-severity findings. A criterion that deducts for missing tier statements on a prompt that never asked for tier information will fire on every run until the description is tightened.

Model. The model’s own behaviour. Findings here typically mean the model is producing unusually short or long output, ignoring format instructions, or behaving inconsistently relative to other models on the same prompt. Low severity is common here – it is informational rather than a pipeline problem you need to fix.

Scoring. The scoring pass itself. Findings here typically mean the judge encountered ambiguity in how to apply a criterion to a specific response. This usually points back to a criterion description that needs tightening – the scoring finding is the symptom, the criterion stage is the cause.

Real examples

The findings below are from real evaluation runs. They show what the diagnostic judge surfaces and what the fix looks like.

The criterion scored 100. The response described a feature without stating which tier it belongs to. The description did not clearly say that omitting tier information is a failure. A tighter description fixes it.

The compliance document had both a compliant and non-compliant example in the same section. Retrieval returned that chunk. The judge saw the non-compliant example but still scored the response as passing. Both findings produced a score of 100 that did not reflect actual compliance posture.

See the Rubric Writing Guide for criterion description patterns that reduce ambiguity, and the RAG Store Guide for document structuring guidance that improves retrieval precision.

Diagnostics in benchmarks

In a benchmark, diagnostics accumulates across every run. The Diagnostics tab gives you three views of that history.

Findings. Every finding from every run, with stage, severity, model, and timestamp. Filter by stage, severity, or date range to identify which criterion or prompt issue is recurring most. A finding that appears on every run for the same model and stage is a pipeline problem, not a one-off.

Heatmap. Finding clusters by pipeline stage and recurrence rate. Bubble size reflects occurrence count, colour reflects severity. A large red bubble on the criterion stage means the same ambiguity is firing on most runs – fix that criterion first. A large bubble on context means your document structure needs attention before the criterion work will have any effect.

Trends. Finding counts over time by severity, stage, or model. Use this to confirm a fix worked. After you update a criterion or restructure a document, re-run the benchmark and check the Trends tab. The count for that stage should drop. If it persists, the description still has ambiguity the judge is catching.

How diagnostics relates to scores

The diagnostic judge runs separately from rubric scoring. Rubric scores tell you whether the response met your criteria. Diagnostics tells you whether your pipeline setup is giving the judge a fair chance to score accurately. A high-severity finding does not automatically mean the score is wrong. It means the score may be unreliable – the same run on a different day might produce a different result because the criterion or context is giving the judge inconsistent instructions. Reliable scores require a clean pipeline. Diagnostics is how you get there.

What’s next