Evaluations Guide

An evaluation does what a comparison cannot: it tells you whether a model’s output was actually good. A comparison gives you cost, latency, and the raw output. An evaluation adds a score, a structured judgment of whether each response met your standard, with per-criterion reasoning you can inspect and act on. If you are making a build decision, choosing a model for a production use case, or monitoring quality over time, evaluations are where that work happens.

See the Comparisons Guide for how comparisons and evaluations relate.

Evaluation types

Every evaluation runs one of three types: Performance, Quality, or Compliance. The type determines what gets scored and how. Performance is available on all tiers and measures cost and latency only, no scoring. Quality scoring attaches a standard to the run: either a gold standard answer you provide, or a rubric with named criteria. Compliance scoring is Quality scoring with an additional step: each criterion is scored against a specific clause retrieved from your compliance document, not against the response in isolation. Choosing the right type before you run saves you from re-running with the wrong scoring attached.

• Performance. Cost and latency only. No judge pass. Available on all tiers.
• Quality. Gold standard or rubric scoring. Per-criterion scores and reasoning. Pro and above.
• Compliance. Rubric scoring against a compliance store. Each criterion scored against a retrieved clause. Pro+ only. Requires a disclaimer checkbox before submit.

Running an evaluation

The evaluation form is the same as the comparison form with one addition: a Scoring section that appears when you select Quality or Compliance type. Everything else (prompt, model picker, system prompt, context, inference parameters) works identically. The Scoring section is where you attach the standard the judge will score against.

• Prompt. Use your real production prompt. Synthetic prompts flatten differences between models and produce results that do not transfer to your actual use case.
• Model picker. Same tier caps as comparisons. See the models page for tier detail.
• System prompt. Optional. Select a saved system prompt to reflect your production setup.
• Context. Optional. Starter accounts can paste context directly into the prompt field. Pro and above can attach a context store. Relevant chunks are injected automatically at run time.
• Inference parameters. Optional. Defaults are applied when not set. Setting a seed makes results reproducible across runs.
• Scoring section (Quality). Toggle between Expected answer and Rubric. Expected answer: paste the ideal response, judge scores semantic fidelity. Rubric: select a saved rubric from the dropdown.
• Scoring section (Compliance). Select a compliance rubric and a compliance store. Check the disclaimer before submit.
• Judge model. Pro+ only. Defaults to the cheapest eligible model. Mid-to-high tier recommended for compliance evaluations.

The results table

The evaluations table shows every completed run. Each row is an evaluation. The columns differ from the comparisons table: Score replaces the raw output columns, and the winner badges reflect quality and efficiency rather than cost and speed. Click any row to open the result drawer.

• Name. The evaluation name you gave it, or the prompt preview if unnamed.
• Type badge. Quality, compliance, or performance, colour-coded.
• Score. The aggregate quality score for the winning model on that run.
• Winner. The best quality model for that run.
• Performance column. Spark bars when multiple runs exist for the same evaluation name.
• Cost. Total cost of all model calls in the run combined.

Inside the result drawer

Click any row to open the result drawer. Three tabs: Snapshot, Head to Head, and Table. Snapshot shows the raw output from each model alongside cost and latency, the same view as a comparison result. Head to Head is where evaluations differ from comparisons: it shows per-criterion scores and the judge’s reasoning for each model, side by side. Table gives you a sortable breakdown of all models with the full diagnostics column.

Snapshot tab:

• Full output text per model
• Cost and latency beneath each output
• Cost projection (1k, 10k, 100k calls per month)
• “New from this” button: carries the full config (prompt, models, system prompt, context, rubric, parameters) into a new evaluation or benchmark

Head to Head tab:

• Per-criterion scores for each model
• Judge reasoning per criterion: the exact text the judge produced
• Efficiency score (quality per dollar): only appears when scoring is attached
• Winner callout at top: best quality and most efficient

Table tab:

• Sortable by score, cost, latency
• Score column: quality score, judge score, and efficiency stacked
• Diagnostics column: severity badge; click any row to open finding detail
• Threshold indicators if thresholds are configured

What the scores mean

The score column in the Table tab shows up to three numbers stacked: quality score, judge score, and efficiency. They measure different things. Quality score is the aggregate of your rubric criteria, the number that tells you whether the response met your standard. Judge score is the judge’s holistic assessment of the response independent of the rubric, a useful sanity check when a rubric score looks unexpectedly high or low. Efficiency is quality per dollar: which model gave you the most quality for the money. A model that costs twice as much but scores 15% higher may or may not be worth it. Efficiency makes that comparison explicit.

• Quality score. Rubric aggregate (0-100). Or semantic similarity to the gold standard if no rubric is attached.
• Judge score. Holistic judge assessment, shown in lighter text below the quality score.
• Efficiency. Quality per dollar. Only appears when scoring is attached.
• Heuristic score. Shown when no rubric or gold standard is attached (Performance type). A composite of length, format, and consistency signals. Useful for relative comparison between models, not as an absolute quality measure.

See the Scoring Guide for a full explanation of how each score type is calculated.

Expected answer vs rubric

Quality evaluations offer two scoring modes. Expected answer scoring is the simpler of the two: you paste the ideal response, and the judge scores each model’s output for semantic fidelity to it. It is the right choice when there is a known correct answer, a specific fact, a required format, a precise policy statement. Rubric scoring is for everything else: tasks where quality is multi-dimensional, where different criteria matter independently, or where you want to understand not just whether a response was good but which aspects of it were good and why.

• Expected answer. Paste the ideal response. Judge scores semantic similarity and factual alignment. No per-criterion breakdown. Available on all tiers with Quality type.
• Rubric. Select a saved rubric. Per-criterion scores and reasoning. Aggregate score. Optional context store for RAG-grounded scoring. Pro and above.

When to use expected answer: known correct output, factual tasks, format compliance checks where there is one right answer.

When to use rubric: tone and accuracy and completeness together, tasks with no single correct answer, anything you want to track over time in a benchmark.

The rubric gives you the breakdown that explains the score. Expected answer gives you a single number faster. Use expected answer when you know what correct looks like. Use a rubric when you need to know which part of the response was good and which was not.

See the Rubric Writing Guide for criterion patterns that produce reliable scores.

Compliance evaluations

Compliance evaluations add a third element to the scoring pass: a compliance store. Instead of scoring each response against the criterion description alone, the judge retrieves the most relevant clause from your uploaded compliance document and scores the response against that clause. The judge never uses its training knowledge of regulations. It reads the document you uploaded. This means your scores reflect your actual policy, not a model’s interpretation of what a regulation probably says.

• Requires Pro+ tier
• Requires a compliance rubric: criteria written as specific questions against your document
• Requires a compliance store: your uploaded policy or regulation document
• Disclaimer checkbox must be checked before submit
• If no matching clause is found for a criterion, that criterion is marked unscored, not zero. An unscored criterion means the document did not contain enough information to evaluate it, not that the response failed.
• The Score tab shows which clause was retrieved per criterion, so every score is traceable to a specific passage in your document

See the Rubric Writing Guide for compliance criteria patterns, and the Compliance Store Guide for document structure guidance.

Pipeline diagnostics

Every evaluation includes a diagnostic pass alongside scoring. The diagnostic judge reads your full pipeline setup (prompt, context, system prompt, parameters, and rubric criteria) and flags anything that might be affecting result quality. It runs separately from rubric scoring and does not change scores. It explains them. When a score is lower than expected, or inconsistent across runs, diagnostics is where you look first.

• Diagnostics column in the results table: green tick (clean) or severity badge (low, medium, high, critical)
• Click any row in the Table tab to see the full finding: observation, evidence, and suggestion
• Findings are tagged by pipeline stage: prompt, context, system_prompt, parameters, criterion, model, or scoring

See the Diagnostics Guide for the full severity reference and how to act on findings.

Thresholds

Thresholds let you define a pass/fail line for any metric: quality score, cost, latency, or efficiency. When a model’s result crosses a threshold, the Table tab flags it. Thresholds do not affect scores. They give you a fast visual signal for which models are in range and which are not, without having to read every number in the table.

• Available on Pro and above
• Set per evaluation or saved as a preference
• Threshold types: quality score (minimum), cost (maximum), latency (maximum), efficiency (minimum)
• Table tab shows a green dot when all thresholds are met, or an amber dot with the count of missed thresholds and breach detail on hover

Useful for enforcing a cost ceiling before committing to a model, or setting a minimum quality bar for a production use case.

From evaluation to benchmark

An evaluation is a snapshot. It tells you how models performed on this prompt, today, with this rubric. A benchmark turns that snapshot into a trend. If this prompt matters to your product (a support response template, a compliance check, a summarisation task) run it on a schedule and track every result. The moment a model’s quality drops, or a cost spike appears, you see it within one run cycle.

Hit “New from this” in the result drawer to carry the full config (prompt, models, system prompt, context, rubric, and parameters) into a new benchmark.

GPT-4o Mini drops sharply on run 5. claude-sonnet-4-6 holds steady across all six runs. Without a benchmark running, that drop is invisible until someone happens to re-run manually. By then the model has already been serving degraded responses in production.

See the Benchmark Guide.

What’s next