Benchmark Guide

A benchmark is a saved evaluation that runs on a schedule. You configure it once: prompt, models, rubric, schedule. It runs automatically, producing a structured result every time. The dashboard tracks every result. The moment a model’s quality drops, costs spike, or a provider silently updates their weights, you see it within one run cycle. Without a benchmark, that change is invisible until a user notices.

When to use a benchmark

A comparison or evaluation is a snapshot. It tells you how models performed today. A benchmark answers the question that follows: is that still true next week? Model providers update weights without notice. Costs shift. Output quality changes. A benchmark running on a daily schedule catches all of these automatically. Use a benchmark when a prompt matters enough to your product that you need to know the moment something changes, not after it has already affected users.

• Monitoring a production prompt for quality regression
• Proving a model choice holds up over time, not just on the day you tested it
• Catching silent provider weight updates before they reach users
• Tracking compliance posture over time against a policy document
• Generating reproducible evidence that a fine-tuned model outperforms a baseline

Creating a benchmark

Creating a benchmark is the same as creating an evaluation, with two additions: a name (required) and a schedule. Everything else (prompt, models, system prompt, context, rubric, inference parameters) is identical to the evaluation form. The name identifies the benchmark in your history and in alerts. The schedule determines how often it runs automatically.

• Name. Required. Appears in run history, leaderboard, alerts, and exports.
• Benchmark type. Performance, Quality, or Compliance, same as evaluations.
• Prompt, models, system prompt, context, inference parameters. Identical to the evaluation form.
• Schedule. Manual only (runs when you click Run Now), Daily (Pro and above), Every 6 hours (Pro+ and above), Hourly (Enterprise).
• Judge model. Pro+ only. Defaults to cheapest eligible.
• Thresholds. Set pass/fail lines for quality, cost, latency, and efficiency. Used in the Thresholds tab and alerts.
• Reliability target. Minimum percentage of runs where all models must pass thresholds. Default 95%. Used in the Thresholds tab verdict.
• Cost estimator. Appears when a schedule and models are selected. Shows estimated per-run and monthly cost broken down by model calls, scoring, and diagnostics. Warns if the estimate exceeds your remaining monthly budget.

The benchmarks table

Every benchmark you have created appears as a row in the table. Each row shows the benchmark name, type, prompt preview, models, and schedule. Click any row to open the benchmark drawer, the main workspace for that benchmark.

• Name. With a “paused” badge if the benchmark is frozen due to a tier downgrade or a deleted dependency (rubric or store).
• Type badge. Quality, compliance, or performance, colour-coded.
• Prompt preview. First 60 characters of the prompt.
• Models. List of model names configured for this benchmark.
• Schedule. Daily, Every 6 hours, Every 15 min, or Manual.
• Date. When the benchmark was created.

Inside the benchmark drawer

Clicking a benchmark opens its drawer. Eight tabs give you different views of the same benchmark: Status, Run History, Leaderboard, Analytics, Diagnostics, Thresholds, Alerts, and Config. Status is the default for manual benchmarks. Analytics is the default for scheduled ones. Each tab loads on demand. Switching tabs does not reload the others.

• Status. Provider health table, Run Now button, judge health. Shows which models are healthy, rate limited, or erroring, and how many runs have failed thresholds.
• Run History. Every run with winner, performance spark bars, diagnostics badge, and anomaly flags. Click any row to open the full result.
• Leaderboard. Win rates and averages across all runs, sortable by any column.
• Analytics. Snapshot, Trends, Compare, and Anomalies sub-tabs. The main view for understanding how performance has changed over time.
• Diagnostics. Findings, Heatmap, and Trends sub-tabs. See the Diagnostics Guide for full detail.
• Thresholds. Pass/fail reliability report per model. Exportable as PDF.
• Alerts. Configure anomaly alert thresholds and webhook destination.
• Config. Edit models, thresholds, judge model, schedule. “New from this” carries the full config into a new benchmark. Delete removes the benchmark and all its run history.

Run History

Run History shows every run the benchmark has executed, most recent first. Each row is a run. Click any row to open the full result, the same Snapshot, Scores, and Table view as an evaluation result.

• Filter chips. Last 5 / Last 10 / Last 25 / All runs, and 7d / 30d / 90d / All time. Both sets work together.
• Winner. Best quality model for that run.
• Performance spark bars. One bar per model, sized by quality score (or cost if no scoring is attached).
• Diagnostics badge. Coloured dot with severity label: clean, low, medium, high, or critical.
• Anomaly flags. Orange dot (cost spike), yellow dot (latency regression), red dot (score drop), blue dot (model version change).
• Failed rows. Show the error message with a Retry button for transient errors, or Delete only for permanent ones.
• Refresh button. Top right. Re-fetches run history without closing the drawer.

Leaderboard

The Leaderboard aggregates every run in the selected time window into a single ranked table. Each row is a model. The columns show win rates (how often each model was cheapest, fastest, and highest quality) alongside averages for cost, latency, and quality score. This is the view that answers the question a run history cannot: across all the runs you have done, which model actually wins most often?

• Filter chips. Same Last 5 / Last 10 / All runs and 7d / 30d / All time as Run History.
• Win rate columns. Cost wins %, speed wins %, quality wins % (only shown when scoring is attached).
• Average columns. Avg cost, avg latency, avg quality.
• Runs column. How many runs contributed to this model’s stats.
• Thresholds column. Percentage of runs where this model passed all thresholds. Only shown when thresholds are configured. Colour-coded green, amber, or red.
• Sortable. Click any column header to sort. Click again to reverse.

Analytics

The Analytics tab gives you four views of your run history as charts. Snapshot shows the latest run as bar charts: cost, latency, and quality side by side. Trends plots each metric over time as a line chart, with anomaly flags highlighted and provider weight changes marked as vertical rules. Compare plots cost vs quality as a bubble chart, with bubble size representing latency. This is the view that answers which model gives the best quality for the money across all runs. Anomalies lists every flagged run with the flag type and the model it affected.

Snapshot sub-tab:

• Scorecard at top: best quality, cheapest, fastest, with trend arrows vs the previous run
• Bar charts: latest run cost, latency, and quality (one bar per model)
• Threshold reference lines on charts when thresholds are configured

Trends sub-tab:

• Sub-tabs: Quality, Efficiency (quality/cost), Cost, Latency, Tokens
• Line chart per metric, one series per model
• Anomaly flags highlighted as coloured points on the line
• Model version change events marked as blue vertical rules
• Threshold reference line on quality, cost, and latency charts

Compare sub-tab:

• Bubble chart: x-axis = avg cost, y-axis = avg quality, bubble size = avg latency
• Aggregated across all runs in the selected filter window
• Only shown when scoring is attached

Anomalies sub-tab:

• Table of every flagged run: date, model, flag type
• Flag types: cost spike, latency regression, score drop, model version change, threshold breach
• Click any row to open a detail drawer with the metric values that triggered the flag

Alerts

Alerts fire when a run crosses a threshold you define. Configure them in the Alerts tab. Each alert has a metric (quality score, cost, latency, efficiency), a direction (drops below / rises above), and a threshold value. When a run triggers an alert, LLM Prover sends a notification to your configured destination.

• Alert types. Score drop, cost spike, latency regression, model version change, threshold breach.
• Notification destinations. Email (account email by default) and webhook URL (Pro+ and above).
• Webhook payload. JSON with benchmark name, run ID, model, metric, threshold, and actual value. Use this to pipe alerts into Slack, PagerDuty, or any HTTP endpoint.
• Alert history. Every fired alert is logged in the Alerts tab with the run that triggered it.

Thresholds

The Thresholds tab shows a reliability report: for each model, what percentage of runs passed all configured thresholds? The reliability target (default 95%) is the bar. Models above it are green. Models below it are amber or red depending on how far they fall short.

• Threshold types. Quality score (minimum), cost (maximum), latency (maximum), efficiency (minimum).
• Reliability target. Set in the benchmark config. Default 95%. A model that passes thresholds on 19 of 20 runs meets the target. One that passes on 15 of 20 does not.
• Export. The Thresholds tab report is exportable as a PDF. Use this when you need to share a compliance posture report or a model selection justification with a stakeholder.

Config

The Config tab shows the full benchmark configuration and lets you edit it without recreating the benchmark. Changes take effect on the next scheduled run.

• Edit models. Add or remove models. Removing a model does not delete its historical run data. It stops appearing in future runs.
• Edit thresholds. Update pass/fail lines. Historical runs are not retroactively re-evaluated.
• Edit judge model. Change the judge model used for scoring. Takes effect on the next run.
• Edit schedule. Change cadence, switch to manual, or pause the benchmark entirely.
• Pause / Resume. Pause stops the schedule without deleting the benchmark or its history. Resume restarts it on the same cadence. See Pausing and resuming a benchmark below.
• “New from this”. Carries the full config into a new benchmark form. Use this to create a variant benchmark with one variable changed: a different rubric, a different model set, or a different schedule.
• Delete. Removes the benchmark and all its run history permanently.

Pausing and resuming a benchmark

Pause stops a scheduled benchmark from running without deleting it or its history. The benchmark stays in your table with a “paused” badge. Resume restarts it on the same cadence from the next scheduled interval.

When to pause:

• You have collected enough data for the current measurement period and want to stop consuming usage until you need a fresh read
• You are making changes to your prompt, rubric, or store and do not want runs firing during the update
• You want to use a benchmark as a periodic test bed rather than a continuous monitor

The test-bed pattern:

Not every benchmark needs to run forever. A useful pattern for active development:

1. Create a benchmark with a daily schedule
2. Run it for several days to establish a baseline and understand the data
3. Pause it to stop consuming usage
4. Resume and run a batch when you want to measure the effect of a change

This gives you reproducible, comparable measurements on demand without the ongoing cost of a continuous schedule.

How to pause and resume:

Open the benchmark drawer, go to the Config tab, and use the Pause / Resume control under the schedule setting. Pausing takes effect immediately. The next scheduled run is cancelled. Resuming schedules the next run at the next interval from the current time.

Exporting results

Export is available from the Run History tab and from individual result drawers. CSV and JSON formats are both supported. The export includes all models, all metrics, and the full output text from each run. Use this when you need to share results outside the dashboard or process the data in a spreadsheet.

The Thresholds tab report exports as PDF only. It is a formatted compliance report, not raw data.

What’s next