Monitoring Guide

Model providers update weights without notice. Costs shift. Output quality changes. None of these changes come with a warning. The model you validated last quarter may not behave the same way today, and the only way to know is to keep measuring. LLM Prover’s monitoring layer runs your benchmarks on a schedule, compares each result against the previous run, and flags anything that changed: quality drops, cost spikes, latency regressions, and silent model version changes. This guide covers what gets flagged, how to read the flags, and how to configure alerts so the right people know the moment something changes.

What monitoring detects

Every benchmark run is compared against the previous run for the same benchmark. When a metric changes beyond a threshold, LLM Prover raises an anomaly flag. Flags are not errors. They are signals. A flag means something changed. Whether that change matters depends on your use case. The five flag types cover the full range of changes that affect production model behaviour.

• score_drop. Quality score fell by more than the configured threshold (default: any drop). The most important flag for production use cases. Indicates the model is producing worse outputs on your prompt than it did on the previous run. Common cause: silent provider weight update.
• cost_spike. Cost per call increased by more than the configured threshold (default: 20%). Indicates the model is generating more tokens, the provider changed pricing, or a model version change introduced different output behaviour. Does not mean quality improved.
• latency_regression. Wall-time latency increased by more than the configured threshold (default: 50%). Indicates provider infrastructure changes, increased load, or a model version change that produces longer outputs.
• model_version_change. The model version string returned by the provider changed between runs. Providers update model weights silently and the version string is the only signal. A version change does not guarantee quality changed, but it means the model you validated is no longer the model running. Always check score and cost alongside this flag.
• threshold_breach. A metric crossed a configured pass/fail threshold. Different from the other flags: threshold breaches are based on absolute values you set (e.g. quality must be above 80, cost must be below $0.001), not relative changes between runs. See Thresholds below.

Where to find anomaly flags

Anomaly flags appear in three places in the dashboard, depending on how much detail you need.

Run History table. The Anomalies column shows coloured dot badges for each flag type on that run. Orange dot = cost_spike. Yellow dot = latency_regression. Red dot = score_drop. Blue dot = model_version_change. Red siren = threshold_breach. Hover any badge to see the flag type and the model it affected. Click the row to open the full run result.

Snapshot tab (inside a run result). When a run has anomaly flags, a collapsible Anomalies section appears below the model outputs. Each flag is listed with the model it affected, the metric values that triggered it, and a comparison to the previous run.

Anomalies sub-tab (Analytics tab). The dedicated anomaly view across all runs. Every flagged run in the selected time window, with flag type, model, and date. Click any row to open a detail drawer showing the metric values that triggered the flag and the previous run values for comparison.

Thresholds

Thresholds are pass/fail lines you set on a benchmark. Where anomaly flags detect relative changes between runs, thresholds detect absolute values, such as a quality score below 80, a cost above $0.001, or a latency above 3000ms. When a model’s result crosses a threshold on any run, a threshold_breach flag is raised and the Thresholds tab updates its reliability verdict for that model.

Set thresholds in the Config tab of any benchmark drawer, under “Thresholds”.

• Quality min. Minimum acceptable quality score (0-100).
• Judge min. Minimum acceptable judge score (0-100).
• Cost max. Maximum acceptable cost per call (USD).
• Latency max. Maximum acceptable latency (ms).
• Efficiency min. Minimum quality per dollar.
• Reliability target. The percentage of runs where all models must pass all thresholds to be considered reliable. Default 95%. Used in the Thresholds tab verdict.

The Thresholds tab shows: overall verdict (met / not yet met), per-model reliability %, run count, and breach breakdown. Click any model row to see a pass/fail timeline chart and per-run per-threshold breakdown. The Performance Report (PDF export) is available from the Thresholds tab. It shows the verdict, pass criteria, and per-model results.

Configuring alerts

Alerts send a notification when an anomaly flag fires or a threshold is breached. Without alerts, you only see flags when you open the dashboard. With alerts, the right person knows within minutes of the run completing. Alerts are configured per benchmark from the Alerts tab in the benchmark drawer.

1. Open a benchmark drawer and select the Alerts tab
2. Toggle “Enable anomaly alerts” on
3. Toggle “Alert on threshold breach” if you want threshold breaches to also trigger notifications
4. Set per-metric sensitivity (leave blank to disable alerting for that metric):
   • Cost spike (%). Alert when cost increases by more than this % vs the previous run (e.g. 20)
   • Latency regression (%). Alert when latency increases by more than this % (e.g. 50)
   • Quality drop (%). Alert when quality score drops by more than this % (e.g. 15)
5. Optionally enter a Webhook URL (Enterprise only). A POST payload is sent to your URL on every alert; use for Slack, PagerDuty, or any webhook receiver
6. Click Save. The change takes effect on the next run

Acting on flags

A flag tells you something changed. It does not tell you whether the change matters or what to do about it. That depends on the flag type and your use case.

Monitoring over time

Individual flags tell you something changed on a specific run. The Analytics tab gives you the longer view: trends across all runs, anomaly history, and a cost vs quality scatter that shows which model has been most reliable over the full benchmark history. Use this when you need to make a model decision based on evidence, not a single data point.

• Analytics tab – Trends sub-tab. Quality, cost, latency, and efficiency (quality/cost) over time. Anomaly flags highlighted as coloured points. Model version changes marked as blue vertical rules.
• Analytics tab – Anomalies sub-tab. Every flagged run in the selected time window, filterable by date range.
• Analytics tab – Compare sub-tab. Cost vs quality bubble chart averaged across all runs. Bubble size = latency. The view that answers which model has been most reliable and efficient over time.
• Leaderboard tab. Win rates and averages across all runs. Threshold pass rate column when thresholds are configured.

GPT-4o Mini drops sharply on runs 5-6 (a model version change on run 5) then recovers on run 7. claude-sonnet-4-6 is flat throughout. Without a benchmark running, the drop is invisible until a user notices.

See the Benchmark Guide for how to set up scheduled runs, and the Diagnostics Guide for pipeline health findings that appear alongside anomaly flags.

What’s next