Every decision about your AI stack, backed by data.

Benchmark your prompts, models, and pipeline on a schedule. Know what changed, what improved, and what to do next.

A/B test

Did the work on your documents actually improve quality?

Same prompt, same models, two store versions. The benchmark answers the question no single run can.

store A/B · support-rag · quality score

Store v1

71

Store v2

96

+25pts: same model, same prompt, better store

quality score (0-100) Claude Opus 5 · 3 runs each

Reliability

Which models hold up across every run?

Same pipeline, same prompt, same threshold. The variation is the model. 12 runs makes the pattern visible.

pass rate · quality threshold 95 · 12 runs

Kimi K3

12/12

DeepSeek V4 Flash

10/12

GPT-4o Mini

6/12

threshold: quality >= 95 GPT-4o Mini: unreliable on this pipeline

Leaderboard

Win rate over time is a different signal than a single score.

A model that scores 91 once and 74 twice has a 33% win rate. A model that scores 88 every time has 100%. The leaderboard surfaces this.

support-v1 · leaderboard · 6 runs

Claude Sonnet 4.6

83%

GPT-4o Mini

17%

Claude Sonnet 4.6 avg quality 88.1

GPT-4o Mini avg quality 74.3

quality win rate (6 runs) avg cost: $0.00299 vs $0.00004

Win rate over time. Not just today's score.

Pro

A model that scores 91 once and 74 twice has a 33% win rate. The leaderboard shows which model wins consistently across all runs, and what it costs.

Pass rate · quality threshold 95 · 12 scheduled runs (RAG pipeline)

A/B test anything. System prompt, store, model.

Pro

Same prompt, same models, one variable changes. Run both versions on a schedule. The leaderboard tells you which variant wins consistently, not just on one lucky day.

Quality score · store A/B test (same model, same prompt)

Catch regressions before they reach users.

Pro

Providers update weights without announcing it. Set a quality floor and you get an email the moment a run breaches it. See the Monitoring Guide.

Quality score · 12 scheduled runs (threshold: 95)

Pipeline problems show up across runs, not just one.

Pro

6 of 7 findings pointed to the pipeline, not the model. The Diagnostics tab clusters findings by stage across all runs so you fix the structural problem, not the symptom.

Diagnostic findings by pipeline stage · compliance benchmark, 7 runs

Bring your model or your pipeline. Benchmark it against the field.

Pro

Register any OpenAI-compatible endpoint: a fine-tuned model, a self-hosted deployment, or a full RAG pipeline. Same prompt, same rubric, same scoring. You own the workflow. LLM Prover owns the measurement. See the BYOE Guide.

Quality score · fine-tune vs frontier models (domain benchmark)

Parameters are a lever, not just a setting.

Pro

For reliability tasks, low temperature and a fixed seed keep scores stable. For creative tasks, the opposite applies: too low and the model never produces outputs creative enough to clear your threshold. The benchmark shows you which parameter profile works for your task, not just which model.

Creative task: low temperature misses threshold, higher temperature passes

Where to start

Manual runs on all plans. Scheduled benchmarks, alerts, and BYOE start at Pro.

	Starter $49/mo	Pro $129/mo	Pro+ $199/mo	Enterprise $499/mo
Benchmark schedule	--	daily	4x daily	hourly
Alert thresholds	--	✓	✓	✓
Webhook alerts	--	--	--	✓
BYOE (models + pipelines)	--	1	5	10

Every decision about your AI stack, backed by data.

Set up your first benchmark in minutes. Pro and above.