Getting Started

Why LLM Prover

LLM model behaviour changes over time. Providers update weights, costs shift, and outputs that passed review last quarter may not perform the same way today. A measurement layer makes those changes visible before they affect users.

LLM Prover is the measurement layer that closes that gap. Here is what teams use it for:

• Picking the right model for your use case. Provider benchmarks are not run on your prompts, your data, or your quality standard. A comparison run on your actual prompt tells you which model wins for your specific task, and what each one costs per call.
• Proving a cheaper model is good enough. For many everyday tasks (summarisation, classification, simple RAG) open source models match frontier quality at a fraction of the cost. Running the comparison properly is what takes time. LLM Prover makes it fast.
• Scoring outputs against your own criteria. Comparisons tell you which model was fastest and cheapest. Evaluations tell you whether any of them actually met your standard: accuracy, tone, format compliance, policy adherence. Different question, more useful answer.
• Checking text you already have against a standard. When the copy, answer, or clause is already written, a scores run grades it against your rubric or compliance store and quotes the exact phrase behind every mark. Score a finished draft before it ships, or rank several versions to pick the strongest.
• Catching model drift before users do. Model providers update weights silently. The model you validated last quarter may not behave the same way today. Scheduled benchmarks run your prompt on a cadence and alert you the moment quality drops, costs spike, or a model version changes.
• Proving your fine-tuned model is better. If you have trained a domain-specific model, BYOE lets you register your endpoint and benchmark it head-to-head against any model in the registry. The result is reproducible, exportable evidence. Not anecdote.
• Measuring compliance posture over time. Upload your regulations, policies, or standards as a compliance store (Pro+ and above). The judge scores every response against the most relevant clause in your document. Scheduled benchmarks catch when a model update breaks your compliance posture before it reaches users.
• Diagnosing inconsistent outputs. When outputs are unpredictable and swapping models hasn’t helped, the pipeline diagnostics judge reads your full setup (prompt, context, system prompt, rubric criteria) and tells you exactly which layer is the problem. Most of the time it is not the model.
• Giving your team a benchmarking budget. Every account is an org. Invite members, control per-member spend, and give engineers a proper tool with a budget you set. The owner sees all usage in one place.
• Letting your AI agents benchmark other AI models. The MCP server exposes over 40 tools. Any MCP-capable agent – Claude Desktop, Cursor, VS Code Copilot, Amazon Q – discovers them automatically and can run comparisons, evaluations, scoring, benchmarks, and preset-driven runs without leaving its workflow. No integration code required.
• Automating quality measurement in your pipeline. Every operation is available via REST API. Fire comparisons, evaluations, and benchmark runs from any script, CI workflow, or application. Developer keys on all plans.

This guide covers the two concepts that underpin everything in LLM Prover: run types and config. Run types are the ways to execute: Comparisons, Evaluations, Scores, and Benchmarks. Config items are the inputs you attach to runs to make results more specific, repeatable, and meaningful. None are required for a first run.

The dashboard menu reflects this split. The top section (Run) is where you execute. The bottom section (Config) is where you set up the inputs that feed into runs.

Run

Comparisons give you a snapshot. Evaluations score a model’s output against your standard. Scores grade text you already have. Benchmarks track results over time. Start with a comparison, add a rubric when you need scoring, score finished text directly when the words already exist, and turn any of it into a benchmark when you need to track drift.

Comparisons

Comparisons fire a single prompt at multiple models simultaneously and return a structured side-by-side result in seconds. Each model gets a cost, latency, and token count. The fastest and cheapest model are highlighted automatically.

The animation above shows 4 models. The charts below are from a broader run across 7 models on the same prompt type.

GPT-4o Mini and DeepSeek V4 Flash are the cheapest by a wide margin. Qwen 3.6 27B is the fastest at 980ms. GPT-4o and Claude Opus 5 are the most expensive. Whether that cost is justified depends on whether their output quality meets your standard, which is what evaluations are for.

See the Comparisons Guide for result views, metrics, and model selection detail.

Evaluations

A comparison tells you which model was fastest and cheapest. It does not tell you whether any of them gave the right answer, or whether the response meets the specific criteria your use case requires.

Evaluations run the same prompt against multiple models but score the results against a standard you define: either a gold standard reference answer, or a rubric of weighted criteria. The scoring is objective and repeatable.

See the Evaluations Guide for scoring modes, rubric setup, and result views.

Scores

An evaluation runs a model and scores the response. A scores run skips straight to the scoring: you give it text you already have, and it grades that text against your rubric or compliance store. Use it when the words are already written and you want to know whether they clear your standard. Hand it several versions at once and it ranks them against the same standard, so picking the strongest option stops being a matter of opinion.

See the Scores Guide for single and multi-artifact scoring, and reading the per-criterion verdict.

Benchmarks

A comparison or evaluation gives you a snapshot: which model wins today. Whether that answer is still correct next month is a different question.

A benchmark is a saved run configuration that executes on a schedule. Every run produces the same structured result. The dashboard tracks changes across runs and flags anomalies automatically.

The charts below are from a real benchmark: the same prompt run across 7 models over 5 consecutive runs.

GPT-4o holds 100 across every run. GPT-4o Mini, Grok 4.3, and DeepSeek V4 Flash all show variation, dropping to 75 on some runs and recovering on others. Without a benchmark running, those drops are invisible until a user notices.

Efficiency (quality divided by cost) tells a different story. GPT-4o Mini and DeepSeek V4 Flash are 20-30x more efficient than GPT-4o when they hit 100, but their efficiency collapses when quality drops. GPT-4o’s efficiency is low but flat. Which model fits depends on whether consistency or cost-efficiency matters more for the task. Benchmarks give you the run history to make that decision with numbers, not instinct.

See the Benchmark Guide for benchmark types, schedules, views, and anomaly flags.

MCP

The MCP server is the fastest way to add LLM Prover to an AI agent workflow. Any MCP-capable client discovers the 30+ available tools automatically. An agent can call run_comparison, poll for results, read the response text, and make a model decision – all without leaving its workflow and without any integration code.

MCP access is on Pro and above. See the MCP Quickstart to connect your client in under five minutes.

REST API

Every operation available in the dashboard is also available as an API call. Fire comparisons, run evaluations, trigger benchmark runs, and retrieve results from any script, pipeline, or CI workflow. The same async pattern applies: POST to start, poll for the job, retrieve the full result.

Developer keys are available on all plans from your dashboard settings. See the API Quickstart to make your first call.

Config

Config items attach to any run type. None are required for a first run. All make results more specific, more repeatable, and more meaningful.

System Prompts

A system prompt is the standing instruction you give a model before the user prompt. It sets tone, persona, constraints, and context. System Prompts in LLM Prover are saved, named templates you can select from a dropdown on any run. Use this when you want every run to reflect your production setup rather than testing the model in isolation, or to A/B test two system prompts against each other on the same prompt.

Rubrics

A rubric is a named, reusable scoring template: a list of criteria, each with a name, description, and weight. Create it once, attach it to any evaluation or benchmark. The judge scores each response against each criterion independently and produces a per-criterion score with reasoning. Use this when a raw quality score is not enough and you need to know whether the response met a specific standard.

See the Rubric Guide before writing your first rubric.

Stores

Stores are uploaded document collections. A context store injects the most relevant chunks from your knowledge base into the prompt automatically. Use it when your models need access to internal documentation, product data, or any content that is not in the model’s training data. A compliance store lets the judge score responses against specific clauses in your uploaded regulation or policy document (Pro+ and above).

See the RAG Store Guide and Compliance Store Guide for setup detail.

Endpoints

Register any OpenAI-compatible endpoint and benchmark it against any model in the registry using the same scoring pipeline. Covers fine-tuned models, self-hosted deployments, Azure OpenAI custom URLs, and GPU cloud deployments. Use this when you need reproducible evidence that your custom model outperforms a baseline.

See the BYOE Guide for endpoint registration and benchmark setup.

Presets

A preset is a named, reusable run configuration. You supply the content each time; everything else comes from the preset. Build one once and invoke it by name from the dashboard, the API, or an agent, so a run you repeat (your standard support eval, your brand-voice check) is one reference instead of a form to rebuild. Every account starts with a curated preset so you can see one at work on your first run.

See the Presets Guide for creating, invoking, and scheduling presets.

Account

Your tier determines your run limits, monthly usage allowance, and which features are available. See the pricing page for a full tier comparison. The Models page lists every available model with its tier requirement and cost per 1k tokens.

Your monthly usage allowance resets on your billing anniversary. Top-ups are available from the account page if you need more mid-month.

What’s next