Getting Started

This guide covers the two concepts that underpin everything in LLM Prover: run types and config. Run types are the three ways to execute a prompt: Comparisons, Evaluations, and Benchmarks. Config items are the inputs you attach to runs to make results more specific, repeatable, and meaningful. None are required for a first run.

The dashboard menu reflects this split. The top section (Run) is where you execute. The bottom section (Config) is where you set up the inputs that feed into runs.

Why LLM Prover

LLM model behaviour is not static. Providers update weights silently, costs shift, and outputs that passed review last quarter may not behave the same way today. Without a measurement layer, those changes are invisible until something breaks.

LLM Prover is the measurement layer that closes that gap. Here is what teams use it for:

• Picking the right model for your use case. Provider benchmarks are not run on your prompts, your data, or your quality standard. A comparison run on your actual prompt tells you which model wins for your specific task, and what each one costs per call.
• Proving a cheaper model is good enough. For many everyday tasks (summarisation, classification, simple RAG) open source models match frontier quality at a fraction of the cost. Running the comparison properly is what takes time. LLM Prover makes it take 30 seconds.
• Scoring outputs against your own criteria. Comparisons tell you which model was fastest and cheapest. Evaluations tell you whether any of them actually met your standard: accuracy, tone, format compliance, policy adherence. Different question, more useful answer.
• Catching model drift before users do. Model providers update weights silently. The model you validated last quarter may not behave the same way today. Scheduled benchmarks run your prompt on a cadence and alert you the moment quality drops, costs spike, or a model version changes.
• Proving your fine-tuned model is better. If you have trained a domain-specific model, BYOE lets you register your endpoint and benchmark it head-to-head against any model in the registry. The result is reproducible, exportable evidence. Not anecdote.
• Measuring compliance posture over time. Upload your regulations, policies, or standards as a compliance store (Pro+ and above). The judge scores every response against the most relevant clause in your document. Scheduled benchmarks catch when a model update breaks your compliance posture before it reaches users.
• Diagnosing inconsistent outputs. When outputs are unpredictable and swapping models hasn’t helped, the pipeline diagnostics judge reads your full setup (prompt, context, system prompt, rubric criteria) and tells you exactly which layer is the problem. Most of the time it is not the model.
• Giving your team a benchmarking budget. Every account is an org. Invite members, control per-member spend, and give engineers a proper tool with a budget you set. The owner sees all usage in one place.

Run

Three ways to execute a prompt. Comparisons give you a snapshot. Evaluations score against your standard. Benchmarks track both over time. Each one builds on the last. Start with a comparison, add a rubric when you need scoring, turn it into a benchmark when you need to track drift.

Comparisons

Comparisons fire a single prompt at multiple models simultaneously and return a structured side-by-side result in seconds. Each model gets a cost, latency, and token count. The fastest and cheapest model are highlighted automatically.

The animation above shows 4 models. The charts below are from a broader run across 7 models on the same prompt type.

GPT-4o Mini and DeepSeek V4 Flash are the cheapest by a wide margin. Qwen 3.6 27B is the fastest at 980ms. GPT-4o and Claude Opus 5 are the most expensive. Whether that cost is justified depends on whether their output quality meets your standard, which is what evaluations are for.

See the Comparisons Guide for result views, metrics, and model selection detail.

Evaluations

A comparison tells you which model was fastest and cheapest. It does not tell you whether any of them gave the right answer, or whether the response meets the specific criteria your use case requires.

Evaluations run the same prompt against multiple models but score the results against a standard you define: either a gold standard reference answer, or a rubric of weighted criteria. The scoring is objective and repeatable.

See the Evaluations Guide for scoring modes, rubric setup, and result views.

Benchmarks

A comparison or evaluation gives you a snapshot: which model wins today. Whether that answer is still correct next month is a different question.

A benchmark is a saved run configuration that executes on a schedule. Every run produces the same structured result. The dashboard tracks changes across runs and flags anomalies automatically.

The charts below are from a real suite: the same prompt run across 7 models over 5 consecutive runs.

GPT-4o holds 100 across every run. GPT-4o Mini, Grok 4.3, and DeepSeek V4 Flash all show variation, dropping to 75 on some runs and recovering on others. Without a benchmark running, those drops are invisible until a user notices.

Efficiency (quality divided by cost) tells a different story. GPT-4o Mini and DeepSeek V4 Flash are 20-30x more efficient than GPT-4o when they hit 100, but their efficiency collapses when quality drops. GPT-4o’s efficiency is low but flat. Which model fits depends on whether consistency or cost-efficiency matters more for the task. Benchmarks give you the run history to make that decision with numbers, not instinct.

See the Benchmark Guide for benchmark types, schedules, views, and anomaly flags.

Config

Config items attach to any run type. None are required for a first run. All make results more specific, more repeatable, and more meaningful.

System Prompts

A system prompt is the standing instruction you give a model before the user prompt. It sets tone, persona, constraints, and context. System Prompts in LLM Prover are saved, named templates you can select from a dropdown on any run. Use this when you want every run to reflect your production setup rather than testing the model in isolation, or to A/B test two system prompts against each other on the same prompt.

Rubrics

A rubric is a named, reusable scoring template: a list of criteria, each with a name, description, and weight. Create it once, attach it to any evaluation or benchmark. The judge scores each response against each criterion independently and produces a per-criterion score with reasoning. Use this when a raw quality score is not enough and you need to know whether the response met a specific standard.

See the Rubric Writing Guide before writing your first rubric.

Stores

Stores are uploaded document collections. A context store injects the most relevant chunks from your knowledge base into the prompt automatically. Use it when your models need access to internal documentation, product data, or any content that is not in the model’s training data. A compliance store lets the judge score responses against specific clauses in your uploaded regulation or policy document (Pro+ and above).

See the RAG Store Guide and Compliance Store Guide for setup detail.

Endpoints

Register any OpenAI-compatible endpoint and benchmark it against any model in the registry using the same scoring pipeline. Covers fine-tuned models, self-hosted deployments, Azure OpenAI custom URLs, and GPU cloud deployments. Use this when you need reproducible evidence that your custom model outperforms a baseline.

See the BYOE Guide for endpoint registration and benchmark setup.

Account

Your tier determines your run limits, monthly usage allowance, and which features are available. See the pricing page for a full tier comparison. The Models page lists every available model with its tier requirement and cost per 1k tokens.

Your monthly usage allowance resets on your billing anniversary. Top-ups are available from the account page if you need more mid-month.

What’s next