Comparisons Guide

A comparison fires a single prompt at multiple LLM providers simultaneously and returns a structured result: cost per call, latency, token count, and the full output from each model, side by side. It is the fastest way to answer the question every team eventually asks: which model should we actually be using for this? Not in general. For this prompt, this use case, this quality bar.

If you want to try the idea before setting anything up, the free Model Compare tool runs a two-model head-to-head on your own prompt with your own keys, no signup.

When to run a comparison

Run a comparison when you need a snapshot. You have a prompt, a real one from your product, not a synthetic test, and you want to know which models can handle it, what each one costs, and how fast they are. A comparison does not score quality. It tells you what each model produced and what it cost to produce it. If you need to know whether any of those outputs actually met your standard, that is what evaluations are for.

• Picking a model before building a feature
• Proving a cheaper model produces comparable output
• Checking whether a new model release is worth switching to
• Getting a cost-per-call baseline before committing to a provider

Running a comparison

Enter your prompt, select the models you want to test, and hit Run. Results appear in the table as they complete. You do not need to wait for all models to finish. Each row is a model. Each column is a metric.

• Prompt field. Use your real production prompt. Synthetic prompts flatten the differences between models and produce results that do not transfer to your actual use case.
• Model picker. Pro and above select individual models up to the tier cap. Starter runs against a default set. See the models page for tier detail.
• System prompt. Optional. Select a saved system prompt from the dropdown to reflect your production setup. See System Prompts for setup.
• Context. Optional. Starter accounts can paste context directly into the prompt field. Pro and above can attach a context store. The most relevant chunks are injected automatically at run time. See the RAG Store Guide for setup.
• Inference parameters. Optional. Temperature, max tokens, seed, top_p, and others. Defaults are applied when not set. See Inference parameters below.
• The pending row. Once you hit Run, each model appears as a pending row with a running badge. Results populate as they complete. The pending row survives navigation. You can leave the page and come back.

The results table

Every completed comparison appears as a row in the table. The winner badges (cheapest and fastest) are highlighted automatically. Click any row to open the full result.

• Cost. Total cost of the call in USD. What you actually paid for that one call.
• Latency. Wall time from request sent to full response received.
• Tokens. Input plus output combined.
• Winner badges. Cheapest and fastest, based on that run only.
• Performance column. When multiple runs exist for the same prompt, spark bars show the trend across runs.

Failed rows show the error message and error type. Transient errors (rate limit, timeout) get a Retry button. Permanent errors get Delete only.

Inside the result drawer

Click any row to open the result drawer. Three tabs: Snapshot, Head to Head, and Table. Snapshot is the default. It shows the full output from each model alongside its cost and latency. Head to Head and Table are for when you have scoring attached (see the Evaluations Guide).

Snapshot tab:

• Full output text per model
• Cost and latency beneath each output
• Cost projection: what this prompt costs at scale (1k, 10k, 100k calls per month)
• “New from this” button: carries the full config (prompt, models, system prompt, context, parameters) into a new comparison or evaluation

Head to Head tab. Quality scores and per-criterion reasoning. Only populated when a rubric or gold standard is attached.

Table tab:

• Sortable breakdown of all models in the run
• Diagnostics column: severity badge with observation and suggestion text
• Threshold indicators if thresholds are configured

What the metrics mean

Cost is the total cost of the API call in USD, not per 1k tokens, not amortised. What you actually paid for that one call. Latency is wall time: from the moment the request was sent to the moment the full response was received. Token count is input plus output combined.

One thing the token count does not tell you: cost. Different providers price input and output tokens differently, and some charge for reasoning tokens that do not appear in the output. Two models with the same token count can have very different costs. The cost column is the number that matters.

Efficiency (quality per dollar) only appears when scoring is attached. It answers a different question: not which model was cheapest, but which model gave you the most quality for the money. A model that costs twice as much but scores 40% higher on your rubric has better efficiency than a cheaper model with a lower score. Efficiency is the metric that tells you whether paying more is actually worth it.

Model selection

The model picker shows every provider and model available on your tier. Starter accounts run against a default set with no picker. Pro and above select individual models up to the tier cap. If you have registered a BYOE endpoint, it appears in the picker alongside managed models.

See the models page for the full list of available models by tier, and the BYOE Guide for endpoint registration.

Adding context

If your prompt relies on internal knowledge (product documentation, a knowledge base, a policy document) you have two options depending on your tier.

Starter: Paste the context directly into the prompt field before the instruction. Every model receives the same pasted text. This is what the run above used. The benchmark data was pasted inline, and the prompt instruction followed it.

Pro and above: Attach a context store. LLM Prover retrieves the most relevant chunks from your uploaded documents and injects them into the prompt automatically at run time. Every model in the comparison gets the same context, so results reflect how each model uses your content, not just how it performs in isolation.

See the RAG Store Guide for store setup and chunking detail.

Inference parameters

Inference parameters (temperature, max tokens, seed, top_p, and others) are optional and default to sensible values. Most comparisons do not need them.

The one worth knowing: seed. Setting a seed makes results reproducible. The same prompt with the same seed will produce the same output on the same model. Useful when you want to isolate a variable and be sure output variation is not just randomness.

Pipeline diagnostics

Every comparison run includes a diagnostic pass. The diagnostic judge reads your full setup (prompt, context, system prompt, parameters) and flags anything that might be affecting result quality. It does not change scores. It explains them. A finding in the diagnostics column does not mean the run failed. It means something in your pipeline is worth looking at.

See the Scoring Guide for a full explanation of diagnostic severity levels and how to act on findings.

Exporting results

Export is available from the result drawer. CSV and JSON formats are both supported. The export includes all models in the run, all metrics (cost, latency, tokens), and the full output text from each model. Use this when you need to share results with someone who does not have an LLM Prover account, or when you want to process the data outside the dashboard.

From comparison to evaluation

A comparison tells you what each model produced and what it cost. It does not tell you whether any of those outputs were actually good. Two models can produce outputs that look similar in the table (similar cost, similar latency) but one answers the question correctly and one does not. That distinction only becomes visible when you score against a standard.

Both outputs follow the prompt. Both are factually consistent with the source. The cost difference is $0.00076 per call. Whether that difference matters, and which output actually meets your standard, is a question a comparison cannot answer. Hit “New from this” in the result drawer to carry your prompt, model selection, and context into an evaluation.

See the Evaluations Guide.

From comparison to benchmark

A comparison is a snapshot. The model that wins today may not win next month. Providers update weights silently, costs shift, and output quality changes without notice. If this prompt matters to your product, turn it into a benchmark. A benchmark runs the same comparison on a schedule and tracks every result. The moment something changes, you see it.

Run 3 shows a cost spike on GPT-4o Mini and a jump on Grok 4.3 on run 5. Neither would be visible from a single comparison. A benchmark running on a daily schedule catches both within 24 hours.

See the Benchmark Guide.

What’s next