FAQ

Comparisons fire a prompt at multiple models simultaneously and return the responses, cost, and latency side by side. They tell you what each model produced, how fast, and at what cost. Evaluations go further: they score each response against a standard you define, either a gold standard reference answer or a rubric of weighted criteria. Use Compare to see the field. Use Evaluate to prove a model meets your quality bar.

A benchmark is a saved run configuration that executes on a schedule. Every run produces the same structured result, and the dashboard tracks changes across runs automatically. You need one when a one-off comparison is not enough: when you want to know whether your model choice is still correct next month, or when you need to catch quality drops before users do.

It depends on your tier and what you attach to the run. Without a rubric or gold standard, the score is a heuristic based on measurable signals like response length, format compliance, and output consistency. With a gold standard, the judge scores semantic similarity to your reference answer. With a rubric, the judge scores each criterion independently and returns per-criterion reasoning. See the Scoring Guide for full detail.

The full registry is at /models. It includes models from OpenAI, Anthropic, xAI, Together AI, Groq, and others, with tier requirements and cost per 1k tokens shown for each. The registry is updated as new models are released.

Yes. BYOE (Bring Your Own Endpoint) lets you register any OpenAI-compatible endpoint and benchmark it head-to-head against any model in the registry. Covers fine-tuned models, self-hosted deployments, Azure OpenAI custom URLs, and any GPU cloud deployment. Available on Pro and above.

The API is a REST interface. You call it from code using an HTTP client and a developer key. The MCP server exposes the same operations as tools that any MCP-compatible agent discovers automatically. Use the API when you are building a pipeline or script. Use MCP when you are working inside an agent IDE like Claude Desktop, Cursor, Amazon Q, or VS Code Copilot and want the agent to call LLM Prover tools without writing integration code.

Yes. The MCP server exposes a full set of tools (currently 60-plus) covering comparisons, evaluations, scoring, benchmarks, presets, RAG stores, rubrics, system prompts, agent state, and model discovery. Any MCP-compatible agent discovers them automatically when connected. No integration code is required. Connect your client from the MCP page in your dashboard. MCP access is on Pro and above.

Agent evaluation checks a whole agent workflow, where model evaluation checks one step. Model evaluation is the unit test: one call to one model. Agent evaluation is the integration test: every step of the workflow and how each feeds the next. An agent analysis reads the trace of a workflow (the step-by-step record of what your agent did) and shows where it wasted money, time or effort and which step caused it. Start one from your agent over MCP or from the dashboard. It is on Pro and above. See agent evaluation.

Kiro and Claude Code. LLM Prover detects which of the two a trace came from, so you do not choose a format. To see what Claude Code itself costs, read Claude Code pricing. Read how to read an agent analysis for what each finding means.

No separate charge. An agent analysis counts toward your plan’s usage like any other use of the product. It is included on Pro, Pro+ and Enterprise, and not on Starter.

Model calls take between 5 and 30 seconds depending on the models selected and the length of the prompt. The async pattern keeps your agent or pipeline unblocked. You fire the job, receive an ID immediately, and poll until the result is ready. This works inside agent loops, CI pipelines, and scripts without stalling the caller.

Two limits apply, both per tier: a limit on paid model calls (the ones that cost money) and a higher limit on all requests including cheap reads. Starter: 10 model calls per minute and 200 per day, 60 requests per minute and 2,000 per day. Pro: 30 per minute and 500 per day, 120 per minute and 5,000 per day. Pro+: 40 per minute and 750 per day, 160 per minute and 8,000 per day. Enterprise: 60 per minute and 2,000 per day, 300 per minute and 20,000 per day. Going over the per-minute limit returns a 429, which you can retry after a short backoff.

For the REST API, use a developer key from the Developer section of your dashboard – pass it as the Authorization header value. For MCP, clients that support OAuth (Claude Desktop, Cursor, VS Code Copilot, Amazon Q) authenticate automatically via the OAuth flow when you connect. Developer keys also work as a fallback for MCP clients that do not support OAuth. Either way, the same tier limits apply.

A RAG store is a collection of documents you upload. When you attach one to a run, the most relevant chunks from your documents are injected into the prompt automatically. You need one when your models need access to internal documentation, product data, or any content that is not in the model’s training data. Available on Pro and above.

No. Compliance scoring is a measurement tool. You upload your own regulations, policies, or standards as a compliance store. The judge scores each model response against the most relevant clause retrieved from your document. The score is traceable to a specific clause. It is not a legal certification or regulatory approval. It is a measurement of how well outputs align with the documents you provide. Available on Pro+ and above.

Yes. We publish 11 free, own-key tools at /tools: Model Compare, Context A/B, Variance, Cost Table, JSON Reliability, Prompt A/B, System-Prompt A/B, Right-Size It, Gold-Standard Check, Faithfulness Check, and Compliance Read. Each does one measurement job with your own API keys. No account, no signup. Your keys stay on your machine and are never sent to us.

The free tools are thin, own-key slices that each do one job and give no quality verdict. Model Compare, for example, runs two models head to head and shows the answers, cost, and speed, but it does not score which is better. The paid product is where the judgment lives: compare up to 20 models at once, score responses against a rubric or a gold standard, see which part of your pipeline caused a weak answer, save benchmarks, track drift over time, and run it all from your agent over MCP.

No. The free tools call the provider directly with your own key. Nothing runs through our servers: no key custody, no prompt storage. The paid product is different: there we run calls on your behalf and store your history, as covered under Data and privacy.

The free Model Compare tool is a two-model head to head by design. Comparing many models at once, and scoring which clears your bar, is part of the paid product. You can compare ChatGPT, Claude, Grok, or Gemini against any one other model your key reaches, free, as often as you like.

Yes, as a first look. Gold-Standard Check, Faithfulness Check, and Compliance Read are free own-key tools that run a single-pass check and report a result to help your review. They are not a certification, and a single pass can be inconsistent. Verification you can rely on, with rubric-based scoring, consistency checks, an audit trail, and checking against your whole policy corpus, is the paid product.

Usage is the cost of the model calls LLM Prover makes on your behalf. Input and output tokens are billed at provider rates. It does not include LLM Prover’s platform fee, which is your monthly subscription.

Yes, a one-week paid trial that converts to the full plan after seven days. Trial pricing by tier: Starter $5 with a $4 usage cap for the week, Pro $10 with an $8 cap, Pro+ $20 with a $16 cap. The trial is offered on specific landing pages. The standard pricing page checks out at the full monthly rate.

Runs stop. You can purchase a top-up from your account page immediately with no downtime and no data loss. Top-ups are consumed as you run and carry over month to month. They are not refundable.

Yes. Top-ups are available from your account page at any time. They do not expire and are consumed as you run. If you have unused top-up balance at the end of the month, it carries over.

Pro adds rubric evaluation, daily benchmarks, alert thresholds, one RAG store, and one BYOE endpoint. Pro+ adds compliance rubric scoring, 4x daily benchmark frequency, up to five RAG stores, five BYOE endpoints, and bulk export.

Enterprise plans are configured to your team size and use case. Get in touch at /contact.

Yes. Cancel from your account page or billing portal at any time. Stripe will apply a credit to your account for the unused portion of your subscription. You can use that credit against a future charge, or continue using the service until the end of the paid term with no further charges after that.

Features above your new tier are locked immediately. Benchmarks that use Pro+ features – compliance stores, 4x daily schedules, extra RAG stores – are frozen automatically. The data is preserved and the benchmarks resume if you upgrade. The dashboard shows a downgrade preview before you confirm so you know exactly what changes.

Yes. Prompts and outputs are stored to power your result history and benchmark drift analysis. Retention by tier: Starter 90 days, Pro and Pro+ 365 days, Enterprise unlimited. You can delete individual records from the History view at any time.

The trace is deleted when the analysis finishes. The report is private to you. The report is scrubbed for common key and token formats, on a best-effort basis: it catches familiar formats, not every possible secret.

No. Your prompts and outputs are not used to train any model.

Your data is retained while your account is active. Upon cancellation, data is retained subject to our standard retention policy and then deleted.

The service is operated from the United States. Data is stored in AWS us-east-1 (Northern Virginia).

No. Open any tool at /tools, click through to the notebook, add your own provider key, and run. An account is only for the paid product: saved benchmarks, scoring, history, and MCP.

Sign up, pick a plan, and run your first comparison. The Getting Started guide covers the full product surface, run types, config, and how the pieces fit together.

Go to Developer > Keys in your dashboard and create a key. It is shown once, so copy it immediately. Use it in the Authorization header for REST API calls. The API Quickstart walks through the full setup.

Go to the MCP page in your dashboard and open the Clients tab. It has copy-paste config for Amazon Q, Claude Desktop, Cursor, and VS Code Copilot. These clients authenticate via OAuth – the config includes your MCP endpoint URL and the client handles authentication automatically. For clients that do not support OAuth, a developer key works as a fallback. Once connected, ask your agent to list the available tools and the server will describe them.

The REST API is available on all tiers including Starter. MCP server access requires Pro or above. Developer keys are available on all tiers from the Developer section of your dashboard.

From the dashboard, go to Benchmarks and create a new benchmark. Add your prompt, select your models, set a schedule, and save. The Benchmark Guide covers the full setup.

Go to Endpoints in the dashboard and register your endpoint. You need a base URL, an auth header, and a model ID. Any OpenAI-compatible endpoint works. See the BYOE Guide for step-by-step instructions.