Benchmark your own model against the field.
Register any OpenAI-compatible endpoint (your fine-tune, self-hosted model, or whole pipeline), and score it head-to-head with any other model.
Get started on Pro#ml-eval
Chat
Y
You
@llmprover register my endpoint: our Grok 4.7 deployment at api.x.ai/v1.
●●●
registering the endpoint and running a connectivity check...
●
llmprover
bot
Registered and reachable. It is now in your model picker as a BYOE endpoint, ready to run alongside any hosted model.
Y
You
@llmprover evaluate how it compares to Claude Sonnet 4.5 and GPT-4o Mini on our onboarding docs.
●●●
running a comparison, your endpoint + 2 hosted models, grounded in your onboarding store...
●
llmprover
bot
Done. All three answered correctly from your onboarding docs. Quality: all correct. Cost per call -- your endpoint (Grok 4.7): $0.0069 | Claude Sonnet 4.5: $0.0093 | GPT-4o Mini: $0.0003.
Your model, in the same run as the field.
ProYour endpoint joins the model picker and gets scored by the same rubric in the same run as every hosted model. In a benchmark we ran, a registered Grok 4.7 endpoint scored highest on quality against two frontier models, and cost the most per call. You see quality and cost side by side and decide on numbers. The full run is here.
Benchmark against your field, not the frontier.
ProYou do not have to beat GPT to have a model worth running. Most custom models are built on a cheap base to beat cheap models, so you pick the comparison set and measure against the ones you would otherwise pay for. The question is never “did you beat the most expensive model,” it is “did you beat the one you would actually use.”
Prove your own deployment complies.
Pro+Run a compliance evaluation against your registered endpoint and get a per-clause pass or fail, each verdict carrying the exact regulation text the judge applied. That is your live deployment scored against your regulations, clause by clause, not a hosted model standing in for it. See compliance scoring that cites the clause.
Catch regressions in your own model over time.
ProA model under active development changes with every redeploy. A scheduled benchmark against your endpoint tracks its quality run over run, so a regression like the drop at v4 above surfaces before your users find it. Test your deployment the way you test software: on every change, not once at launch.
Benchmark your whole pipeline.
ProA RAG pipeline, an agentic workflow, or any multi-step backend can be registered as an endpoint and measured as a black box: prompt in, output scored by the same rubric as any model. That catches what unit-testing a model cannot, since a model that scores 95 alone can score far lower as the generation step of a weak pipeline. If it speaks the OpenAI API format, it can be registered.
Put your own model in the ring.
Register an OpenAI-compatible endpoint and benchmark it against the field. Pro and above.
Reproducible evidence, not a one-off.
The value is that the run repeats: the same benchmark reruns on a schedule or on demand, with run history, timestamps, and exportable results a reviewer can audit. Register once, and the evidence stays current through every new fine-tune and provider update.
Starter $49/mo | Pro $129/mo | Pro+ $199/mo | Enterprise $499/mo | |
|---|---|---|---|---|
| BYOE endpoints | -- | 1 | 5 | 10 |
| Benchmark your endpoint vs the registry | -- | ✓ | ✓ | ✓ |
| Scheduled benchmarks (catch regressions) | -- | daily | 4x daily | hourly |
| Compliance scoring against your endpoint | -- | -- | ✓ | ✓ |
Put your own model in the ring.
Register an OpenAI-compatible endpoint and benchmark it against the field. Pro and above.
Benchmark costs for your endpoint are estimated from the per-token rates you supply, and the real inference charges are billed to your own provider account. See the terms for detail.