How to evaluate an LLM

You have an LLM answering real requests in your product, and you need to know whether its answers are actually good. Not good in a demo, good on the requests your users actually send. Testing a few by hand and reading them over tells you almost nothing, because your judgement drifts, you only check a handful, and the next model update can change everything without you noticing.

The way out is to score the answers against a fixed definition of what good means, so the judgement is consistent and repeatable instead of a gut call. This post walks through doing that on one real task, so you can see what you set up, what you get back, and what you do with it.

First, decide what a good answer actually is

Before you can score anything, you have to say what you would accept as a good answer, because without that there is nothing to measure against. Take a real request your product has to handle. A customer wants to cancel and asks for a refund, for example.

You run three models on it to see your options. Here is what each one sent back.

All three are plausible, and eyeballing them you could talk yourself into any of them. That is the problem with judging by hand. So instead you write down the reply you actually want, once, and let every model be scored on how close it gets. That written-down ideal is called a gold standard.

Sometimes you cannot write one ideal answer, because what you really care about is a set of qualities. In that case you list those qualities instead, and put a weight on each one to say how much it matters. That list is called a rubric.

Now score the same three replies against your gold standard, then against your rubric, and you get a different winner each way.

Against your gold-standard answer, DeepSeek V4 Flash wins. Against your rubric, GPT-4.1 Mini wins. Same replies, different definition of good.

Against the gold standard, DeepSeek V4 Flash wins at 76.3, because it committed to the action and apologised, which is what your ideal answer did. Against the rubric, GPT-4.1 Mini takes it at 81.2 and DeepSeek drops to 75.0, because the rubric rewards brevity and DeepSeek’s longer reply gives some of that back. The replies never changed. What changed was how you defined good, and that alone moved the winner.

So the first real decision is yours, not the model’s. If you can write down the one answer you want, use a gold standard. If what you care about is a handful of qualities, use a rubric. Pick whichever matches how you would actually judge the reply if it landed in front of you.

Read more than the score

Now you have a winner, but the score alone is not enough to make the call, because a reply you ship also costs money and takes time. So alongside the quality score you get the cost of that answer and how long it took, and from those you get the number that usually decides it, which is how much quality you are getting per dollar.

That last one changes decisions. The top-scoring model is often not the one you want running on every request, because a model a point or two behind at a fraction of the cost is the smarter choice for everyday volume. You can only see that trade-off when the quality and the cost are sitting next to each other, which is the whole point of looking past the single number.

The quality score also splits into the qualities you asked for, so when a reply scores low you can see which part let it down rather than guessing. A reply at 40 might be accurate but incomplete, and that breakdown tells you so, which is exactly what you need before you decide whether to fix the prompt, switch the model, or leave it.

When a reply scores badly, find out why before you fix it

Sooner or later a reply scores badly, and the score alone will not tell you what to change. You could tune the prompt, swap the model, adjust the system prompt, or fix the documents the model was answering from, and all four shaped the answer. A bare score blames them all at once, so on its own it leaves you guessing which one to touch.

You do not have to make that guess yourself. Alongside the score, LLM Prover returns a diagnostic finding that reads the whole chain behind an answer and names the part that caused the low score. An example makes it concrete.

Say your product has a support assistant that answers from your own help documents. When a customer asks about plan limits, the model looks up the limit in your docs and replies. Your docs include this line.

The real limit is 100. Your team raised it months ago, but that line in the help doc was never updated. You do not know that yet. You just score the assistant against the correct answer, which is 100 requests, and all three models come back saying 40 and score badly.

On the score alone the models look broken. The finding tells you they are not. It points at the context, meaning the help document the models were reading from, and quotes the exact line at fault, the one above that still says 40. Every model read that line correctly and answered from it faithfully. The models did their job. The document was out of date, and the fix is to correct that one line. Swapping models would never have found it, because the problem was never in the model, and the finding is what pointed you at the part that actually failed.

Run it again, because the answer does not stay true

Scoring your model once tells you it is good enough today, and that is where it stops being true. The companies behind these models update them without telling you, so a reply that scored well last month can quietly start scoring worse, and you would never know from a one-off check.

The fix is to run the same scoring on a schedule, so the moment quality slips you see it instead of hearing about it from a user. On a scheduled check it looks like this.

The score holds for weeks, drops below the bar at week 4, and stays down until the cause is fixed at week 7.

The drop at week 4 is the signal. It stays below your bar through weeks 5 and 6, because a problem does not fix itself, and it only climbs back once you have found and corrected the cause. Without a scheduled check that whole stretch is invisible, and your users live through it before you hear a word. Keep a few models in the same check and the shape of a drop tells you where to look.

• They all drop together – something in your own setup changed, like a prompt or a document everyone reads from.
• Only one drops – that provider changed its model underneath you, and the others are your control group.

Either way you find out early, while it is still your decision to make.

A full evaluation is these four pieces working together.

• You decide what good means, with a gold standard or a rubric.
• You read past the single score to what the answer costs and how long it takes.
• The diagnostic names the layer to fix when a reply falls short.
• A scheduled check keeps the answer honest as the models shift under you.

Set that up once on a task that matters, and you stop guessing whether your AI is good and start knowing.