The LLM evaluation metrics that actually matter (and the ones that don't)

Search for “LLM evaluation metrics” and you get a wall of acronyms. BLEU, ROUGE, perplexity, F1, BERTScore. You read the list, you nod, and you still have no idea whether the AI in your product is good enough to ship. That gap is the problem. Most of those scores were built to compare research models on academic datasets, to answer a question that is not yours.

This is the short list that answers that, in plain English, with what each number means and when it is worth your attention.

The one idea the whole thing rests on

A metric is a number, and a number means nothing without a standard behind it. “It scored 87” tells you nothing until you know: out of what, judged how, and against whose idea of a correct answer.

So before any metric is useful, you have to write down what a good answer looks like for your task. A correct reply to grade against, or a short list of qualities you care about. That written-down standard is what turns a score from noise into something you can act on. Every metric below assumes you have one. If you do not, fix that first, because no metric rescues a missing definition of good.

The metrics that actually matter

Group them by the question each one answers. There are only three you need to start.

“Is it correct, or good enough?” – a quality score

This is the one that matters most, and it is the one the acronyms rarely give you. You take the AI’s answer, compare it against your standard, and get a score for how well it did. If you have a single correct answer, you grade against that. If “good” is fuzzier (i.e. a helpful tone, covers the right points, etc), you write a short list of qualities and grade against those instead.

The score itself is only half of it. The grading is done by a second model (a “judge”) that reads each answer against your standard, and the useful part is that it returns the reasoning for the score alongside the number. So you can see whether it graded the way you would have. A number you cannot explain is a number you cannot trust, which is why a raw score from a script that only spits out a figure is worth less than a judged score that shows its working. (More on how that works in LLM-as-a-judge.)

“What does it cost me?” – cost per answer, not per token

Most people reach for the price per token, and it is the wrong number. Price per token tells you what the raw material costs, not what a finished answer costs. A model with a low token price can still be the expensive choice, because it rambles, retries, or thinks out loud for a page before answering, and you pay for every one of those words.

The number that matters is the cost of one complete answer to one real request. Measure that, and the cheap-looking model is sometimes more expensive, and the premium model is sometimes a bargain. You only find out by measuring the whole answer, not the sticker rate.

“Is it worth the price?” – quality per dollar

This is the metric almost nobody tracks, and it is the one that decides your bill. Quality per dollar puts the first two numbers together: how good the answer is, next to what it cost to produce. It is the number that tells you whether you are overpaying.

Here is why it matters, from a real run. We scored the same support reply on two models against a correct answer. One scored 94.4. The other scored 94.2. A tie, for all practical purposes. But one of them cost about nine times more per reply than the other. On a task you run thousands of times a day, that is the whole difference in your bill, for no difference in quality. Looking at quality alone, you would call it a toss-up. Looking at quality per dollar, the choice is obvious.

What about BLEU, ROUGE, perplexity, and the rest?

They are not useless, they are just answering a different question than yours. In one plain line each:

• BLEU and ROUGE check how much your text overlaps with a reference text, word for word. Built for translation and summarisation research.
• Perplexity measures how “surprised” a model is by a piece of text. A training-time diagnostic, not a quality score.
• F1 and BERTScore are overlap and similarity measures from the same academic world.

The common thread: they compare against a fixed reference answer, mechanically, and they were made to rank research models on shared datasets. They do not know what good means for your task, your customer, your policy. For deciding whether your product’s AI is good enough to ship, they mostly miss the point. If a vendor leads with them, ask the question they do not answer: good enough for what?

The metric everyone frames wrong: consistency

Here is one most teams never check. Ask the same model the same question twice and you can get two different answers, and sometimes two different quality scores. A single run can flatter a model that got lucky, or punish one that had an off moment.

We saw this on a hard task: three identical runs produced three different winners. Same prompts, same models, run three times, and the best performer changed each time. If we had run it once and trusted the number, we would have picked a “winner” that was really just noise.

Three identical runs of one hard task. The lines cross repeatedly: a different model leads each run, and the cheapest one swings from near the bottom to the top.

So consistency is a metric in its own right. Run the task more than once and see how much the score moves. A model that scores well and scores the same every time is one you can rely on. A model that swings is a risk, no matter how high its best run looked.

Reading a score like a diagnostician

One last shift that changes how useful your metrics are. A score tells you that something is wrong. It does not tell you what. A reply that scored low could be a weak model, or a vague prompt, or the wrong background information fed in, or a standard that was fuzzy to begin with. The number alone cannot tell those apart.

The useful read is the one that names the layer. Was it the model, the prompt, the context it was given, or the standard you set? That is the difference between “this scored 60, try a different model” and “this scored 60 because the prompt never told it the refund window, fix the prompt.” One is guesswork. The other is a fix.

A bare score cannot do this on its own, and this is the part a plain eval script leaves you to work out by hand. It is worth running your evaluations somewhere that returns a diagnostic alongside the score, one that names which layer caused the low mark instead of leaving you to guess. That is the difference between a number and an answer.

It changes, so measure more than once

A last thing the acronym lists never mention: the right answer moves. The model you measured can update and slip. A rival can get better, or reach the same quality for less. The metric you were happy with this quarter can quietly stop being true, and nothing tells you. So the useful version of all of this is not a one-time grade. It is a reading you take again on a schedule, so the picture stays current.

The short version

Skip the acronym soup. Pick the few metrics that map to your actual job:

1. Write down what good means for your task first. Without that standard, every metric below is just a number with nothing to measure against.
2. Score quality against that standard with a judge that shows its reasoning, so you read the why, not just the number.
3. Measure cost per finished answer, not per token.
4. Watch quality per dollar. It is the one that decides your bill.
5. Run it more than once, because consistency is its own metric.
6. Use a diagnostic that names which layer broke, so a low score points you at the fix instead of leaving you guessing.
7. Re-measure on a schedule, because the right answer changes.

You do not need a research lab’s metric sheet. You need your task, your standard, and the handful of numbers that tell you whether your AI is good enough, at a price that makes sense.