LLM-as-a-judge: how one model scores another
You have an AI answering questions in your product, and you want to know whether its answers are any good. You could read them yourself, but there are thousands of them, and reading a sample by hand does not scale and does not stay consistent from one sitting to the next.
So instead you hand that job to a second model. You give it the answer, you give it a standard for what a good answer would be, and it returns a score with its reasoning. That setup, one model grading another against a standard, is what people mean by LLM-as-a-judge. The rest of this post shows it running on a real task and then looks at the one thing you have to check before you trust a grade it gives you.
Watch a judge grade three real answers
Start with a question your support AI has to handle, where there is a clear right answer.
The question and the policy behind it
A customer asks: “Can I get a refund? I bought the annual plan 20 days ago.”
Your policy: full refund within 14 days of purchase, no refunds after 14 days.
The purchase was 20 days ago, so the correct answer is no. You write that down as the standard the judge will grade against.
The correct answer (what the judge grades against)
Three models each take a turn at the reply. Here is what they wrote.
GPT-4.1 Mini's reply
GPT-4o Mini's reply
DeepSeek V4 Flash's reply
Now the judge grades each one against the correct answer.
The order matches what you would decide reading them yourself. GPT-4.1 Mini scores highest at 91.8, because it states the 14-day rule and applies it to the customer’s 20 days to reach the right conclusion. GPT-4o Mini is close behind at 83.6, correct but never mentioning the 20 days that make the answer clear. DeepSeek V4 Flash lands at 32.9, because “No refunds after 14 days” states the policy but never actually answers the person asking.
That is the whole value. You read one score per answer instead of a thousand replies, and the ranking is the one you would have reached by hand, so you can trust it across far more answers than you could ever check yourself.
Have a judge score your answers against your standard
Set a gold standard or a rubric, and every answer comes back scored with the reasoning shown. Judge scoring starts on Pro.
Read the reasoning, not just the number
A judge gives you a score and the reasoning behind it, and the reasoning is the part that tells you whether to trust the score. A number on its own asks you to take it on faith. The reasoning shows you the judge applied your standard the way you would have.
Look back at why DeepSeek scored 32.9. The judge did not just say “low.” It said the reply states the rule but never answers the customer, which is exactly why a human would mark it down too. When the reasoning lines up with how you would have judged it, the score earns your trust. When it does not, that is your signal to tighten the standard you gave it, because a vague standard is what produces a grade you cannot explain.
This is also why the standard matters more than the model doing the grading. The judge is only applying the definition of good that you handed it. Give it a clear one and the grades are consistent and defensible. Give it a vague one and even a strong judge will produce scores you cannot stand behind. Writing a good standard is where most of the quality of an evaluation comes from.
A judge lets you measure quality across every answer your AI gives, not just the handful you have time to read. Set the standard well, read the reasoning the first few times to confirm it grades the way you would, and from then on you have a consistent read on quality at a scale no human review could reach.