LLM-as-a-judge: how one model scores another
You have an AI answering questions in your product, and you want to know whether its answers are any good. You could read them yourself, but there are thousands of them, and reading a sample by hand does not scale and does not stay consistent from one sitting to the next.
So instead you hand that job to a second model. You give it the answer, you give it a standard for what a good answer would be, and it returns a score with its reasoning. That setup, one model grading another against a standard, is what people mean by LLM-as-a-judge. The rest of this post shows it running on a real task and then looks at the one thing you have to check before you trust a grade it gives you.