How to monitor LLM quality over time

In how to evaluate LLM output we showed that picking a model is a trade-off you control. You set a quality bar for the task and take the cheapest model that still clears it, and sometimes the fastest one too when your pipeline is serving users live. You evaluated your models that way, picked one, and shipped it.

That score was true the day you measured it, but it does not stay true. The company behind the model updates it without telling you, someone edits the prompt, a document the model reads from changes, and the quality you signed off on slips without a single alarm going off.

The fix is to stop treating evaluation as a one-time check. You take the same task and the same standard you already set up, and you run them on a schedule so the score is tracked instead of taken once. A scheduled evaluation like this is a benchmark, and it turns quality from something you hope is holding into something you watch.

Watch a benchmark catch a drop

A benchmark runs your evaluation on a timer, every day say, and records the score each time. As long as the score holds, you do nothing. When it falls below the bar you set, you get told. Here is what a week looks like when something breaks on Friday.

Illustrative. Quality holds near 93 all week, drops to 79 on Friday when a model update lands, and stays down until the cause is fixed.

The drop on Friday is the whole point. Without the benchmark, nobody runs an evaluation that day, so the model keeps serving worse answers through the weekend and the first you hear of it is a customer complaint. With it, the score crosses below your bar on Friday, an alert fires, and you are looking at the problem the same day it started instead of days later.

This is the regression test your AI has been missing. In normal software you would never ship without tests that fail when something breaks. A scheduled benchmark is that safety net for a part of your product that has no fixed behaviour to unit-test, only quality you can measure.

Tell your own change from the model’s

A drop tells you quality fell. The next question is always whether you broke it or the provider changed the model underneath you, and running more than one model in the same benchmark answers it. The two causes make two different shapes on the chart.

Illustrative. On Wednesday all three models drop together. On Saturday only Model B drops while the other two hold.

• They all drop together (Wednesday) – the cause is something you changed that they all share, like a prompt edit or a document they all read from.
• Only one drops (Saturday) – that provider changed its model, and the other two are your control group proving the rest of your pipeline is fine.

That one distinction decides what you do next. A shared drop sends you to your own recent changes. A single-model drop sends you to that provider, to pin a version or switch away from it. Without the comparison both look identical, a number that went down with no clue why.

Three jobs the same schedule covers

The same scheduled benchmark does three jobs at once.

• A regression test – it catches the day your quality falls, so you fix it before users feel it.
• A smoke test – a steady heartbeat confirming that on an ordinary day your pipeline still answers the way it should, so a green run is itself a signal that nothing moved.
• An ongoing model-selection decision – because it scores several models every run, not just the one in production, it keeps answering which model is the cheapest that still clears your bar. The winner is not fixed. A model that was overkill last month can become the right call when a cheaper one improves, or your current pick can fall behind, and the benchmark surfaces that without you re-running the search by hand.

Wire it into how you ship

A benchmark is useful as something you watch. It gets more useful when the thing you are about to deploy reads it first. The results are available two ways beyond the dashboard, so the latest answer can reach your release process directly.

Over the API, your CI pipeline can pull the most recent result, or the aggregate across the last several runs, and gate a deploy on it, so a model only ships if it is still the cheapest one clearing your bar this week.

Over MCP, your agent can pull the same answer in the moment, as a step in getting ready to release. It looks like this.

Nobody re-ran the comparison by hand. The agent read the current benchmark, applied the bar you already set, and surfaced that the config was pointing at a model costing twice what the job needs. You make the call and deploy. The decision you made once has become something your pipeline keeps current on its own, so you are always on the most efficient model that still delivers the quality you need. This is what agentic optimization looks like in practice: your agent reading live measurement and keeping your stack on the right model without you running a thing.

You already did the hard part when you set up the evaluation in the first place. Putting it on a schedule is the step that keeps its answer true, so the model you proved was good enough stays that way, or tells you the moment it stops. Setting up the underlying evaluation is covered in how to evaluate an LLM.