AI model drift is not data drift

You shipped an LLM feature months ago. It worked, you moved on, and lately the answers feel worse, or the bill looks higher than it should for what you are getting. Nothing in your code changed. You did not touch the prompt. Yet the thing you signed off on is no longer the thing running in production.

Search for what is happening and you land on “model drift”, and almost everything written about it is describing a different problem from the one you have. The classical definition comes from machine learning you train yourself, and the fix it points you to does not apply to an app built on someone else’s model. The word is the same. The problem underneath is not.

Two things called drift

In classical machine learning, model drift means the world moved away from your model. You trained it on last year’s data, reality shifted, and the patterns it learned no longer match what it sees. Fraud patterns change, customer behaviour changes, and the model you trained is now out of date. This is the sense that most references, from IBM to Splunk, describe, and the two usual culprits are data drift and concept drift.

An app built on a hosted LLM has the opposite problem. You did not train the model, so you cannot retrain it, and the data it learned from is not yours to age. What moves is the model itself. The provider updates the weights behind the same API, and the same prompt you sent last month returns a different answer this month, often without an announcement or a version bump you can pin to.

That difference is the whole reason the classical advice does not help you. Every classical remedy ends in retraining, and you have nothing to retrain. The model is not yours. So the question stops being “how do I fix the model” and becomes “how do I know when it changed”, which is a different job with a different tool.

Drift is only one of three ways your pick goes stale

Here is the part the word “drift” hides. Your model drifting is only one of three ways the model you picked stops being the right one, and the other two have nothing to do with your model changing at all.

• Your model drifted. The provider updated it, and it now scores worse on your task than it did when you chose it.
• A rival got better. Your model is exactly as good as it always was, but someone else released a model that now beats it on your task.
• A rival got cheaper. Your model holds its quality and its price, but another model just reached the same quality for a fraction of the cost.

Only the first is drift. In the other two your model has not moved an inch. The field moved past it. The outcome is identical in all three, you are no longer running the best option, but your model only degraded in one of them. This is why watching for “drift” alone misses most of what actually makes a one-time model choice go stale.

What “best for my task” actually looks like right now

Before extrapolating to how any of this shows up over time, it helps to see what a single reading looks like, because the whole thing rests on being able to measure “best for my task, at this cost” at one moment. Take a real task. A support agent has to answer a stalled-delivery complaint under a policy.

Scoring two models against that policy, as a graded answer, gives a result that already makes the point about cost.

GPT-4o Mini and Claude Haiku 4.5 score within a fifth of a point of each other. One costs about nine times more per reply.

Both models clear the bar on quality. GPT-4o Mini scored 94.4 out of 100 and Claude Haiku 4.5 scored 94.2, close enough to call a tie on the answer itself. The gap is in the cost, where the same reply runs about nine times more expensive on one than the other. On a task you serve thousands of times, that is the difference that decides which model you actually want running, and you can only see it with the quality and the cost sitting side by side. The point for now is simply that “best for my task, at this cost” is a thing you can read off a real run, today.

Why you cannot know ahead of time

That reading was true the moment it was taken, and that is exactly the catch. Which model is the cheapest one clearing your bar is a fact about this week, not a permanent property. A model update, a rival’s new release, or a rival’s price cut can each change the answer, and none of them sends you a notice. The only way to know that today’s answer still holds is to take the same reading again later and compare.

Run the same task on a schedule and the three stale-making events each leave a visible mark against your bar.

Illustrative. Your model holds, then drifts down below the bar at month 4, while a rival climbs past it from the other direction.

Reading the shape tells you which of the three you are looking at. Your line dropping is drift, your model getting worse on the task you already defined. A rival’s line crossing above yours while yours holds is the field getting better, nothing wrong with your model except that it has been overtaken. A rival reaching your quality at a lower cost, which the same run shows in the cost column, is the re-sizing opportunity, where switching saves money at no loss of quality. You cannot predict any of these, but you can catch all of them the moment they happen, as long as you are still measuring.

One run reads all three, if it holds several models

A single-model check can only ever show you the first of the three, because with nothing to compare against, a steady score just looks steady and a falling score cannot tell you whether you slipped or everyone did. Keeping several models in the same scheduled run is what separates the causes.

• Your model drops while others hold. That points at drift in your pick, and the steady models are your control group.
• Others climb while yours holds flat. That is the field getting better, and your model being overtaken rather than failing.
• Another model reaches your bar at a lower cost. That is the re-sizing opportunity, visible because the run scores cost alongside quality on every model, not just the one in production.

The mechanics of setting that up, catching a drop the day it happens, and telling your own pipeline changes from a provider’s update, are covered in how to monitor LLM quality over time. The groundwork, deciding what a good answer is and scoring against it, is in how to evaluate an LLM.

The takeaway is narrower and more useful than “watch out for drift”. A model choice is not a decision you make once. It is a reading that was true on the day you took it, and it goes stale three different ways, only one of which is your model’s fault. Measuring it once told you what was best that day. Measuring it on a schedule is what keeps that answer true.