How to gate your CI/CD on LLM output quality
Your pipeline already blocks a merge when a unit test fails. The LLM feature in the same codebase has no such gate. A model that quietly got worse ships like nothing happened, and the first you hear of it is a user telling you the output looks off.
The fix is the idea you already trust for code, pointed at output quality. You score the model’s answers against a fixed standard on a schedule, and you let the pipeline read that score and act on it, the same way it reads a test result and acts on it.