More quality, less spend: a practical framework for your LLM stack

LLM decisions are typically made once and rarely revisited – not because teams are satisfied, but because there has been no systematic way to measure whether a different choice would be better. This guide walks through the 13 levers that determine your stack’s cost and quality, shows real benchmark data for each one, and gives you a framework for measuring and improving all of them continuously.

Before the framework, one data point. A 25-point quality improvement on a real benchmark run, with no model change. The only variable was the content in the RAG store.

Not every lever applies to every stack. The table below maps your current situation to the right starting point.

The evidence

Four data points from real benchmark runs. Each one shows a lever that most teams are not measuring.

The framework

Before you can pull any of the 13 levers, you need a benchmark. A benchmark is a named prompt you run repeatedly against the same models on a schedule. It is the measurement instrument that makes every lever in this guide actionable.

Without a benchmark, you are comparing snapshots taken at different times under different conditions. With one, you are comparing the same task, the same models, and the same scoring criteria – run after run. Every lever in this guide produces a number. The benchmark is how you get that number.

Lever 1: Quality tolerance calibration

Before you can find the cheapest model that clears your quality bar, you need to define what your quality bar actually is. A quality threshold set at 80 qualifies different models than one set at 95. Most teams skip this step and end up either over-specifying – paying for quality they do not need – or under-specifying, shipping outputs that do not meet their actual standard.

Quality tolerance is not a feeling. It is a number. Setting it precisely is the first step in building a stack that is neither over-engineered nor under-specified.

Lever 2: Model selection

The cheapest model that clears your quality threshold is the right model for that task. Finding it requires running the comparison on your actual prompts – not a leaderboard someone else published on their data.

For this task, Qwen 3.8 27B clears a quality threshold of 95 at $2.38/month and 1136ms. Claude Opus 5 also clears it at $27.35/month and 5660ms. Both pass – the cost difference is 12x. The right choice depends on whether that quality gap matters for your use case.

Lever 3: Prompt optimisation

A vague prompt produces inconsistent results across models. A constrained prompt – one that specifies format, structure, and scope – produces more consistent outputs and often qualifies a cheaper model that could not clear the bar before.

With a vague prompt, Qwen scored 83.7. With a specific prompt covering the same task, it scored 88.2 – a 4.5-point lift at essentially the same cost. The prompt changed which models cleared the quality threshold. GPT-4.1 Nano dropped 3.4 points with the specific prompt, which is also useful data: not every model responds the same way to constraints.

Lever 4: System prompt audit

A system prompt is added to every call. A well-specified system prompt can lift quality and reduce cost by constraining output format and length. Most teams write one once and never measure whether it is still necessary, whether it could be shorter, or whether it is helping or hurting quality.

With a specific prompt and no system prompt, Claude scored 91.7 and Qwen scored 88.2. Adding a structured system prompt lifted both to 92.5 and reduced cost per call – the system prompt produced shorter, more constrained outputs. The difference between lever 3 and lever 4 is the variable: lever 3 tests the prompt text, lever 4 tests the system prompt layer on top of it.

Lever 5: Pipeline quality – RAG and context

On a complex RAG pipeline, the quality of what you feed the model determines which models are even eligible. A better pipeline unlocks cheaper models that could not clear the bar before.

GPT-4o Mini passes on a simple prompt. On a RAG pipeline with a quality threshold of 95, it passes half the time. The pipeline complexity is the variable – not the model.

Lever 6: Store content – the variable most teams never measure

If your RAG store content is ambiguous, incomplete, or poorly structured, no model can compensate for it. Testing two versions of your store against the same model and prompt tells you whether improving your content improves your outputs – and by how much.

For teams running RAG on a knowledge base, the quality of the underlying documents is the ceiling on what any model can achieve. A benchmark with two store versions makes that ceiling visible and measurable for the first time.

Content teams can now answer a question they have never been able to answer before: did the work we did on our documents actually make the AI better?

Lever 7: Diagnostic improvement loop

When a score is lower than expected, the question is which of the seven pipeline layers caused it. The diagnostic reads all seven – prompt, system prompt, context, parameters, criterion, model, and scoring – and returns a specific finding with a specific fix.

The diagnostic did not say the model was wrong. It said what to change and where. This is the finding that most teams never reach because they optimise prompts and parameters while the real problem sits in the source documents.

Lever 8: Architecture – task-model matching

Most teams apply one model uniformly across every task. Every task produces different output complexity. A simple classification task does not need the same model as a customer-facing response that will be read by a regulator.

A team running four LLM tasks – summarisation, classification, generation, and extraction – with a single frontier model is almost certainly over-spending on at least three of them. Each task is a separate benchmark with its own threshold. The table shows four separate benchmarks, each with a threshold matched to that task’s quality requirement. Run each task type independently – the benchmark shows you which model clears your threshold at the lowest cost.

Illustrative task types and quality scores. Cost figures are real per-call costs from benchmark runs. Each task is a separate benchmark with its own threshold.

Lever 9: Fine-tune ROI

A fine-tuned model trained on your domain data can outperform frontier models on your specific tasks. Running the comparison on your own context, with your own quality threshold, tells you whether the training investment is paying off – before you commit to the next run.

Illustrative – register your endpoint and run this benchmark on your own fine-tune.

The benchmark does not tell you fine-tuning is wrong. It tells you whether this fine-tune, on this task, at this cost, is the right choice.

Lever 10: Drift monitoring – catch regressions before your users do

Model providers update their models without announcing it. The model you benchmarked last quarter may behave differently today. Scheduled benchmarks catch that change before it reaches your users.

Teams that pin to a specific model version avoid unexpected drift but miss improvements. Teams that do not pin get improvements automatically but may get regressions too. Drift monitoring gives you the signal either way.

Drift monitoring is reactive – it catches a regression you did not cause and did not expect. You find out first, not from your users.

Lever 11: Re-optimisation – find better options proactively

Drift monitoring catches regressions. Re-optimisation finds opportunities. A new model enters the market. A provider drops their pricing. A model that could not clear your quality bar six months ago now can. A scheduled benchmark tells you when the landscape has shifted in your favour.

Lever 12: Threshold alerting

Once you know your quality, cost, and latency thresholds, you can set alerts that fire the moment any dimension breaches them. You stop monitoring dashboards and start getting notified when something actually needs attention.

Lever 13: Compliance – model updates break compliance posture

If your AI outputs need to align with internal policies, regulatory documents, or operational standards, a model update can break that alignment without warning. Scheduled compliance benchmarks catch the change before it reaches a user, an auditor, or a safety-critical system.

In the real benchmark run shown in the evidence section, the Availability criterion scored 0. The diagnostic pointed to the criterion definition – it applied a tier-stating requirement the prompt did not supply. That is a Criterion stage finding: the rubric itself needed tightening, not the model.

The improvement loop

The levers are not independent. Improving one often unlocks another – and the compound effect is where the real ROI lives.

A single diagnostic finding that improved three levers simultaneously:

One finding. Three levers improved. The diagnostic pointed to the store. Fixing the store changed which models cleared the threshold. The improvement loop ran once and produced a compounding result.

This is the loop: measure, diagnose, fix, re-measure. Each pass produces a better pipeline and a lower cost floor.

Where to start

Start with the lever most relevant to where your stack is today.

You do not know which model to use. Start with lever 1 – set your quality threshold. Then lever 2 – run the comparison. The cost and quality table will show you where to draw the line.

Your outputs are inconsistent and you do not know why. Start with lever 7 – run a diagnostic. The finding will tell you which layer to fix. Most of the time it is not the model.

Your AI costs are growing and you cannot explain why. Start with lever 4 – audit your system prompts. Then lever 3 – tighten your prompts. Then lever 2 – check whether a cheaper model now qualifies.

You now have a framework for measuring and improving every layer of your LLM stack. Thirteen levers. A benchmark as the measurement instrument. A diagnostic that reads all seven pipeline layers and returns specific, actionable findings. And an improvement loop that compounds – each fix unlocks the next. The stack is never done, because the landscape keeps changing. The benchmark keeps it current.