The system prompt is not neutral. Here is what happens when you measure it.
Most teams write a system prompt once. It goes into a config file, gets added to every call, and is never revisited. It is not treated as a variable – it is treated as a setting. We ran a 2x2 benchmark to find out what it is actually doing:
- two prompt variants
- two system prompt variants
- five models
- same task, same rubric
The results show that the system prompt is one half of a two-variable equation, that the two variables interact differently depending on which model you are running, and that getting both right costs less – not more – than getting neither right.
The task
Support ticket summarisation. A Pro subscriber reports that their scheduled daily benchmark suite stopped running. They want a fix or a refund. The rubric scores three criteria: accuracy (the summary states only what the ticket explicitly says, no inferred goals), completeness (exactly three bullets covering want, cause, and next step), and actionability (correct handling of absent information – “not stated” and “not specified” are the required exact phrases when the ticket provides no cause or next step).
The rubric is designed to penalise inference. That is the most common failure mode in summarisation tasks: a model that reads between the lines and adds context the ticket does not contain. A rubric that rewards only what is explicitly stated surfaces this failure clearly.
We ran this as a benchmark suite with a custom rubric in LLM Prover. Two runs per cell.
The two prompts:
Prompt A -- vague
Prompt B -- specific
The two system prompts (SP):
System prompt A -- vague
System prompt B -- specific
Five models:
- Claude Opus 5
- Grok 4.5
- Qwen 3.8 27B
- GPT-4o Mini
- GPT-4.1 Nano
The baseline
Before looking at what optimisation does, it helps to see what no optimisation looks like.
Every model either collapsed or produced inconsistent outputs. The diagnostic findings pointed to the same structural cause: the prompt gave no output format, and the system prompt gave no content constraints. Both variables were loose. The models filled the gap with their own defaults – labelled sections, prose summaries, headers – none of which matched the rubric. One finding from the run:
Grok scored 25 rather than 0 because it partially followed a three-part structure on one of two runs. Whether that reflects stronger default summarisation priors or run variance is not distinguishable from two runs. That is itself a finding about what a single benchmark run can and cannot tell you.
The full picture
Four cells. The pattern across them: output quality is a function of both variables together, and the interaction is not symmetric across models.
| Vague SP | Specific SP | |
|---|---|---|
| Vague prompt | Near zero across the board | Grok and Claude: 66.7. Mini, Nano, Qwen: 16.7 |
| Specific prompt | Grok, Mini, Nano, Qwen: 83-100. Claude: 16.7 | All five models: 100 |
The third cell is where the interesting finding lives.
Precision is cheaper
Before examining why Claude behaves differently, the cost implication of the bottom-right cell:
| Model | Per call | Monthly @100/day |
|---|---|---|
| GPT-4.1 Nano | $0.00004 | $0.13 |
| GPT-4o Mini | $0.00006 | $0.17 |
| Qwen 3.8 27B | $0.00040 | $1.19 |
| Grok 4.5 | $0.00164 | $4.91 |
| Claude Opus 5 | $0.00341 | $10.23 |
Specific prompt + specific system prompt. All five models scored 100.
Claude at vague+vague: $0.01884/call. Claude at specific+specific: $0.00341/call. A 5.5x cost reduction, and the scores went from 0 to 100. Precise inputs produce shorter, more constrained outputs. Fewer output tokens, lower cost, better scores.
Claude Opus 5 -- quality score by input precision
Run the 2x2 on your own task
Two prompts, two system prompts, five models. The gap between your baseline and your ceiling is your headroom.
Get startedClaude is a system prompt loyalist
Specific prompt + vague SP scores:
Specific prompt + vague SP -- quality scores by model
Every other model followed the specific prompt and treated the system prompt as secondary context. Claude did the opposite. The diagnostic finding explains the mechanism:
The vague system prompt said “cover everything.” Claude did. It ignored the three-bullet structure, the exact phrasing requirements, and the constraint against inference. The score was consistent across both runs at this cell – 16.7 each time. This is not run variance. It is a structural behaviour.
Claude is the most instruction-following model in this set. That is usually an advantage. Here it is a liability: when the system prompt and the prompt give conflicting signals, Claude resolves the conflict in favour of the system prompt. The cheaper models resolve it in favour of the prompt.
The fix is to make the system prompt self-reinforcing. System prompt B explicitly restates the content rules from the prompt: exact phrasing for absent information, one sentence per bullet, no inference. With both inputs aligned, Claude scores 100 on both runs.
The redundancy is the point. For high-stakes tasks, the system prompt and the prompt should say the same thing about content structure, not divide the labour between them.
You cannot tune your stack on one model and ship it on another
This is the operational finding that goes beyond prompt optimisation.
If you prototype on GPT-4o Mini – because it is cheap and fast to iterate on – and then switch to Claude in production, you have not tested the system you are running. You have tested a different system that happens to use the same prompt text.
Quality score -- dev (GPT-4o Mini) then prod switch to Claude Opus 5, same prompt + vague SP
Mini weights the prompt. Claude weights the system prompt. The same inputs produce different outputs for structural reasons, not capability reasons.
The data makes this concrete. In the specific prompt + vague SP cell, Mini scored 87.5 and Claude scored 16.7. Same prompt. Same system prompt. A 70-point gap driven entirely by how each model resolves the conflict between them.
The same applies in reverse. A stack tuned on Claude – with a carefully specified system prompt – may produce unexpected results when a task moves to a cheaper model. The cheaper model may follow the prompt and partially ignore the system prompt, producing outputs that look correct but are not being evaluated against the same constraints.
The cheap model as a canary
The data shows a useful diagnostic signal that runs in the other direction.
In the specific prompt + vague SP cell, Nano and Mini scored 83-87 while Claude scored 16.7. In the specific prompt + specific SP cell, all five models scored 100. The gap between those two cells for Nano and Mini is small – 83 to 100. The gap for Claude is large – 16.7 to 100.
In this instance, if your cheap model scores well, your prompt is doing the work. The system prompt is not the bottleneck. If only your capable models score well, your system prompt is compensating for a prompt that needs a rewrite – and you are paying for model capability you should not need.
Running Nano alongside your production model is not about replacing it. It is a diagnostic. A Nano score close to your production model’s score means the prompt is the signal. A large gap means the system prompt or model capability is doing compensatory work that a prompt rewrite could eliminate – and that a cheaper model could then handle.
What to do
Run the 2x2 on your own task. It takes four evaluation runs.
| Vague SP | Specific SP | |
|---|---|---|
| Vague prompt | Cell 1 – your baseline | Cell 3 – isolates the SP variable |
| Specific prompt | Cell 2 – isolates the prompt variable | Cell 4 – your ceiling |
The gap between cell 1 and cell 4 is the headroom in your current stack. The pattern across cells 2 and 3 tells you which variable is limiting you – and which model you need to test on.
Run it again after any system prompt change, model upgrade, or task change. The 2x2 is not a one-time audit. It is a regression test.
This post covers Lever 4 (system prompt audit) from the ROI Maxxed AI framework – 14 levers for building an LLM stack where every task gets the cheapest model that delivers the quality it needs.