System Prompts Guide

A system prompt is the standing instruction you give a model before the user prompt runs. It sets persona, tone, constraints, scope, and output format. The model reads it on every call – it is the difference between a model that answers anything in any style and a model that behaves like your production system.

In LLM Prover, system prompts are saved, named templates you attach to any comparison, evaluation, or benchmark from a dropdown. You write one once and reuse it across runs. This means every run reflects your actual production setup rather than testing the model in isolation – and it means you can A/B test two system prompts against each other on the same prompt and models to see exactly what changes.

For a quick, no-signup version of that last idea, the free System-Prompt A/B tool runs one question under two sets of standing instructions and shows what each costs.

This guide covers when to use a system prompt, how to write one that produces consistent results, how it interacts with rubrics and context stores, and how to use LLM Prover to test and improve it.

When to use a system prompt

Reflect your production setup

If your application uses a system prompt in production, every comparison or evaluation without it is testing a different system than the one you ship. Attach your production system prompt to get results that predict real behaviour.

Constrain model behaviour

Models have defaults – they add caveats, use casual language, produce variable output lengths, and make formatting decisions you did not ask for. A system prompt replaces those defaults with your requirements. Tone, persona, output format, scope limits, prohibited content – all of these belong in the system prompt, not the user prompt.

A/B test two approaches

Run the same user prompt and models twice – once with each system prompt. The score difference tells you exactly how much the system prompt is contributing. This is the fastest way to answer “is my system prompt helping or hurting?”

How to create and attach a system prompt

1. Navigate to System Prompts in the sidebar (Config section)
2. Click New System Prompt
3. Give it a name that identifies its purpose – “Customer support v2”, “Legal review strict”, “Concise JSON output”
4. Write the content (see the next section for guidance)
5. Click Save

To attach to a run: on any comparison, evaluation, or benchmark form, select the system prompt from the System Prompt dropdown before submitting. The system prompt name appears in the result config box so you can always see which prompt was used.

Saved system prompt limits: Starter: 1. Pro: 10. Pro+: 20. Enterprise: unlimited.

What to put in a system prompt

Persona and role

Tell the model who it is. This shapes tone, vocabulary, and the frame it uses to interpret the user prompt.

Output format

If your application expects a specific structure, specify it here. The model will not guess correctly every time without explicit instruction.

Constraints and prohibitions

What the model must not do. Explicit prohibitions are more reliable than implicit ones.

Tone and register

Especially important when comparing models – without a tone constraint, different models have different defaults and you end up comparing house styles, not capabilities.

System prompts and rubric scoring

The system prompt shapes the response; the rubric scores it. If they are misaligned, scores will be low even for responses that are correct given the system prompt’s instructions.

The alignment problem

If your system prompt says “respond in two sentences maximum” and your rubric scores for completeness, the judge will penalise responses for being brief – which is exactly what you asked for. The rubric does not know about the system prompt’s constraints.

The fix

Review your system prompt alongside your rubric before running an evaluation. Every constraint in the system prompt that affects output shape should be reflected in the rubric. If the system prompt says “JSON only”, the rubric should not score for prose structure.

Using the diagnostic judge

When scores are lower than expected and the rubric looks correct, check the Diagnostics tab. The diagnostic judge often surfaces system prompt / rubric misalignment as a finding – it reads the full pipeline including the system prompt and flags when a constraint is working against a criterion.

System prompts and context stores

When you attach both a system prompt and a context store to a run, the model receives the system prompt, the retrieved context chunks, and the user prompt – in that order. The system prompt should tell the model how to use the context.

Without explicit instruction, models handle context inconsistently – some summarise it, some quote it verbatim, some ignore it when it conflicts with their training knowledge. A system prompt that specifies context behaviour produces more consistent, scoreable results.

Recommended pattern:

This is especially important for compliance evaluations where the model must score against your document, not its training knowledge of the regulation.

A/B testing two system prompts

1. Create two system prompts – name them clearly so results are identifiable (“Support v1”, “Support v2”)
2. Run a comparison or evaluation with System Prompt A – note the run name or timestamp
3. Run the same comparison or evaluation with System Prompt B – same prompt, same models, same rubric
4. Compare the results side by side in your run history

What to look for:

• Quality score difference – which system prompt produces higher rubric scores?
• Per-criterion breakdown – does one system prompt improve accuracy but hurt tone?
• Cost and latency – a more constrained system prompt often reduces output tokens and therefore cost
• Consistency – run each version twice and check whether scores are stable. High variance means the system prompt is underspecified

Common mistakes

Putting constraints in the user prompt instead of the system prompt

Format instructions, tone requirements, and scope limits belong in the system prompt. Putting them in the user prompt means they apply to one call only and cannot be reused or tested systematically.

Changing the system prompt between runs

If you change the system prompt between runs, you are comparing system prompts, not models. Keep it identical across all models in a comparison run. Use separate runs to test system prompt variants.

Overloading the system prompt

A system prompt that tries to do everything – persona, format, tone, constraints, examples, edge cases – becomes hard to reason about and hard to score against. Start with the minimum required for your use case. Add constraints only when a specific failure mode appears.

Not testing the system prompt before benchmarking

Run a single evaluation with a response you know is correct before attaching a system prompt to a benchmark. If the scores do not reflect that the response is good, the system prompt and rubric are misaligned. Fix it before scheduling recurring runs.

What’s next