Prompting Guide

Vague prompts produce noise, not signal. When you run a comparison with an underspecified prompt, you are not testing which model is better – you are testing which model makes the most plausible guess about what you actually wanted.

This guide covers how to write prompts that give you results you can act on.

Why vague prompts break comparisons

A prompt like “Summarise this.” is not one prompt. It is five different prompts depending on which model reads it. One model returns a paragraph. Another returns bullets. A third returns a sentence. A fourth adds interpretation the source text never contained.

When you compare those outputs, you are comparing answers to different questions. The quality scores reflect that – not because the models are inconsistent, but because you gave them room to be.

The specificity principle

The more constrained the prompt, the more meaningful the comparison. Constraints are not limitations – they are the thing that makes results comparable.

A constrained prompt specifies:

• The output format (one sentence, three bullets, a table)
• The length limit (under 15 words per bullet, 200 words maximum)
• What to include (the three most important data points)
• What to exclude (no interpretation beyond what the text states)

Every constraint you add is a variable you remove from the comparison. Fewer variables means the score difference between models reflects actual capability, not formatting preference.

A real comparison: vague vs specific

We ran both versions of the same task through LLM Prover. The source text was a short passage about open source vs frontier model performance on MMLU benchmarks.

Prompt A (vague):

Results:

Prompt B was faster and cheaper for two of three models despite being a longer, more specific instruction.

The outputs from Prompt B were directly comparable: each model returned one summary sentence and three bullets. Prompt A outputs ranged from a single sentence (GPT-4o Mini) to a multi-paragraph response with added interpretation (DeepSeek, which also ignored the format and returned 544 tokens vs 64 for the others).

One model that followed the format precisely: Grok 4.3. One that partially followed it: GPT-4o Mini. One that ignored it: DeepSeek V4 Flash, which prefixed its response with “Summary:” and “Key data points:” headers not requested.

That format compliance difference is itself a signal – and you only see it when the prompt is specific enough to make non-compliance visible.

What to constrain

Not every prompt needs every constraint. Match the constraint to what you are actually measuring.

Format constraints – use when output structure matters for your use case:

• “Return a JSON object with keys: summary, confidence, source_quote”
• “Respond in exactly three bullet points”
• “One sentence only. No preamble.”

Length constraints – use when you need outputs to be comparable in scope:

• “Under 50 words”
• “Each bullet under 15 words”
• “Maximum 200 words”

Scope constraints – use when you want to test faithfulness to source material:

• “Do not add interpretation beyond what the text states”
• “Only use information from the provided context”
• “If the answer is not in the text, say so explicitly”

Role constraints – use when you are testing instruction-following in a specific context:

• “You are a customer support agent. Respond only to the question asked.”
• “You are reviewing a contract. Flag only clauses that create liability.”

System prompts

A system prompt sets the model’s default behaviour before your user prompt runs. It is the right place for constraints that apply to every call in a workflow – persona, output format, tone, scope.

When comparing models with a system prompt, keep the system prompt identical across all models. Changing it between runs means you are comparing system prompts, not models.

To test the effect of a system prompt, run two comparisons: one with it, one without. The score difference tells you how much the system prompt is contributing vs the model’s defaults.

Adding context (RAG)

When you attach a context store to a comparison, you change what you are testing. Without a context store, you are testing the model’s general knowledge and instruction-following. With one, you are testing retrieval quality and how well the model uses provided context.

Both are valid tests. They are different tests. Run them separately if you want to understand each variable independently.

If your production system uses a context store, test with one. The comparison result without it will not predict production behaviour.

Common mistakes

Testing the model’s knowledge instead of its capability on your task. “What is the capital of France?” tells you nothing useful about which model to use for your customer support workflow. Use prompts from your actual use case.

Changing the prompt between models. If you adjust the wording for one model because it “needs more context”, you are no longer running a fair comparison. Write one prompt that works for all models, or accept that you are testing prompt sensitivity.

Ignoring format compliance. A model that returns the right answer in the wrong format may fail in production even if its quality score is high. Check whether each model followed your format constraints, not just whether the content was accurate.

Using a single prompt to represent a task. One prompt is one data point. If your use case has variation – different customer questions, different document types, different input lengths – build a benchmark suite with representative examples. A single prompt comparison is a starting point, not a conclusion.

What to do next

• Benchmark Suite Guide – run your prompt set on a schedule and track results over time
• Rubric Writing Guide – score outputs against criteria you define, not just model defaults
• Scoring Guide – understand what the quality scores measure and when to trust them