Reference
Inference Parameters Guide
What temperature, seed, max tokens, and the other inference parameters do -- and why locking them is the difference between a benchmark and a lottery.
The parameters form sits below the prompt in every run type. The label says it plainly: match your production LLM settings so benchmark results reflect real-world behaviour. This guide explains what each parameter does, when to change it, and why getting this right matters.
What each parameter controls
Leaving any field blank sends no value to the provider, which then applies its own default. Provider defaults vary – if you need reproducible results, set values explicitly rather than relying on placeholders.
| Parameter | Placeholder | What it controls | Tier |
|---|---|---|---|
| Temperature | 0.7 | Output randomness. Higher = more variation. Lower = more deterministic. | All |
| Max tokens | 1000 | Maximum output length. Caps cost and affects completeness scores. | All |
| Seed | e.g. 42 | Fixes the random state for near-identical outputs on repeated runs. | Pro |
| Top-p | 1.0 | Nucleus sampling threshold. Rarely needs changing. | Pro |
| Frequency penalty | 0.0 | Discourages repetition of tokens already in the output. | Pro |
| Presence penalty | 0.0 | Discourages returning to topics already covered. | Pro |
| Prompt caching | toggle | Caches repeated prompt prefixes to reduce cost. Anthropic only – auto-enabled by OpenAI and Grok. | All |
Temperature
Temperature controls how much randomness the model introduces when selecting each output token. At high values the model explores a wider range of possible continuations. At low values it converges on the most probable one.
The practical effect on benchmark scores:
DATA ORDER A -- Temperature variance (pending)
Goal: Show score spread at high temperature vs stability at low temperature + seed on the same task.
Prompt: “Summarise the support ticket in exactly three bullet points: what the customer wants, what caused the problem (if stated), what the next step should be. If no cause is stated in the ticket, write exactly ’not stated’ in the second bullet. If no next step is stated in the ticket, write exactly ’not specified’ in the third bullet.”
System prompt: none
Context (paste inline – do not use a store): “Hi, I’m a Pro subscriber and my scheduled daily benchmark suite stopped running three days ago. I set it up two weeks ago and it was working fine. I haven’t changed anything on my end. I need this fixed urgently or I’d like a refund for the days it hasn’t been running.”
Gold standard: none
Rubric: Create a rubric named “Ticket Summary – Temp Variance” with these three criteria:
Criterion 1 – Accuracy | Weight: 3 The response states only what the ticket explicitly says. The first bullet must state the customer wants the suite fixed or a refund (both acceptable, either alone acceptable). The second bullet must state the cause is not stated, using exactly the phrase “not stated”. The third bullet must state the next step is not specified, using exactly the phrase “not specified”. A response that infers goals beyond what is stated, or omits the required exact phrases, fails this criterion.
Criterion 2 – Completeness | Weight: 2 The response contains exactly three bullet points covering (1) what the customer wants, (2) the cause, (3) the next step. A response with fewer or more than three bullets scores zero.
Criterion 3 – Actionability | Weight: 1 The response uses exactly “not stated” in the second bullet when no cause is given, and exactly “not specified” in the third bullet when no next step is given. Any other phrasing for absent information fails this criterion.
Model: GPT-4o Mini Condition 1: temperature 0.7, no seed – 5 runs – name “Temp Variance – High – Run 1” through “Run 5” Condition 2: temperature 0.1, seed 42 – 5 runs – name “Temp Variance – Low – Run 1” through “Run 5” Max tokens: 500 | All other parameters: leave blank
Paste back: overall score for each of the 10 runs
Results: Condition 1 (temp 0.7, no seed): Run 1: | Run 2: | Run 3: | Run 4: | Run 5: Condition 2 (temp 0.1, seed 42): Run 1: | Run 2: | Run 3: | Run 4: | Run 5:
When to use high temperature: Creative tasks, brainstorming, diversity testing. If you want to see the range of outputs a model can produce, high temperature surfaces it. Treat the spread across multiple runs as the result, not any single run.
When to use low temperature: Summarisation, classification, extraction, compliance scoring – any task where you need consistent, comparable results. Set temperature to 0.1 or lower.
Leaving the field blank uses the provider’s default, which may not be 0.7.
Seed
A seed fixes the random state the model uses to generate output. Combined with low temperature, it produces near-identical outputs on repeated runs of the same prompt.
Pick any integer and use it consistently across all runs in a suite. The specific value does not matter. Consistency does. If you change the seed between runs, you are running a different experiment.
Tier gate: Seed is a Pro feature. On Starter, the field is visible but disabled.
Provider support: Not all providers support seed. When set for a provider that does not support it, the value is ignored. Check the Models page for provider capability flags.
Max tokens
Max tokens caps the length of the model’s output. It affects two things directly:
- Cost – providers charge per output token. A lower cap reduces cost in proportion to how often the model would have generated more tokens than the cap allows.
- Completeness scores – a cap that truncates the response mid-answer will fail completeness criteria. The model did not produce an incomplete answer. You cut it off.
DATA ORDER B -- Max tokens truncation (pending)
Goal: Show completeness score drop when max tokens forces truncation mid-answer.
Prompt: Same as Order A – “Summarise the support ticket in exactly three bullet points…”
System prompt: none
Context (paste inline): Same ticket as Order A – “Hi, I’m a Pro subscriber and my scheduled daily benchmark suite stopped running three days ago…”
Gold standard: none
Rubric: Reuse “Ticket Summary – Temp Variance” from Order A
Model: GPT-4o Mini Temperature: 0.1 | Seed: 42 Condition 1: max tokens 500 – 2 runs – name “Max Tokens – Sufficient – Run 1” and “Run 2” Condition 2: max tokens 50 – 2 runs – name “Max Tokens – Truncated – Run 1” and “Run 2” All other parameters: leave blank
Paste back:
- Overall score per run
- Completeness criterion score per run
- finish_reason from result detail drawer for each run (look for “length” on truncated runs)
- Full response text from one Condition 2 run
Results: Condition 1 (max tokens 500): Run 1: overall | completeness | finish_reason: / Run 2: overall | completeness | finish_reason: Condition 2 (max tokens 50): Run 1: overall | completeness | finish_reason: / Run 2: overall | completeness | finish_reason: Truncated response text:
When truncation occurs, the response ends mid-sentence with no warning in the output text. To confirm truncation, open the result detail drawer and check the finish_reason field. A value of length indicates the output was cut at the token limit.
Reasoning models: If max tokens is set too low for a reasoning model (Claude, o-series), the response pane shows: “Reasoning model consumed full token budget – no visible response. Increase max_tokens.” Reasoning models consume tokens on internal chain-of-thought before producing visible output. Set a higher cap than you would for a standard model.
Leaving the field blank uses the provider’s default, which varies.
Top-p, frequency penalty, presence penalty
These are secondary levers. Most benchmarks do not need to touch them.
Top-p (placeholder 1.0) controls nucleus sampling – the model only considers tokens whose cumulative probability reaches the top-p threshold. At 1.0 all tokens are considered. Lowering it narrows the model’s vocabulary. Leave it at 1.0 unless you have a specific reason to constrain token selection.
Frequency penalty (placeholder 0.0) discourages the model from repeating tokens that have already appeared in the output. Useful for long-form generation tasks where repetition is a quality problem. For short structured outputs it has no meaningful effect.
Presence penalty (placeholder 0.0) discourages the model from returning to topics already covered. Similar use case to frequency penalty.
Tier gate: All three are Pro features. On Starter, the fields are visible but disabled.
Prompt caching
When enabled, repeated prompt prefixes are cached at the provider level. If your benchmark suite uses a fixed system prompt across many prompts, caching means the provider does not re-process that prefix on every call – reducing cost and latency.
Provider behaviour varies:
- OpenAI and Grok: Prompt caching is enabled automatically by the provider regardless of this setting. The toggle has no effect on these providers.
- Anthropic: Caching is opt-in. The toggle controls whether caching is requested for Anthropic calls. Enable it when running suites with a long fixed system prompt against Claude models.
- Other providers: Check the Models page for caching support.
The cost shown in results is always the actual provider-reported cost per call. There is no before/after comparison surface in the dashboard.
Why this matters for benchmarks
Run your first benchmark with locked parameters
Set your parameters once. Every run after that is a measurement, not a guess.
Get startedParameters must be set before a suite starts and kept consistent across every run. Changing them mid-suite breaks comparability.
The practical rule: decide your parameter profile before you run anything. Apply it to every run in the suite. If you need to test a different profile, create a new suite.
If your production system runs at temperature 0.3 with a 500-token cap, your benchmark suite should use the same settings. A benchmark run at temperature 0.7 with no cap does not predict production behaviour for a capped system.
The compliance case for low temperature and seed
For compliance benchmarks, parameter discipline is not optional. A compliance score produced at default temperature is not a compliance score. It is a sample. Run it again and you get a different number. You cannot show a sample to an auditor.
DATA ORDER C -- Compliance score variance (pending)
Goal: Show compliance score spread at high temperature vs stability at low temperature + seed.
Prompt: “Does our refund policy permit partial refunds for unused subscription days? What is the process for requesting one?”
System prompt: none
Context store: Create a new context store named “Refund Policy – Temp Variance”. Upload the following as refund-policy-temp-variance.txt:
Refund Policy
Subscription Refunds: Subscribers may cancel at any time from the Account page or billing portal. Cancellation takes effect at end of current billing period. Pro-rata refunds are available for unused subscription days. To request a refund, contact support at support@llmprover.com with your account email and reason for cancellation. Refunds are processed via Stripe and typically appear within 5 to 10 business days.
Top-Up Credits: Non-refundable once any portion has been consumed. Unused balance at cancellation is forfeited.
Monthly LLM Budget: Resets on billing anniversary date. Unused budget does not roll over and is not refundable.
Exceptions: Refund requests made more than 30 days after the charge date will not be processed. Refunds are not available for accounts suspended for policy violations.
Compliance store: Create a new compliance store named “Refund Policy Compliance – Temp Variance”. Upload the same document.
Rubric: Create a rubric named “Refund Policy – Temp Variance” with these two criteria:
Criterion 1 – Accuracy | Weight: 3 The response correctly states that pro-rata refunds are available for unused subscription days, and correctly states the process (contact support at support@llmprover.com with account email and reason for cancellation). A response that invents a different process, omits the contact method, or states refunds are not available fails this criterion.
Criterion 2 – Completeness | Weight: 2 The response addresses both questions: (1) whether partial refunds are permitted, and (2) the process for requesting one. Partial marks for covering only one aspect.
Model: GPT-4o Mini Condition 1: temperature 0.7, no seed – 5 runs – name “Compliance Variance – High – Run 1” through “Run 5” Condition 2: temperature 0.1, seed 42 – 5 runs – name “Compliance Variance – Low – Run 1” through “Run 5” Max tokens: 500 | All other parameters: leave blank
Paste back: overall compliance score for each of the 10 runs
Results: Condition 1 (temp 0.7, no seed): Run 1: | Run 2: | Run 3: | Run 4: | Run 5: Condition 2 (temp 0.1, seed 42): Run 1: | Run 2: | Run 3: | Run 4: | Run 5:
Low temperature plus a fixed seed produces near-identical outputs on repeated runs. The score is reproducible. That reproducibility is what makes a compliance result defensible – the same inputs produce the same outputs, and the score reflects a stable measurement of your pipeline’s compliance posture, not a draw from a high-variance distribution.
What’s next
Scoring Guide
How LLM Prover calculates scores, what affects them, and how to use the diagnostic judge to find out why.
Prompting Guide
How to write prompts that produce reliable, comparable results across models.
Benchmark Guide
Save a run configuration, schedule it on a cadence, and track quality and cost drift over time.
Compliance Store Guide
Upload regulations or policies as a compliance store and score model outputs against specific clauses.