Inference Parameters Guide

What each parameter controls

Leaving any field blank sends no value to the provider, which then applies its own default. Provider defaults vary, so if you need reproducible results, set values explicitly rather than relying on placeholders.

Temperature

Temperature controls how much randomness the model introduces when selecting each output token. At high values the model explores a wider range of possible continuations. At low values it converges on the most probable one.

We ran GPT-4.1 Nano on a support ticket summarisation task: five runs at temperature 0.7 with no seed, then five runs at temperature 0.1 with seed 42. The rubric required three exact bullets: what the customer wants, the cause (“not stated” if absent), the next step (“not specified” if absent). The Accuracy criterion required no inference beyond what the ticket explicitly says.

The lower-temperature run scored worse. At temperature 0.1 with seed 42, Nano locked into the same output on 4 of 5 runs:

The first bullet infers “to resume running,” a reasonable reading, but one the rubric penalises. It also drops the refund option the customer explicitly stated. At temperature 0.7, output variation meant Nano usually landed on a more literal restatement:

The diagnostic named the cause on every failing run: the prompt does not constrain the first bullet’s content, so the model fills the gap with an inference. Low temperature locked that inference in. High temperature introduced enough variation that Nano occasionally produced a passing restatement.

When to use high temperature: Creative tasks, diversity testing, or when you want to see the range of outputs a model can produce. Treat the spread across multiple runs as the result, not any single run.

When to use low temperature: Summarisation, classification, extraction, compliance scoring. Any task where consistent, comparable results matter. Run a few passes first to confirm the model’s default behaviour on your task is what you want locked in. For compliance benchmarks specifically, parameter discipline is what makes a score auditable. See the Compliance Store Guide for the full workflow.

Seed

A seed fixes the random state the model uses to generate output. Combined with low temperature, it produces near-identical outputs on repeated runs of the same prompt.

Pick any integer and use it consistently across all runs in a benchmark. The specific value does not matter. Consistency does. If you change the seed between runs, you are running a different experiment.

Tier gate: Seed is a Pro feature. On Starter, the field is visible but disabled.

Provider support: Not all providers support seed. When set for a provider that does not support it, the value is ignored.

Max tokens

Max tokens caps the length of the model’s output. Every provider charges per output token, so the cap directly controls cost. It also affects scores: a response cut mid-sentence fails completeness criteria not because the model got it wrong, but because you stopped it before it finished.

When to set it deliberately:

• Cost control: if your task has a predictable output length, cap it. A three-bullet summarisation task doesn’t need 1000 tokens of headroom. Set it to roughly 2x your expected output length.
• Enforcing conciseness: a tight cap forces the model to be brief. If your rubric rewards brevity, the cap is part of the spec.
• Detecting verbose models: a model that uses far more tokens than the task requires is a signal. A cap surfaces that.
• Matching production: if your production system caps output at 200 tokens, your benchmark should too. A benchmark with no cap doesn’t predict production behaviour.

Using the same ticket and rubric as the temperature section, we ran GPT-4.1 Nano at max tokens 100 (sufficient headroom) and max tokens 10 (forced truncation), both at temperature 0.7 with no seed.

At max tokens 10, every run scored zero. The response was cut mid-sentence before the model could produce a single complete bullet:

The diagnostic named the cause immediately:

No rubric adjustment, no prompt change. The fix is a single parameter. This is the diagnostic layer doing its job: it reads the parameter settings alongside the output and tells you which layer caused the failure.

Reasoning models: Reasoning models (Claude, o-series) consume tokens on internal chain-of-thought before producing visible output. If max tokens is set too low, the response pane shows: “Reasoning model consumed full token budget – no visible response. Increase max_tokens.” Set a higher cap than you would for a standard model on the same task.

Leaving the field blank uses the provider’s default, which varies.

Top-p, frequency penalty, presence penalty

These are secondary levers. Most benchmarks do not need to touch them.

Top-p (placeholder 1.0) controls nucleus sampling. The model only considers tokens whose cumulative probability reaches the top-p threshold. At 1.0 all tokens are considered. Lowering it narrows the model’s vocabulary. Leave it at 1.0 unless you have a specific reason to constrain token selection. Changing it without understanding the effect on your task is more likely to introduce unexpected behaviour than improve results.

Frequency penalty (placeholder 0.0) discourages the model from repeating tokens that have already appeared in the output. Useful for long-form generation tasks where repetition is a quality problem. For short structured outputs it has no meaningful effect.

Presence penalty (placeholder 0.0) discourages the model from returning to topics already covered. Similar use case to frequency penalty.

Tier gate: All three are Pro features. On Starter, the fields are visible but disabled.

Prompt caching

When enabled, repeated prompt prefixes are cached at the provider level. If your benchmark uses a fixed system prompt across many prompts, caching means the provider does not re-process that prefix on every call. This reduces cost and latency.

Provider behaviour varies:

• OpenAI and Grok: Prompt caching is enabled automatically by the provider regardless of this setting. The toggle has no effect on these providers.
• Anthropic: Caching is opt-in. The toggle controls whether caching is requested for Anthropic calls. Enable it when running benchmarks with a long fixed system prompt against Claude models.
• Other providers: Caching support varies. Check provider documentation.

The cost shown in results is always the actual provider-reported cost per call. There is no before/after comparison surface in the dashboard.

Why this matters for benchmarks

Parameters must be set before a benchmark starts and kept consistent across every run. Changing them mid-benchmark breaks comparability.

The practical rule: decide your parameter profile before you run anything. Apply it to every run in the benchmark. If you need to test a different profile, create a new benchmark.

If your production system runs at temperature 0.3 with a 500-token cap, your benchmark should use the same settings. A benchmark run at temperature 0.7 with no cap does not predict production behaviour for a capped system.

The compliance case for low temperature and seed

For compliance benchmarks, parameter discipline is not optional. A compliance score produced at default temperature is not a compliance score. It is a sample. Run it again and you may get a different number. You cannot show a sample to an auditor.

The temperature section above shows exactly how this plays out: low temperature locks in the model’s default behaviour, high temperature introduces variation. For a summarisation task, that variation occasionally helps. For a compliance task, you need the opposite. You need to know that the score you got is the score you will get next time. Low temperature plus a fixed seed is what makes that true.

What’s next