Presets Guide

A preset is a named, reusable run configuration you invoke by name. You build it once and run it from the dashboard, the API, or an agent over MCP. The content stays separate. You supply the prompt or the text to check each time you run, and everything around it comes from the preset. That separation is the point: the same configuration, applied to new content, every time.

This guide covers what a preset holds, the two ways to create one, how to invoke it across all three surfaces, and how a preset becomes a standard your whole team can run against anything.

What a preset captures

A preset saves the configuration, not the content. Everything needed to run is stored on the preset. The prompt or source text is not. It is injected at run time, so the same preset can run against a release note today and a support reply tomorrow and stay a fair comparison.

A preset holds:

• Run type. Performance, quality, or compliance. This is the one axis that determines what gets measured.
• Models. The exact set the run executes against.
• Scoring setup. For a quality preset, the rubric or gold standard. For a compliance preset, the rubric plus the compliance store the judge scores against.
• Context store. Optional. The documents the models read from when answering, separate from any compliance store.
• System prompt. Optional. A saved system prompt applied to every model in the run.
• Inference parameters. Temperature, max tokens, and the Pro+ parameters, when they were part of the run the preset came from.

Every account also starts with one preset already in place. Quality spread is a curated preset: a performance run across four low-cost models (GPT-4.1 Nano, GPT-4o Mini, Claude Haiku 4.5, and DeepSeek V4 Flash) that works on every tier. It carries a curated badge, is read-only, and does not count against your preset limit. It is the fastest way to see what a preset does: give it a prompt, run it, and read cost and latency across four models in one pass.

The Add Preset drawer saves the config; the prompt field lives on the run form, not here

Run types and what each one needs

A preset carries exactly one run type, the same three the dashboard offers for any run.

• Performance. Cost and latency only, no scoring. Available on any tier that can run comparisons, which is Starter and above.
• Quality. Rubric or gold-standard scoring, with per-criterion reasoning. Requires Pro.
• Compliance. Rubric scoring against a compliance store, each criterion judged against a retrieved clause. Requires Pro+.

Two kinds of store can attach to a preset, and they do different jobs:

• Context store. What the models read from when they answer. It applies to any run type and shapes the response.
• Compliance store. What the judge scores against. It applies only to a compliance preset and never feeds the model.

Keeping the two straight is the difference between changing the answer and changing the standard the answer is measured by.

Two ways to create a preset

You can build a preset from scratch, or save one from a run you just liked.

Add a preset directly. In the Presets view, + Add Preset opens a drawer with the fields above: Name, Use for (the description that helps you and an agent match the preset to a task later), Type, Models, the scoring section for the type you picked, and an optional system prompt. The Type is locked once a preset exists, so pick it deliberately. This is the right path when you know the configuration you want.

Save a run you already ran. After an ad-hoc comparison or evaluation that you configured yourself, a Save as preset banner appears. You name it, tick which parts to keep (Models, Inference params, System prompt, RAG store, Rubric, or Gold-standard answer), review, and save. This is the path when a one-off run turned out to be exactly the setup you want to repeat.

Invoking a preset by name

A preset is worth saving because of how little you do to run it afterward. The configuration is already decided, so invoking it is a matter of naming it and handing over the content.

• Dashboard. On the Compare and Evaluate forms, a Start from a preset (optional; prefills the form, you can still tweak) dropdown fills in the models and scoring for you, and you can still change anything before you run. Compare offers performance presets; Evaluate offers all three run types. The field hides itself when you have no presets for that surface. From the Presets view, Run this preset takes you straight there.
• API. Reference the preset by id and pass the content. The run executes with the stored config.
• MCP. An agent calls list_presets to find the one whose Use for matches the task, then run_preset with the preset id and the content. The agent supplies only the text; the config comes from the preset.

The same four-model config, applied to a release note: cheapest and fastest fall to different models

The results above come from the curated Quality spread preset run against a release note. Four models answered the same prompt; GPT-4.1 Nano came back cheapest and Claude Haiku 4.5 fastest. That split is the everyday value of a performance preset: the same question, priced and timed across a representative field, without rebuilding the run each time.

A preset stays valid as models and your plan change

A saved configuration can go stale. A model is retired, or your plan changes and a model moves out of reach. A preset checks itself against the live model registry and your current tier every time it is saved, run, or inspected, so a stale preset tells you before it wastes a run.

• A preset referencing a model that no longer resolves shows a red needs attention badge, with the specific issue listed so you know what to swap.
• After a downgrade, a preset that is now over your limit or above your tier shows a frozen badge and refuses to run until you edit it or restore the plan.

The checks mean a preset you saved three months ago either still runs exactly as it did, or tells you precisely what changed. You never discover a dead model halfway through a run.

From a one-off run to a daily benchmark

A preset is a snapshot of how you want to measure something. A benchmark turns that snapshot into a trend. If a prompt matters enough to standardise, it usually matters enough to watch, and seeding a benchmark from a preset means the schedule runs the exact configuration you already approved.

From the API or an agent, create_benchmark_from_preset takes the preset, a prompt, and an optional schedule. What it builds depends on the preset’s run type:

• A scored preset (quality or compliance) seeds a benchmark that tracks score drift over time.
• An unscored preset (performance) seeds a benchmark that tracks cost, latency, and reliability.

The preset supplies the models and scoring, so there is no second configuration to get right and no risk of the scheduled run measuring something different from the one you tested.

The same preset, run on a schedule: the Day 11 dip is the whole reason to watch it

The daily run held near the top of its range until day 11, where the score dropped below the line and recovered two runs later. A one-off run would have missed it entirely. The preset made the standard; the benchmark made it continuous.

See the Benchmark Guide for schedules, thresholds, and alerts.

A preset as a shared standard

The real power shows up when the preset is a quality or compliance standard your team agreed on. A rubric that encodes your brand voice, or a compliance store that holds your communications rules, saved as a preset, becomes a named standard anyone can run against anything: “marketing-standard”, “legal-review”, “compliance-v2”.

This is where the Score surface comes in. Running a preset with as: "score" judges text you already have against the preset’s rubric or store, with no model generating a response. The content you pass is the artifact to check, not a prompt. It works for quality and compliance presets, since a performance preset has nothing to score against. “Check this against our marketing-standard preset” stops being a meeting and becomes one line, in the tool you are already in.

One message in the channel, scored against the standard the team already saved, with the exact phrase at fault and the clause it breaks. No dashboard, no form, no handoff. The preset carried the standard; the Score surface applied it to the text that was already written.

See the Scores Guide for the full Score surface, and the Compliance Guide for building the store a compliance standard scores against.

What’s next