Skip to content

Onboarding

Pro

Onboarding: Set Up Drift Detection

Onboarding arc for the drift goal. Create a scored preset, stand up a scheduled benchmark from it, run a baseline, and leave you monitoring for regressions.

⚠

Agentic and programmatic use. Agents and scripts you connect can make cost-generating calls on your behalf, and AI agents behave non-deterministically. You are responsible for monitoring and bounding your own automation. See Terms for details.

Run it: paste into your MCP agent

Highlighted parts are placeholders. Replace them with your own values before (or after) copying.

You are completing the user's onboarding for the goal "detect model drift". Set up the whole
arc in one guided pass: a scored preset, a scheduled benchmark seeded from it, and a baseline
run. Then hand off to the ongoing drift monitor. Reuse the preset and benchmark primitives;
do not hand-build a benchmark config when a preset can seed it.

1. Confirm the tools you need are available (list the tools): list_models, list_presets,
   create_preset, create_benchmark_from_preset, trigger_benchmark_run, list_benchmark_runs,
   poll_job, and the agent-state tools (get_agent_notes, put_agent_note, update_agent_note).
   If the tool list does not load, STOP and tell me the connection is not working.

2. Read your notes for this recipe: get_agent_notes(recipe_id="recipe_id"). If an active
   note already records a drift setup, tell me it is already set up and hand off to
   mcp-drift-monitor rather than creating a duplicate benchmark. Otherwise continue.

3. Settle the config with me:
   - The prompt and the model(s) to monitor. List the models available to my account first
     and respect my plan's per-run cap; do not exceed it.
   - Scoring is REQUIRED here: drift in quality cannot be detected on an unscored run. Ask
     me for a gold-standard answer (expected_output) or a saved rubric (rubric_id). If I have
     neither, offer to help me build a rubric first (mcp-build-a-rubric) before continuing.

4. Save the config as a reusable preset so the monitor and I can refer to it by name:
   create_preset(name=<something like "drift-watch">, run_type="quality",
     models=<the models>, and the scoring I chose). Show me the exact config before saving;
   do not save silently. (If I already have a fitting preset from list_presets, reuse it and
   skip creating one.)

5. Seed a SCHEDULED benchmark from the preset rather than building one from scratch:
   create_benchmark_from_preset(preset_id=<the preset>, content=<the prompt>,
     schedule=<daily / 4x_daily / hourly>). Tell me which schedules my plan allows; the
   server enforces the cap. Note the suite_id it returns.

6. Record the standing note as the series anchor (this note stays RUNNING for the life of the
   series, unlike a one-shot):
   put_agent_note(recipe_id="recipe_id", recipe="onboard-drift",
     intent="Drift monitor for <model(s)> on <short prompt label>", status="running",
     data={ suite_id: "<the suite_id>" }).

7. Trigger a first run now to establish the baseline: trigger_benchmark_run(suite_id). It
   returns a JOB id; poll poll_job every 5 seconds to completion (stop after 5 minutes). The
   job id and run id are DIFFERENT. Once complete, call list_benchmark_runs(suite_id) and
   take the newest run's run_id as the BASELINE.

8. Update the note so a later session can diff against the original baseline:
   update_agent_note setting job_id=<latest job id> and
   data={ suite_id, baseline_run_id: "<baseline run_id>", last_run_id: "<same baseline run_id>" },
   keep status "running".

9. Report the baseline numbers per model and tell me plainly: the benchmark is live on
   <schedule>, and from now on the drift monitor reconciles each run against this baseline.
   Hand off: tell me to run mcp-drift-monitor (or just ask "any drift?") in future sessions.
   Leave this note "running".

ON ANY FAILURE: tell me honestly at which step and with what error. Do NOT fabricate a
baseline or claim the benchmark is live when a call failed. Mark the note "failed" with
data.failure_stage, data.failure_reason, and the job_id if a run was involved; never leave a
note "running" after a genuine failure.

Goal

Walk out of onboarding already monitoring. The agent builds the whole drift setup in one pass: a reusable scored preset, a scheduled benchmark seeded from it, and a baseline run, so the only thing left is to hear about regressions when they happen.

When to use

When you arrived to detect model drift and want the setup done for you end to end, rather than assembling a preset, a schedule, and a baseline by hand.

How it works

  • Scored or nothing. Drift in quality is invisible on an unscored run, so the recipe insists on a gold-standard answer or a rubric before it builds anything.
  • Preset seeds the benchmark. It saves the config once as a preset, then seeds the scheduled benchmark from that preset rather than hand-building a config, so the monitor and future runs refer to the same named setup.
  • One standing note. The baseline run id is persisted in a note that stays running for the life of the series, which is what lets any later session diff the newest run against the original baseline. Ongoing reconciliation is the drift monitor’s job (mcp-drift-monitor).

Verification

  • A scored preset exists, saved only after you approved the config.
  • A scheduled benchmark exists and has run at least once (the baseline).
  • A standing note exists with suite_id and baseline_run_id in its data, status running.
  • The hand-off to mcp-drift-monitor is clear: a later “any drift?” finds the note and reports against the baseline without you supplying ids.

Where this leads

Natural next recipes once you are comfortable with this one.

Monitor a Model for Drift Over Time Run it over time

Benchmark is live. Use the drift monitor each session to reconcile the newest run against the baseline and hear only about real movement.

Build a Scoring Rubric From a Description Go deeper

Score drift on exactly what you care about. Turn a plain description of 'good' into a reusable rubric.