Refine a Draft Until It Passes Your Standard

Using the LLM Prover MCP tools, refine a draft in ROUNDS until it passes my standard, or until you hit a budget I set. This is a population-based search: YOU are the creative engine that writes variations; the SCORE run type (judge + my rubric, and for compliance my regulation store) is the objective fitness function. No model is generated here, so you author every variant yourself and Score judges exactly the text you supply. Keep the agent-state note status “running” until the loop ends.

Set up:

1. Confirm the tools you need are available (list the tools): score, poll_job, list_rubrics, list_rag_stores, and the agent-state tools get_agent_notes, put_agent_note, update_agent_note. Several are EXTENDED tools that may not be in your visible list – if any is missing, load it via list_tool_catalog then request_tools and call it through invoke(…); do not conclude it is unavailable.

2. Read your notes: get_agent_notes(recipe_id="{{recipe_id}}"). If an active note exists we are RESUMING: CAPTURE its note_id (every update this session targets that note_id), and read its data (round, cost_spent, best, history) AND its stop contract (terminate, outcome). BEFORE anything else, re-check the stop contract against the current counters (same check as step 6): if a ceiling was already crossed, or a passing variant was already found, do NOT resume, go straight to the stop handling in step 6. Only if nothing is crossed, continue from “LOOP”. If no active note, we are STARTING.

3. Agree the STANDARD and the BUDGET with me.

   • Standard, one of: a saved PRESET that holds the standard (run it as a score surface), OR a quality rubric (list_rubrics, rubric_type “quality”), OR for compliance a rubric_type “compliance” rubric PLUS a compliance store (list_rag_stores(type= “compliance”), which MUST be sync_status “ready”; if “syncing” tell me to wait, if “dirty” stop and tell me to fix it). If no suitable standard exists, offer to help me build it and stop until it exists. The standard is the fitness function; without it this loop is meaningless.
   • Budget (NEVER run unbounded): POOL SIZE N (variants per round), a hard ROUND ceiling, and a hard COST ceiling in dollars. If I do not give them, propose sensible defaults (e.g. N=4, 5 rounds, a small cost cap) and state them explicitly. Remember cost scales with N x rounds: every variant is a judged score.
   • Define PASS precisely from the standard, and tell me which applies:
     • compliance: a variant passes only if EVERY criterion passes (all-clauses; compliance is all-or-nothing per clause).
     • quality: a variant passes if it meets the standard’s threshold (the preset’s / my configured quality_min or judge_min). If I have set no quality threshold, ask me for a target score (0-100) to treat as the bar; do not invent one silently.

4. Record loop state in a note BEFORE spending, including the STRUCTURED STOP CONTRACT so any session reads the loop’s own limits. put_agent_note CREATES the note and returns a note_id – CAPTURE that note_id; every later change to this note uses update_agent_note (note_id, recipe_id, …), NOT another put_agent_note (a second put would create a duplicate note, not update the first). When resuming, take the note_id from the active note get_agent_notes returned.

   put_agent_note(recipe_id="{{recipe_id}}", recipe="mcp-refine-until-it-passes",
     status="running", intent="Refine <what> until it passes <standard>",
     data={ standard:{ rubric_id|preset_id, compliance_store_id? }, pass_rule:"all_criteria"|"target",
            pass_target:<score if quality target>, pool_size:N,
            round:0, cost_spent:0,
            best:{ text:<starting draft or null>, score:null, passed:false, job_id:null },
            history:[],
            terminate:{ rounds:<round ceiling>, cost_usd:<cost ceiling> },
            outcome:{ pass_rule:"all_criteria"|"target", target:<score if quality> } }).
   

   terminate = hard brakes (stop because you ran out of budget); outcome = success (stop because a variant passed). Both OPTIONAL in the schema, but for THIS loop the terminate ceilings are mandatory, so never write a note without them.

Loop (each round):

5. BUILD THE POOL of N variants to score this round:
   • Round 0: the pool is size N. If I gave a starting draft, include it as one variant (so the baseline is measured) and fill the remaining N-1 with your own creative takes. If I gave only a brief, generate all N fresh.
   • Later rounds: the pool stays size N. Carry the current best forward UNCHANGED as ONE pool member (elitism, so a round can never do worse than the champion), and generate N-1 FRESH variants to fill the rest. The pool is therefore {best-so-far} + (N-1 new), never N new plus the champion (that would grow the pool and the cost every round). Be genuinely creative and DIVERSE with the N-1, do not just nudge one line; the rubric+ store is the fitness function, so explore. INFORM the variation by what the last pool lost points on: read each criterion’s judge_reasoning and per-criterion scores from the previous result and target the weak criteria.
6. CHECK THE STOP CONTRACT first, against the note’s counters. Check OUTCOME, then TERMINATE; first to trip wins; do the comparison explicitly.
   1. OUTCOME (success): if the best-so-far already PASSES (per pass_rule), STOP as SUCCESS. Go to REPORT, then at cleanup mark the note “completed” with data.stop_outcome = “passed”.
   2. TERMINATE (ceiling): if round >= terminate.rounds OR cost_spent >= terminate.cost_usd, STOP as CEILING. Go to REPORT, then mark the note “completed” with data.stop_outcome = “halted_at_ceiling” and data.stop_detail naming which ceiling fired (e.g. “rounds: 5/5” or “cost: 2.10/2.00”). NEVER silently continue past a ceiling. A ceiling hit is terminal, not a zombie that keeps spending. If a ceiling is hit and I am present and want to go further, that is a NEW decision: raise terminate.rounds / terminate.cost_usd in the note only on my explicit approval, then continue.
7. SCORE THE POOL in one call: score(artifacts=[<the N variants, best-so-far included>], rubric_id=) and, for compliance, also compliance_store_id=. If the standard is a preset, run the preset as a score surface instead (content = the pool). Do NOT pass models, providers, params, or any benchmark setting; Score runs no model and rejects them. Async: it returns a JOB id; poll_job every 5 seconds until complete (stop after 5 minutes), then read the result.
8. EVALUATE the pool: each artifact has per-criterion scores + judge_reasoning (and, for compliance, judge_criterion_chunks with the actual clause text). Determine, per the pass_rule, which variants PASS. Pick the new best-so-far: a PASSING variant if any (the highest-scoring passer), else the highest-scoring variant overall. Elitism: if the old champion still beats every new variant, keep it.
9. UPDATE the note with update_agent_note(note_id=, recipe_id="{{recipe_id}}", status=“running”, data={…}) – this is the loop’s memory, so do NOT call put_agent_note again (that creates a duplicate). In data: round += 1; cost_spent += this round’s total score cost; best = the new best-so-far { text, score, passed, job_id }; append { round, pool:[{score, passed}], best_score, cost } to history. Then READ THE NOTE BACK (get_agent_notes) and confirm the written state matches; correct it before continuing. Do NOT carry loop state only in your working memory; the note is the source of truth.
10. Go back to step 6 (re-check stop, then next round).

Report (when a stop condition fired):

11. Tell me plainly:
    • If SUCCESS: the passing variant(s), ranked, each with the per-criterion detail that proves it passed (for compliance, cite the clause from judge_criterion_chunks per criterion; distinguish “not triggered” = pass from “no source” = could-not-verify, not a pass). If several passed, present them ranked so I can choose on style.
    • If CEILING: the BEST version found, its score, and EXACTLY which criteria it still misses and why, so I know what is left. Say which ceiling stopped the loop.
    • Always: the round-by-round trajectory (best score per round) and total cost_spent.
12. HUMAN EYEBALL (important, Goodhart guard): a passing score is not proof the copy is genuinely good. A creative optimizer can OVERFIT to the judge, gaming the judge by learning the phrasing a rubric criterion rewards (judge-pleasing wording) rather than truly meeting its intent. The judge score is not ground truth. Show me the ACTUAL winning text, not just the number, and recommend I read it before I ship it. Do NOT present a passing variant as done on score alone.
13. CLEANUP: update_agent_note(note_id=, recipe_id="{{recipe_id}}", status=“completed”, data={… stop_outcome:“passed”|“halted_at_ceiling”, stop_detail for a ceiling}). It then drops from the active list and frees a slot. Do NOT leave a loop that hit a ceiling in “running”.

ON ANY FAILURE or clear mismatch (the standard is missing or not ready, a score will not run, the note write did not read back correctly, the judge returns nothing for every variant): STOP the loop and update_agent_note(note_id=, recipe_id="{{recipe_id}}", status=“failed”, data={failure_stage, failure_reason, detail, the last good job_id}). Never keep spending in a loop whose state you cannot trust, and never report a variant as passing when you could not actually score it against the standard.