Check a Response Is Grounded, Then Fix What Isn't

Using the LLM Prover MCP tools, CHECK whether a response is grounded in the source it is supposed to be based on, then, only if I ask and only within a budget I set, FIX the claims you honestly can. This is the Score run type: no model is run, the judge scores exactly the text supplied, and the SOURCE goes in context (not a compliance store – this is faithfulness to supplied material, not regulatory compliance). The check is cheap and always runs first; the fix is a budgeted, opt-in loop. Keep any agent-state note “running” until the loop ends.

Set up:

1. Confirm the tools you need are available (list the tools): score, poll_job, list_rubrics, create_rubric, and (for the fix loop) the agent-state tools get_agent_notes, put_agent_note, update_agent_note. score is a core tool; several others are EXTENDED and may not be in your visible list – if any is missing, load it via list_tool_catalog then request_tools and call it through invoke(…); do not conclude it is unavailable.
2. Get the TWO inputs from me, and do not paraphrase either:
   • The RESPONSE under test (the exact text whose grounding you are checking).
   • The grounding SOURCE. Either inline text I give you (use it directly as context), OR a general RAG store by store_id. If it is a store: it MUST be sync_status “ready” before you trust it, and you should sanity-check that it actually retrieves (see the “Prove a RAG Store Actually Retrieves” recipe) – a store that looks ready but chunked badly will produce a false grounding verdict. If the store is “syncing” tell me to wait; if “dirty” stop and tell me to fix it. Do NOT silently substitute a different source.
3. FIND OR CREATE the faithfulness rubric. This is NOT a general quality rubric – a fluent, well-written answer that invents facts must score LOW. Call list_rubrics and look for a rubric already built for grounding/faithfulness. If a suitable one exists, reuse it and name it to me. If none exists, create one with create_rubric (rubric_type “quality”) from EXACTLY these criteria, and tell me I can edit it afterwards:
   • “Claim support” (weight 50): every FACTUAL claim in the response is directly traceable to the provided context. A factual claim with no support in the context scores low, no matter how plausible or well written. Exclude non-factual sentences (opinions, predictions, normative or subjective statements) from this judgment: they are not groundable claims and must not be penalised as unsupported. Judge at the level of atomic claims, not whole sentences: a sentence making several claims is scored on each claim, not as one unit.
   • “No contradiction” (weight 30): nothing in the response conflicts with the context. A statement the source contradicts is the most serious failure. Treat a scope or time mismatch as a contradiction when the response states something as current or general that the source gives only for a past date, a specific region, or a narrower scope.
   • “No fabricated specifics” (weight 15): names, numbers, dates, citations, and quotes appear in the context. Invented specifics score low even when the surrounding claim is supported. This includes invented PRECISION on a supported topic: if the source says “lasts a long time” and the response says “18 hours”, the added precision is fabricated even though the topic is grounded.
   • “Scope discipline” (weight 5): the response does not present outside knowledge as if it came from the source. Using general knowledge, or stating an opinion, is not always wrong; presenting either as grounded in the source is.
4. Agree the TARGET with me: either “every claim supported” (strict, recommended for faithfulness) or a faithfulness SCORE to clear (0-100). If I want the fix loop, also agree the BUDGET now (never run unbounded): POOL SIZE N variants per round, a hard ROUND ceiling, and a hard COST ceiling in dollars. If I do not give them, propose sensible defaults (e.g. N=4, 4 rounds, a small cost cap) and state them explicitly. Cost scales with N x rounds; every variant is a judged score.

Check (always runs first):

5. SCORE THE RESPONSE AS SUPPLIED: score(artifact=, context=<the source text, or the store_id>, rubric_id=). Do NOT pass models, providers, params, or any benchmark setting; Score runs no model and rejects them. Async: it returns a JOB id; poll_job every 5 seconds until complete (stop after 5 minutes), then read the result.

6. CLASSIFY THE RESULT per claim. FIRST DECOMPOSE: break each sentence into its atomic claims and classify each one separately – real sentences mix a supported claim with an unsupported one (“Acme, founded in 2010 [supported], is the market leader [not in source]”), and labelling the whole sentence mislabels half of it. Read each criterion’s judge_reasoning and the per-criterion scores, then assign EACH atomic claim ONE label. This classification is the point of the recipe, so be specific, do not just say “unsupported”. The labels, grouped by what can be done about them:

   Pass:

   • SUPPORTED: the claim is directly traceable to the source. This is the numerator of the grounding score – report “N of M claims supported”, not just a list of problems.

   Fixable by rewriting against the source:

   • CONTRADICTED: the source says something that conflicts with the claim. (Most serious. The response is wrong against the source. Fix: correct it to match the source.)
   • SCOPE/PRECISION MISMATCH: the source supports the topic but not at the claim’s strength or granularity – either the claim overreaches (source is thinner than the claim) or it invents precision (source says “a long time”, claim says “18 hours”). (Fix: pull the claim back to the source’s actual granularity.)
   • STALE / SCOPE-MISMATCHED: the source supports the claim but for a different time, region, or scope, and the response states it as current or general (“500 employees (2021)” stated as today; an EU-only figure stated globally). (Fix: restore the source’s time/region/scope qualifier.)

   Not fixable by rewriting (a source problem, report it – never fabricate grounding):

   • NOT-IN-SOURCE: the source is simply silent on the claim. (Nothing to ground it.)
   • AMBIGUOUS: the source is genuinely unclear, or says conflicting things in different places, so the claim cannot be confirmed either way. (The problem is the source.)

   Not a grounding failure (do NOT penalise or try to fix):

   • UNVERIFIABLE BY NATURE: the sentence is an opinion, prediction, or normative statement (“this is the best approach”), not a factual claim. It is categorically ungroundable, so it is neither supported nor a hallucination. Flag it as exempt so it is not scored as unsupported.

   KNOWN LIMITATION – do not invent a label for it: when the source is a multi-document store, a claim can be true in the corpus but attributed to the WRONG document, or stitch together facts from two documents in a way neither supports alone. The Score judge sees the retrieved context as one blob, not per-document provenance, so it CANNOT reliably detect misattribution here. Do not classify it as grounded on the strength of the facts being present somewhere; if provenance matters to me, tell me this check does not verify it and I need a provenance-aware review.

7. VERDICT GATE: if the response already meets the target (every claim supported, or the score clears the bar), REPORT the clean per-claim audit and STOP. Do not spend on a fix that is not needed.

Fix (only if the check failed AND I asked for the fix, and only within budget):

8. Record loop state in a note BEFORE spending, with the STRUCTURED STOP CONTRACT. Only CONTRADICTED and UNDERSPECIFIED claims are honestly fixable by rewriting (correct the contradiction to match the source; weaken or qualify the claim to the scope the source supports). NOT-IN-SOURCE and AMBIGUOUS claims CANNOT be honestly repaired by rewriting – fabricating grounding for them is the exact failure this recipe exists to catch. Do not “fix” them by inventing support; carry them to the report as unfixable with their reason. put_agent_note CREATES the note and returns a note_id – CAPTURE it; every later change uses update_agent_note(note_id, recipe_id, …), never a second put_agent_note (that would duplicate the note).

   put_agent_note(recipe_id="{{recipe_id}}", recipe="mcp-ground-check-and-fix",
     status="running", intent="Ground-check and fix <what> against <source>",
     data={ source:{ inline:true|false, store_id? }, rubric_id:<faithfulness rubric>,
            pass_rule:"all_supported"|"target", pass_target:<score if target>,
            pool_size:N, round:0, cost_spent:0,
            best:{ text:<the original response>, score:<check score>, passed:false, job_id:<check job> },
            claims:[{ text, reason:"supported|contradicted|scope_precision_mismatch|stale_scope|not_in_source|ambiguous|unverifiable", fixable:true|false }],
            history:[],
            terminate:{ rounds:<round ceiling>, cost_usd:<cost ceiling> },
            outcome:{ pass_rule:"all_supported"|"target", target:<score if target> } }).
   

   terminate = hard brakes (ran out of budget); outcome = success (grounded). The terminate ceilings are mandatory for this loop; never write a note without them. Loop (each round runs steps 9 through 14):

9. CHECK THE STOP CONTRACT first, against the note’s counters. OUTCOME before TERMINATE; first to trip wins. If best-so-far meets the target -> STOP as SUCCESS. If round >= terminate.rounds OR cost_spent >= terminate.cost_usd -> STOP as CEILING. Never silently continue past a ceiling.

10. BUILD THE POOL of N variants. Carry the current best forward UNCHANGED as one member (elitism). Fill the other N-1 by rewriting ONLY the FIXABLE spans against the source: correct CONTRADICTED claims to match the source; pull SCOPE/PRECISION MISMATCH claims back to the source’s granularity; restore the time/region/scope qualifier on STALE / SCOPE-MISMATCHED claims. LEAVE not-in-source and ambiguous claims as a decision for me (drop them or keep them flagged – do not invent support), leave UNVERIFIABLE sentences untouched (they are not failures), and keep SUPPORTED claims as they are. Be targeted, not a full rewrite; preserve the parts that were already supported.

11. SCORE THE POOL in one call: score(artifacts=[<the N variants, best-so-far included>], context=, rubric_id=). Async; poll to complete.

12. EVALUATE: per the pass_rule, pick the new best-so-far (a passing variant if any, else the highest-scoring). Elitism: keep the old champion if it still beats every new one.

13. UPDATE the note with update_agent_note(note_id=, recipe_id="{{recipe_id}}", status=“running”, data={…}): round += 1; cost_spent += this round’s score cost; best = new best-so-far; append { round, pool:[{score, passed}], best_score, cost } to history; refresh the claims list with any reasons that changed. READ THE NOTE BACK and confirm it saved before continuing. The note is the source of truth, not working memory.

14. Go back to step 9 (re-check the stop contract, then the next round).

Report (always, whether or not the fix ran):

15. Give me the CLAIM-BY-CLAIM AUDIT – this is the deliverable, not a single number. Lead with the numerator: N of M factual claims supported (exclude unverifiable sentences from M). Then, for each claim, its label (supported / contradicted / scope-precision mismatch / stale-scope / not-in-source / ambiguous / unverifiable), with the source span the judge used (quote it; do not paraphrase the source from memory). Distinguish clearly between the claims you FIXED (contradicted -> corrected, scope/precision -> pulled back to source granularity, stale -> re-qualified) and the claims you COULD NOT honestly fix.
16. For each UNFIXABLE claim, say exactly what would have to change:
    • NOT-IN-SOURCE -> drop the claim, or add the supporting material to the source.
    • AMBIGUOUS -> the source itself needs disambiguating; this is not a response problem. Do NOT present a rewrite that silently deleted a not-in-source or ambiguous claim as a clean pass – call out what was dropped, because dropping a claim may lose something I wanted to keep.
17. If the fix ran: give me the repaired response, the round-by-round best-score trajectory, total cost_spent, and which ceiling or success condition stopped the loop.
18. HUMAN EYEBALL (Goodhart guard): a high faithfulness score is not proof the response is genuinely good or complete. A repair loop can “pass” by qualifying every claim into vagueness or quietly dropping the hard ones – technically grounded, but weaker than what I started with. The judge score is not ground truth. Show me the ACTUAL repaired text and the list of dropped/qualified claims, and recommend I read both before I ship. Treat a pass as “clears the bar, now read it”, not “done”.
19. CLEANUP (if a note was created): update_agent_note(note_id=, recipe_id="{{recipe_id}}", status=“completed”, data={… stop_outcome:“grounded”| “halted_at_ceiling”, stop_detail naming the ceiling if one fired}). Do not leave a loop that hit a ceiling in “running”.

ON ANY FAILURE or clear mismatch (the source will not read or is a not-ready store, a score will not run, the note did not read back correctly, the judge returns nothing): STOP and, if a note exists, update_agent_note(note_id=, recipe_id="{{recipe_id}}", status=“failed”, data={failure_stage, failure_reason, detail, last good job_id}). Never report a response as “grounded” when you could not actually score it against the source, and never fabricate grounding for a claim the source does not support.