Onboarding: Evaluate Your RAG Pipeline

Onboarding arc for the rag goal. The shortest path to one scored evaluation grounded in your own knowledge base, with the pipeline diagnostic naming the layer behind any low score, then hands off to the RAG recipes so you can go deeper.

⚠

Agentic and programmatic use. Agents and scripts you connect can make cost-generating calls on your behalf, and AI agents behave non-deterministically. You are responsible for monitoring and bounding your own automation. See Terms for details.

Run it: paste into your MCP agent

Parts in double braces (like {{recipe_id}}) are placeholders. Replace them with your own values before (or after) copying.

You are completing the user’s onboarding for the goal “evaluate your RAG pipeline”. This is the SHORTEST path to one aha: a single scored evaluation grounded in the user’s OWN documents, where the pipeline diagnostic names the LAYER behind any low score (the context/store, or the model). The diagnostic is the point of this goal, not the raw score. Do NOT build a benchmark, a schedule, or a rubric chain. One store, one scored evaluation, then read back the finding.

run_evaluation and poll_job are core tools. create_rag_store is EXTENDED (not in your core list): if you cannot see it, discover it via list_tool_catalog, then request_tools, then invoke. RAG and diagnostics are Pro+; respect tier gates, as the server enforces them – read any error and relay it rather than retrying blindly.

CONFIRM TOOLS. List the tools; confirm you can reach list_rag_stores, list_models, run_evaluation, and poll_job, and can load create_rag_store if it is not already present. If the tool list will not load at all, STOP and tell me the MCP connection is not working.
SETTLE SCOPE WITH ME (one short exchange, before running anything).
- The document(s) to ground on: the knowledge base the models should read. This is the one thing that must exist, since the whole goal is evaluating on YOUR content, not generic prompts.
- A question whose answer lives in those documents (a question answerable from general knowledge is a poor test – the point is to see retrieval pull your content).
- The correct answer (a gold standard / expected_output), so every response is scored. A gold standard is the fastest start and is all we need here; do not build a rubric.
- The candidate models: list what my tier allows (list_models), respect my per-run cap, and propose a small spread so the comparison is meaningful.
STAND UP THE CONTEXT STORE.
- If I already have a suitable general store (list_rag_stores, store_type general, sync_status ready), reuse it – do not create a duplicate.
- Otherwise create_rag_store(store_type=“general”, name=…), then walk me through uploading the document(s) (upload-url -> confirm) and poll until sync_status is ready. A store that is not ready cannot be attached to an evaluation, so do not proceed until it is.
- Sanity-check chunk_count against file_count: a multi-page document that produced only one or two chunks ingested badly and will retrieve poorly even when status says ready. Flag it to me before spending on the evaluation if the chunk count looks wrong for the material.
RUN ONE SCORED EVALUATION, GROUNDED. run_evaluation with: the grounded question as the prompt, the candidate models, store_id set to the store from step 3 (this is what attaches your knowledge base), and expected_output set to the gold standard (so every row is scored). This returns a JOB id; poll poll_job every 5 seconds to completion (stop after 5 minutes). The job id and the evaluation id are DIFFERENT.
READ BACK THE DIAGNOSTIC – THIS IS THE AHA. For each model, report the quality score AND its breakdown.pipeline_diagnostics finding(s). The finding’s “source” names the layer:
- source “context” means the STORE is the ceiling – the retrieved documents are missing the answer, are stale, or contradict the gold standard. The model answered faithfully from what it was given; the fix is the CONTENT, not the model. Quote the finding’s evidence and suggestion.
- source “model” (or prompt / parameters / system_prompt) means the fault is elsewhere in the pipeline, named explicitly.
State plainly, per model, whether a low score is a store problem or a model problem. If the scores are low but every finding points at “context”, say directly: the store is the bottleneck, not the models – fix the source and rerun. This is the whole reason you used a RAG evaluation instead of reading a bare number.
HAND OFF. Tell me this was the first scored, grounded run and point me onward: run mcp-verify-retrieval to prove the store retrieves on more questions, or mcp-rag-upload to add documents or rebuild with better chunking if the diagnostic blamed the context layer. I can also pick up another onboarding goal (detect drift, find the cheapest model, benchmark my own endpoint) from the Getting started entry in the sidebar.

ON ANY FAILURE: tell me honestly where it failed and with what error (a tier gate is a clear, expected error – relay it, do not retry blindly). Do NOT claim the store is ready when it is still syncing, do NOT run the paid evaluation against a syncing or dirty store, and do NOT invent a diagnostic finding – report only what breakdown.pipeline_diagnostics actually returns.

You are completing the user's onboarding for the goal "evaluate your RAG pipeline". This is
the SHORTEST path to one aha: a single scored evaluation grounded in the user's OWN documents,
where the pipeline diagnostic names the LAYER behind any low score (the context/store, or the
model). The diagnostic is the point of this goal, not the raw score. Do NOT build a benchmark,
a schedule, or a rubric chain. One store, one scored evaluation, then read back the finding.

run_evaluation and poll_job are core tools. create_rag_store is EXTENDED (not in your core
list): if you cannot see it, discover it via list_tool_catalog, then request_tools, then
invoke. RAG and diagnostics are Pro+; respect tier gates, as the server enforces them -- read
any error and relay it rather than retrying blindly.

1. CONFIRM TOOLS. List the tools; confirm you can reach list_rag_stores, list_models,
   run_evaluation, and poll_job, and can load create_rag_store if it is not already present.
   If the tool list will not load at all, STOP and tell me the MCP connection is not working.

2. SETTLE SCOPE WITH ME (one short exchange, before running anything).

   - The document(s) to ground on: the knowledge base the models should read. This is the one
     thing that must exist, since the whole goal is evaluating on YOUR content, not generic
     prompts.
   - A question whose answer lives in those documents (a question answerable from general
     knowledge is a poor test -- the point is to see retrieval pull your content).
   - The correct answer (a gold standard / expected_output), so every response is scored. A
     gold standard is the fastest start and is all we need here; do not build a rubric.
   - The candidate models: list what my tier allows (list_models), respect my per-run cap, and
     propose a small spread so the comparison is meaningful.

3. STAND UP THE CONTEXT STORE.

   - If I already have a suitable general store (list_rag_stores, store_type general,
     sync_status ready), reuse it -- do not create a duplicate.
   - Otherwise create_rag_store(store_type="general", name=...), then walk me through uploading
     the document(s) (upload-url -> confirm) and poll until sync_status is ready. A store that
     is not ready cannot be attached to an evaluation, so do not proceed until it is.
   - Sanity-check chunk_count against file_count: a multi-page document that produced only one
     or two chunks ingested badly and will retrieve poorly even when status says ready. Flag it
     to me before spending on the evaluation if the chunk count looks wrong for the material.

4. RUN ONE SCORED EVALUATION, GROUNDED. run_evaluation with: the grounded question as the
   prompt, the candidate models, store_id set to the store from step 3 (this is what attaches
   your knowledge base), and expected_output set to the gold standard (so every row is scored).
   This returns a JOB id; poll poll_job every 5 seconds to completion (stop after 5 minutes).
   The job id and the evaluation id are DIFFERENT.

5. READ BACK THE DIAGNOSTIC -- THIS IS THE AHA. For each model, report the quality score AND
   its breakdown.pipeline_diagnostics finding(s). The finding's "source" names the layer:

     - source "context" means the STORE is the ceiling -- the retrieved documents are missing
       the answer, are stale, or contradict the gold standard. The model answered faithfully
       from what it was given; the fix is the CONTENT, not the model. Quote the finding's
       evidence and suggestion.
     - source "model" (or prompt / parameters / system_prompt) means the fault is elsewhere in
       the pipeline, named explicitly.

   State plainly, per model, whether a low score is a store problem or a model problem. If the
   scores are low but every finding points at "context", say directly: the store is the
   bottleneck, not the models -- fix the source and rerun. This is the whole reason you used a
   RAG evaluation instead of reading a bare number.

6. HAND OFF. Tell me this was the first scored, grounded run and point me onward: run
   mcp-verify-retrieval to prove the store retrieves on more questions, or mcp-rag-upload to
   add documents or rebuild with better chunking if the diagnostic blamed the context layer. I
   can also pick up another onboarding goal (detect drift, find the cheapest model, benchmark
   my own endpoint) from the Getting started entry in the sidebar.

ON ANY FAILURE: tell me honestly where it failed and with what error (a tier gate is a clear,
expected error -- relay it, do not retry blindly). Do NOT claim the store is ready when it is
still syncing, do NOT run the paid evaluation against a syncing or dirty store, and do NOT
invent a diagnostic finding -- report only what breakdown.pipeline_diagnostics actually returns.

Goal

Walk out of onboarding with the one thing the rag goal promises: a scored evaluation grounded in your own knowledge base, where the diagnostic tells you WHY a score is low. A number alone says an answer was wrong. The diagnostic names the layer behind it, so when the store is the problem you fix the content instead of trading models.

When to use

When you arrived to evaluate your RAG pipeline on your own documents and want your agent to stand up the store and run the single scored evaluation that shows both the score and its cause, rather than building the store and reading a bare number yourself.

How it works

Shortest path, not a chain. One general store plus one scored evaluation. No benchmark, no schedule, no rubric build. The store is the one irreducible prerequisite, since the goal is evaluating on your content.
Grounded and scored. The store is attached so the models read your documents, and a gold-standard answer is supplied so every response is scored. A grounded, scored run is what makes the diagnostic meaningful.
The diagnostic is the aha. Every low score carries a pipeline diagnostic naming the layer behind it. A low score whose finding is source: context is a store defect (stale, missing, or contradictory content), not a model defect – the model reported your store faithfully. That distinction is the entire point of a RAG evaluation over a bare score.
Honest about the store. The agent sanity-checks chunking before spending, never evaluates against a store that is not ready, and reports only the findings the run actually returned.

Verification

A general RAG store exists, loaded with your documents and sync_status: ready, and an existing fit was reused rather than duplicated.
Exactly one scored evaluation was run with the store attached (store_id) and a gold standard (expected_output), across a tier-scoped spread of models.
Each model’s score was reported together with its breakdown.pipeline_diagnostics finding, and any low score was attributed plainly to the layer the finding names (context vs model).
The paid evaluation was never run against a syncing or dirty store.
The hand-off to mcp-verify-retrieval / mcp-rag-upload is clear.

Where this leads

Natural next recipes once you are comfortable with this one.

Prove a RAG Store Actually Retrieves Go deeper

Prove the store retrieves before you trust a score. Smoke-test it with a grounded question and confirm the answer is really coming from your documents.

Load a File Into a RAG Store, End to End Do more at once

Add more documents or rebuild the store with better chunking when the diagnostic points at the context layer.