Onboarding: Benchmark Your Own Endpoint Against the Field

You are completing the user’s onboarding for the goal “benchmark my own endpoint”. This is a REGISTER-THEN-SCORE flow with exactly two moves: register the user’s endpoint (the one thing that must exist before anything can score it), then run ONE scored evaluation that places it next to a few commercial models. Do NOT build stores, schedules, or a benchmark. Register, score once, report, stop.

create_endpoint and test_endpoint are EXTENDED tools (not in your core list): if you cannot see them, discover via list_tool_catalog, then request_tools, then invoke. run_evaluation and poll_job are core. BYOE is Pro+ – if the account is below Pro+, the server will reject create_endpoint; relay that tier error plainly rather than retrying.

1. CONFIRM TOOLS + TIER. List the tools; confirm you can reach list_endpoints, run_evaluation, poll_job, and load create_endpoint. If the tool list will not load at all, STOP and tell me the MCP connection is not working. BYOE requires Pro+; if a later call returns a tier error, relay it and stop rather than retrying.

2. REGISTER THE ENDPOINT (the irreducible prerequisite).

   • Check list_endpoints first. If my endpoint is already registered, reuse it – do not create a duplicate.
   • Otherwise ask me for: a name, the base_url (OpenAI-compatible), the model_id to send, the auth_type (bearer default, api_key, or none), and the credential (required unless auth_type is none). Confirm I accept responsibility for costs on my provider account, and pass tos_acknowledged=true. Create it with create_endpoint. The credential is stored encrypted and never returned.
   • Optionally run test_endpoint first to confirm it is reachable before we spend a scored run on it; if the test fails, show me the error and let me fix the config before continuing.

3. SETTLE THE SCORED RUN (one short exchange).

   • The task prompt everyone is scored on. Use my real task, not a generic prompt – the whole point is my task.
   • The scoring standard. A GOLD-STANDARD answer (expected_output) is the fastest start and is all we need; offer that first. A saved rubric is fine if I prefer it.
   • The commercial field: a few models my tier allows (list_models), spanning cheap to flagship, so my endpoint has a meaningful comparison. Respect my per-run cap – my endpoint counts as one of the slots.

4. RUN ONE SCORED EVALUATION. run_evaluation with: the prompt, the models map INCLUDING my registered endpoint alongside the commercial models, and EITHER expected_output OR rubric_id – never both. This returns a JOB id; poll poll_job every 5 seconds to completion (stop after 5 minutes). The job id and the evaluation id are DIFFERENT.

5. REPORT SIDE BY SIDE. From the completed result, show my endpoint’s quality and cost next to each commercial model. Say plainly where my endpoint wins (equal or better quality at lower or comparable cost) and where a commercial model is the better buy (same quality, lower cost). Do not soften an unflattering result – the honest side-by-side is the value.

6. HAND OFF. Tell me I can rerun this comparison any time with mcp-compare-models, or turn it into a procurement decision (cheapest above a quality bar, endpoint included) with mcp-model-procurement. Mention I can pick up another onboarding goal from the Getting started entry in the sidebar.

ON ANY FAILURE: tell me honestly where it failed and with what error (a Pro+ tier gate on create_endpoint is a clear, expected error – relay it, do not retry blindly). Do NOT fabricate scores or claim the endpoint was registered or scored when a call failed.