Changelog

November 2026

• Agent evaluation (agent analysis) – Check a whole agent workflow, not just one step. An agent analysis reads the trace of a Kiro or Claude Code workflow (detected automatically) and shows where it wasted money, time or effort and which step caused it, with cost and time reported separately. Start one from your agent over MCP or from the dashboard. On Pro and above; it counts toward your plan’s usage. See agent evaluation and how to read an agent analysis.

• Free AI Tools – a set of free, open source tools for testing AI models, each doing one job well. You run them with your own API key, in the browser or on your machine, so your prompts and your key never reach us. Eleven are live, from comparing models on your own question to testing how context changes an answer and checking output against a source. No account, no signup. See Free AI Tools.

October 2026

• Presets – save a run configuration once (models, scoring, rubric, context, parameters, or any subset) and invoke it by name from anywhere: the dashboard, the API, or a plain-language ask to your agent. Three run types fit a preset: a performance comparison, a quality evaluation, or a compliance check. A curated starter preset ships ready to run, and after a run you configured by hand the agent offers to save it for next time. On Pro and above. See Measure your prompts, models, and pipelines wherever you work.
• Scoring – score text you already have (statements, answers, opinions, clauses) against a rubric or your compliance documents, and get a per-criterion verdict. Paste one piece, or several to rank them against one standard. Scoring available on Pro and above. See You don’t need to generate to evaluate.
• MCP recipes – copy-paste playbooks that turn the MCP tools into jobs your agent runs end to end. Paste one into Cursor, Claude Desktop, Amazon Q, or Kiro, and the agent runs the tools in order, handles the async polling, and reports back. Covers comparison, scoring, rubric building, prompt A/B tests, drift monitoring, compliance checks, and more, including a closed optimization loop that tunes a prompt against a scoring standard. On Pro and above. See the MCP Recipes Guide.
• MCP server – the full tool surface is now available to any MCP-capable AI agent. Claude Desktop, Cursor, VS Code Copilot, and Amazon Q discover the tools automatically. Agents can run comparisons, evaluations, and benchmarks without leaving their workflow. MCP access on Pro and above. See the MCP and API docs.
• REST API – every dashboard operation is now available via REST. Fire comparisons, evaluations, and benchmark runs from any script, pipeline, or CI workflow. Developer keys on all plans from your account page.

September 2026

• Pause and resume benchmarks – stop a scheduled benchmark without deleting it, then resume when you are ready. Run a batch of measurements, pause to stop consuming usage, and resume when you need a fresh read. See the Benchmark Guide for the full workflow.
• Benchmark cost estimator – see estimated cost before scheduling a benchmark, including judge and diagnostic scoring
• Refresh buttons – one-click refresh on Comparisons, Evaluations, and Benchmark run history
• Model substitution – deprecated models are automatically substituted at run time; results show what was used
• BYOE endpoints – bring your own OpenAI-compatible model endpoint and benchmark it alongside hosted providers

August 2026

• Initial launch
  • Multi-model comparison
  • Rubric evaluation
  • Benchmarks
  • RAG context