What public agent benchmarks miss about your workflows
This page lists what five widely used public agent benchmarks test, what they cannot show about your own agent, and how to benchmark your own workflows. An AI agent is software that uses an AI model to work through steps on its own, such as editing code, browsing a website or calling tools. Public agent benchmarks score agents on a fixed set of tasks so that agents and models can be compared on equal terms; their ranked results are called leaderboards.
What the main public agent benchmarks test
| Benchmark | What the agent must do | Size | Source |
|---|---|---|---|
| SWE-bench Verified | Fix a real GitHub issue in a Python repository; the fix is checked by running the project’s tests | 500 tasks, each checked by human annotators | OpenAI |
| WebArena | Complete tasks on self-hosted copies of real kinds of websites, such as shopping, forums and code hosting | 812 tasks | WebArena paper |
| GAIA | Answer real-world questions that need reasoning, web browsing and tool use | 466 questions | GAIA paper |
| OSWorld | Complete tasks on a real computer: web and desktop apps, files, and workflows across several apps | 369 tasks; OSWorld 2.0 adds 108 longer workflows | OSWorld paper, OSWorld 2.0 |
| tau-bench | Hold a conversation with a simulated customer and use a company’s tools and policies, in retail and airline settings | Retail and airline domains | tau-bench paper |
Sources checked 9 November 2026. Also cited below: Automation Anywhere, AI agent benchmarks guide.
What public benchmarks measure well
- Whether an agent can finish a hard, fixed task. SWE-bench Verified, for example, checks a code fix by running the project’s own tests.
- How agents and models compare on equal terms. Everyone runs the same tasks with the same scoring.
- Consistency, in some cases. The tau-bench authors report that even strong agents that call tools succeeded on fewer than half of the tasks, and succeeded on all 8 repeated runs of the same task in fewer than 25% of retail cases (tau-bench paper, linked in the table above).
What public benchmarks cannot tell you about your agent
A public benchmark scores someone else’s tasks. It does not see:
- your tasks and data, which are what your agent actually works on,
- cost per task in your setup, with your models and prompts,
- time per task, including steps that wait on each other,
- whether each step’s work was needed, as opposed to whether the final answer passed, and
- how your agent changes over time, as models and prompts change.
Automation Anywhere, a vendor of agent software, argues that standard benchmarks are becoming saturated and easy to game, so a leaderboard score is a directional signal, not a guarantee of reliability in production (Automation Anywhere 2026 guide, listed in the sources below the table).
Know what your agent costs, how long it takes and where it wastes work
Score your own prompts against your bar on a schedule, and analyse your real agent workflows step by step. On Pro and above.
How to benchmark your own agent
1. Each model call: a scheduled benchmark on your own prompts
Run your own prompts against several models, score every answer against your own pass mark, and repeat on a schedule. This tells you which model passes your task, at what cost and speed, and whether that changes.
Evaluation 9457c72f, a single run on 4 November 2026 (support-ticket triage, scored against a known correct answer, pass mark 80)
- gemini-3.8-flash: score 100 (pass), $0.000153 per call, 15.7 s
- claude-sonnet-4-5: score 90 (pass), $0.000639 per call, 1.7 s
- gpt-4.1: score 75 (fail), $0.000340 per call, 0.6 s
In this run, the cheapest model cleared the pass mark but was the slowest; the fastest failed it. A public leaderboard could not have told you that for this task. A scheduled benchmark repeats the same check so you can see whether it holds. Scheduled benchmarks are available on Pro and above.
2. The whole workflow: an agent analysis
An agent analysis reads the trace of a real agent workflow (the record of every step) and checks every step for:
- the same work done twice,
- a step that kept going after repeated failures,
- work split unevenly between agents, and
- which earlier step caused a problem.
Cost and time are reported separately.
On one of our own workflows, it showed three workers (the agents sharing one task) costing 12.05, 11.26 and 4.97 credits and taking 436, 345 and 225 seconds. The work had been split by number of items rather than by effort, and the run waited about 100.7 seconds longer for the slowest worker. See all five problems that run’s cost didn’t show.
Questions
Is a high leaderboard score enough to pick an agent? It is a good way to shortlist. Before you commit, run the shortlisted models or agents on your own tasks and check cost and time as well as the score.
How often should I re-check? Whenever you change a model, a prompt or a tool, and on a regular schedule in between.
Run scheduled benchmarks and agent analyses on your own agent from your agent over MCP (the protocol agents use to call outside tools), on Pro and above: see how agent evaluation works.