Five problems an agent workflow's cost didn't show

This is the first test of agent evaluation in LLM Prover. An agent analysis checks whether each step of an AI agent’s workflow (software using an AI model to work through steps on its own) did useful work, not just what it cost.

We ran it on two real workflows from a multi-agent job (one task split between several agents) that we run ourselves. In the first workflow, the agent analysis reported three problems and our step-by-step review of the same trace found two more, five in all. The second workflow had one low and one medium problem. Below is each problem, its numbers, and why the workflow’s cost total (the credits the workflow was billed) did not show it.

The job we evaluated

The job researches a list of public web sources and records what it finds. It runs in Kiro, an AI coding and agent tool, which records cost in credits, its billing unit. The job has five steps:

One step builds the list of sources and splits it between the workers.
Three workers each research their share, all at the same time.
A final step checks and records the results.

Two terms used below:

Turn: one round of the AI model reasoning and acting.
Tool call: one action taken during a turn, such as opening a web page.

What the evaluation does

An agent analysis looks at a workflow as a map of its steps and which step feeds which (how agent evaluation checks a whole workflow). An automatic pass walks that map the same way every time and flags likely problems:

the same work done twice,
a step whose output nothing else uses,
a step that produced nothing, or kept working after repeated failures, and
work split unevenly between parallel workers.

Some flagged problems then get a reasoned review: what was the step for, did it do that, and does the cause sit in an earlier step? The number of these reviews is capped to keep the cost down, because reasoning over every step of a large or complex workflow would be very expensive.

Insight: Cost, time and step counts tell you how much an agent did. The evaluation asks whether each step’s work was needed and where a problem started.

The findings in workflow 1

Workflow 1 cost 31.3 credits. Workflow 2 cost 16.0, about half, but it researched a different list. The two cost totals are not a like-for-like comparison, and the gap alone explains nothing.

Finding	Found by	Where	Costs money or time?	Why the cost total in credits missed it
Items split evenly by number, not by how much work each needed (severity: high)	Agent analysis	The step that builds and splits the list	Time	Uneven work costs time, not money, and the cost total in credits measures only money
Two workers took 44 and 43 turns, against 23 for the third	Agent analysis	Workers 1 and 2	Money	The cost total shows the credits spent, not that part of them bought extra turns
A check the instructions asked for was not done	Our review	The step that builds and splits the list	No cost effect observed	Skipping work adds no cost and no error
Each step read the same two documents on its own	Agent analysis	All five steps	Money	The cost total shows the credits spent, not that part of them paid for the same reading five times
Results were saved under the wrong kind of label	Our review	Where the workflow’s results are recorded	No cost effect observed	A wrong label costs nothing extra

Severity was recorded for the first finding only. Here is what the trace shows for each step.

Workflow 1, per step (agent analysis output on the trace, 4 Nov 2026)

Build and split the list: 1.07 credits, 43 s, 7 turns, 8 tool calls
Worker 1: 12.05 credits, 436 s, 44 turns, 111 tool calls
Worker 2: 11.26 credits, 345 s, 43 turns, 94 tool calls
Worker 3: 4.97 credits, 225 s, 23 turns, 63 tool calls
Check and record: 1.96 credits, 102 s, 11 turns, 22 tool calls
Cost total: 31.3 credits. The three workers ran at the same time and finished in 436 s.

1. The uneven split

The first step gave each worker the same number of items, without regard to how much work each item needed. Worker 1 took 436 seconds and worker 3 took 225, and the workflow waited for the slowest. The agent analysis reported the uneven split and traced its cause to the step that made the split.

Costs: time. The workers did the same amount of work either way, so no extra money.
Why the cost total missed it: it counts credits, not seconds spent waiting.

2. The extra turns

Workers 1 and 2 took 44 and 43 turns and made 111 and 94 tool calls. Worker 3 took 23 turns and made 63. The agent analysis flagged both workers and judged the extra turns wasteful on worker 1, but not on worker 2.

Costs: money. Every turn is paid for.
Why the cost total missed it: it shows the credits spent, not that part of them went on extra turns.

3. The missed check

Our step-by-step review found this one. The first step’s instructions said to check that each item exists. The trace shows the step did not do it, in a step that cost 1.07 credits and took 7 turns.

Costs: no cost effect observed.
Why the cost total missed it: leaving out work adds no credits and produces no error.

4. The repeated reading

Each of the five steps read the same two documents, its instructions and a description of the task, on its own. That is about 21 KB per step and about 105 KB across the workflow. Once read, the documents stay in that step’s working memory, so each step read them once. The agent analysis reported this one and put it at about 1.07 credits.

Costs: money.
Why the cost total missed it: it includes the credits for the reading but cannot separate them from useful work.

5. The wrong label

Each workflow’s results are saved under a short label naming the kind of sources it covered, so that results of the same kind can be grouped. In this workflow a long description of the workflow was saved there instead, which splits results that belong together. Our step-by-step review found this bug independently; we had shipped a fix for it the same day we ran this analysis.

Costs: no cost effect observed.
Why the cost total missed it: a wrong label costs no credits.

Insight: One cost total of 31.3 credits held an uneven split, extra turns, a skipped check, repeated reading and a labelling bug. None of them changes that figure in a way you could spot.

Make your agent workflows faster, cheaper and more reliable

An agent analysis checks every step of a workflow and shows where it wasted money, time or effort, and what caused it. On Pro and above.

See Pro

The findings in workflow 2

We evaluated workflow 2 blind, meaning we carried nothing over from workflow 1. Its three workers cost 3.98, 4.85 and 4.28 credits, so the work was roughly balanced. The evaluation reported two findings:

repeated reading of documents (low severity), and
the same labelling bug (medium severity).

It did not report an uneven split or extra turns.

Insight: The evaluation told a workflow with several problems from one with few, using only each workflow’s own record.

Analyse your own agent workflows

You analyse your agent workflows from your agent, over MCP. Your agent sends the workflow’s trace straight from disk, so the trace never takes up its working memory, and gets back the findings report: each problem, whether it costs money or time, and the step where it started. It is on Pro and above. Setup: MCP quickstart.