The $323 model and the $0.60 model scored the same. Until the task got harder.
We ran a root cause classification task across seven models, four times each. The task was designed to be hard in a specific way: five support tickets, all describing problems with the same product feature, but with different root causes buried in timezone and timing signals. A human reads it and sees the pattern immediately. An LLM has to reason across all five tickets simultaneously, convert local check times to UTC, and resist grouping by surface symptom rather than mechanism.
On a simple summarisation task, every model in this set scores 100 with a clean rubric. This task was different. Here is what happened.
4-run average quality score by model
4-run averages. Kimi K3 timed out on all four runs and is excluded.
The task
The cognitive demand is specific: simultaneous multi-ticket reasoning with cross-timezone normalisation. Five tickets, three root causes, two deliberate distractors. The model has to hold all five in working memory, identify which share a mechanism, and explain why the surface symptoms differ.
Prompt
A customer success team has received five support tickets in the past week, all relating to the same product feature. Your task is to analyse the tickets and produce:
- A grouping of the tickets by root cause – which tickets share the same underlying problem and which are distinct issues
- For each group, a one-sentence explanation of the root cause
- A confidence level for each grouping: high, medium, or low
Do not guess. If the evidence is insufficient to determine whether two tickets share a root cause, keep them separate and state why.
Context store -- 5 tickets
Ticket 1 – Submitted 9:14am SGT (UTC+8), Monday From: sarah.chen@acmecorp.com / Subject: Daily report not generated
Hi, our daily summary report didn’t appear this morning. I checked at 9am Singapore time and it wasn’t there. This has happened occasionally before but today it’s important because we have a board meeting. The report was there last Friday and every day the week before. We’re based in Singapore (UTC+8). Can you look into this urgently?
Ticket 2 – Submitted 2:31pm GMT, Monday From: marcus.webb@brightside.io / Subject: Can’t access premium reports
Hi, I tried to run our weekly report this afternoon and got an error saying my account doesn’t have access to premium features. I’ve been on the Pro plan for four months. I checked my billing and my card was declined last Thursday. I’ve since updated it but the access hasn’t come back. Can you restore my access?
Ticket 3 – Submitted 8:47am WAT (UTC+1), Tuesday From: james.okafor@deltaventures.ng / Subject: Report missing again
This is the second time this week our morning report hasn’t been generated. Monday and Tuesday both. We’re in Lagos (UTC+1) and I check the report first thing when I arrive at the office around 8am local time. It was there every morning last week without fail. Nothing has changed on our end. The strange thing is my colleague in our London office (currently UTC+1) checked at 10am their time on Monday and said the report was there for them. We’re both UTC+1 but I check at 8am and they check at 10am.
Ticket 4 – Submitted 11:05am IST (UTC+5:30), Tuesday From: priya.nair@horizonhealth.in / Subject: Report showing stale data
Our daily report generated fine this morning – it was there when I checked at 11am Mumbai time. But the data looks like it’s from two days ago. I cleared my browser cache and refreshed and it’s still showing old data. Our data team confirmed the underlying database was updated yesterday evening. This seems like a display or caching issue on your end rather than a generation problem.
Ticket 5 – Submitted 9:22am CEST (UTC+2), Wednesday From: tom.fischer@alphalogistics.de / Subject: No report this morning – third day running
Our daily report hasn’t generated for three consecutive days now. We’re in Frankfurt (UTC+2 in summer). I check at 9am local time every morning. My counterpart at our New York office checked the same report at 9am their time (UTC-4) on Monday and said it was there fine. This is affecting our morning operations meeting at 9:30am. Please treat this as urgent.
Rubric
Grouping Accuracy: The response correctly identifies that tickets 1, 3, and 5 share a root cause and that tickets 2 and 4 are distinct issues unrelated to each other and to the first group. A response that correctly groups 1/3/5, isolates 2, and flags 4 as ambiguous with a reasoned explanation is also acceptable. A response that groups all five together, splits 1/3/5 incorrectly, or merges 2 or 4 into the main group fails this criterion.
Root Cause Reasoning: The response identifies the correct mechanism linking tickets 1, 3, and 5: a time-dependent process failing for users whose local morning falls before a reset or generation window completes – inferable from the timezone and timing signals. The response does not need to use the word “timezone” but must identify the shared temporal pattern and the contrast with last week. A response that identifies the grouping correctly but attributes it to a vague or invented cause fails this criterion.
Distinction Quality: The response correctly identifies why tickets 2 and 4 are distinct: ticket 2 is a billing/access issue with a clear stated cause, ticket 4 is a data freshness or caching issue with no generation failure. A response that correctly identifies ticket 2 as a billing issue and flags ticket 4 as ambiguous with a reasoned explanation is also acceptable.
The correct answer requires converting every check time to UTC, identifying that the affected users all check before a generation window closes, and using the London and New York colleague signals in tickets 3 and 5 as controls. Ticket 4 (Mumbai, UTC+5:30) is a deliberate distractor: the right timezone profile, but a different symptom entirely. A model that groups ticket 4 with 1/3/5 based on geography alone rather than symptom analysis is reasoning lazily.
One model type failed before the reasoning even started. Kimi K3 timed out on all four runs. It is a thinking-token model: it generates an extended internal reasoning chain before producing output. On a task this long, it hit the timeout window every time. That is an operational constraint, not a quality signal – but it matters for production. Some task shapes are simply incompatible with certain model architectures regardless of how capable the model is.
The results
Grok 4.5 swung 67 points between its worst run and its best. That number is the story.
The per-run variance is more informative than the average. Here is the full picture:
| Model | R1 | R2 | R3 | R4 | Avg | Min | Cost/call | Monthly @100/day |
|---|---|---|---|---|---|---|---|---|
| Claude Opus 5 | 100 | 100 | 96.7 | 100 | 99.2 | 96.7 | $0.10772 | $323 |
| Grok 4.5 | 33.3 | 100 | 100 | 100 | 83.3 | 33.3 | $0.00571 | $17 |
| Qwen 3.8 27B | 100 | 41.7 | 75 | 100 | 79.2 | 41.7 | $0.00318 | $9.50 |
| GPT-4.1 Nano | 75 | 75 | 66.7 | 66.7 | 70.9 | 66.7 | $0.00021 | $0.60 |
| GPT-4o Mini | 66.7 | 66.7 | 66.7 | 66.7 | 66.7 | 66.7 | $0.00028 | $0.84 |
| DeepSeek V4 Flash | 41.7 | – | 75 | 41.7 | 52.8 | 41.7 | $0.00115 | $3.50 |
| Kimi K3 | – | – | – | – | – | – | $0.06401 | $192 |
Monthly cost at 100 calls/day. If this is a weekly batch job, Claude Opus 5 costs $46/month, not $323.
Score range across 4 runs (min / avg / max)
Min, avg, and max across 4 runs. GPT-4o Mini is the only model with zero variance – consistent, but consistently below the 80 threshold.
Three distinct risk profiles emerge:
- Claude Opus 5: floor of 96.7, ceiling of 100. The only model that never dropped below the 80 quality threshold on any run. $323/month at 100 calls/day.
- Grok 4.5 and Qwen 3.8 27B: capable of 100, but unreliable. Grok scored 33.3 on run 1. Qwen scored 41.7 on run 2. Both recovered on subsequent runs. At $17/month and $9.50/month respectively, the cost case is compelling – if you can tolerate the variance.
- GPT-4.1 Nano and GPT-4o Mini: consistent, but consistently below 80. Never collapsed, never broke through. At $0.60/month and $0.84/month, this is the profile for tasks where 66-75 is acceptable and predictability matters more than peak performance.
What the best response looked like
Claude Opus 5 run 2 normalised every check time to UTC before drawing any conclusions. No other model did this explicitly. That single step separated it from every other model in the set.
Response -- Claude Opus 5, run 2 (score: 100)
Normalising the evidence to UTC
| Ticket | Local check time | UTC | Report present? |
|---|---|---|---|
| 1 Singapore (UTC+8) | Mon 09:00 | Mon 01:00 | No |
| 4 Mumbai (UTC+5:30) | Tue 11:00 | Tue 05:30 | Yes, but stale |
| 3 Lagos (UTC+1) | Mon and Tue 08:00 | 07:00 | No |
| 5 Frankfurt (UTC+2) | Mon-Wed 09:00 | 07:00 | No |
| 3 London colleague (UTC+1) | Mon 10:00 | Mon 09:00 | Yes |
| 5 New York colleague (UTC-4) | Mon 09:00 | Mon 13:00 | Yes |
The pattern is not geographic and not account-specific. It is purely a function of when in UTC the user looks. Everyone checking at or before 07:00 UTC sees nothing. Everyone checking at or after 09:00 UTC sees the report.
Group A – Tickets 1, 3, 5 (high confidence): The daily report generation job is completing several hours later than it used to – now somewhere between 07:00 and 09:00 UTC – so users who check before that window find no report, while colleagues in the same organisation checking later find it present and correct. Ticket 1 bounds the regression: the report was reliably available at 01:00 UTC every day the previous week, so the job previously finished before 01:00 UTC and has now slipped by at least six hours.
Group B – Ticket 2 (high confidence): A billing/provisioning failure. The account was downgraded when the card was declined, and the entitlement restoration did not fire when payment was updated. Different symptom, different trigger, no time-of-day component.
Ticket 4 – held separate (low confidence on distinctness): Priya’s report was present at 05:30 UTC Tuesday, which directly contradicts Group A’s 07:00-09:00 UTC boundary – unless what she is seeing is not Tuesday’s report at all, but the last successfully generated report being served as a fallback. Under that reading, Monday’s and Tuesday’s runs both failed, leaving Sunday’s output as the newest available – which matches “data from two days ago.” I cannot choose between these interpretations from the tickets alone. What would resolve it: generation job logs showing actual completion timestamps, and the generation timestamp embedded in the specific report Priya viewed.
Two things stand out beyond the correct grouping. First, the UTC normalisation table: Claude converted every check time before drawing any conclusions. Second, the treatment of ticket 4: rather than forcing a classification, it flagged the ambiguity, explained both interpretations, and named exactly what evidence would resolve it. The rubric was updated to accept this as correct – the model’s response was better than the original criterion, which required a definitive answer.
A model that refuses to guess when the evidence is genuinely insufficient is more useful in production than one that guesses confidently.
What the weaker responses looked like
The failure modes split into two types.
Type 1: Correct grouping, vague mechanism. GPT-4o Mini grouped the tickets correctly on every run – 1/3/5 together, 2 and 4 separate. But the root cause explanation never got past the surface:
Response -- GPT-4o Mini, run 1 (score: 66.7)
Group 1: Missing Reports (Tickets 1, 3, 5) Root Cause Explanation: These tickets all report instances where daily reports were not generated at expected times, suggesting a potential issue with the report generation system. Confidence Level: High
Group 2: Access Issue (Ticket 2) Root Cause Explanation: This ticket highlights a problem with account access to premium features, likely due to a billing issue that has since been resolved but was not updated in the system. Confidence Level: High
Group 3: Stale Data (Ticket 4) Root Cause Explanation: This ticket indicates a problem with displaying outdated data in a generated report, rather than an issue with the report generation itself. Confidence Level: High
“A potential issue with the report generation system” is not a root cause. It is a restatement of the symptom. The grouping is correct but the mechanism is missing. GPT-4o Mini scored 0 on Root Cause Reasoning on every run – not because it failed to read the tickets, but because it never connected the timezone signals to the generation window.
Type 2: Partial reasoning, wrong conclusion. Qwen 3.8 27B run 2 is the more interesting failure. It understood the evidence in ticket 3 – the Lagos/London 8am vs 10am contrast is the smoking gun, and Qwen identified it. But it treated ticket 3 as a separate group from tickets 1 and 5, rather than recognising that all three share the same mechanism:
Response -- Qwen 3.8 27B, run 2 (score: 41.7)
Group A: Ticket 3 Root cause: The missing report is likely caused by a timezone or timestamp logic error where the report generation or visibility window does not align with the user’s local 8am check time, despite the user and a colleague in the same UTC offset (UTC+1) seeing different results at different local times (8am vs 10am). Confidence: Medium
Group B: Ticket 1, Ticket 5 Root cause: The daily report failure is likely caused by a systemic generation outage or failure affecting users in specific time zones (UTC+8 and UTC+2) during their morning check-ins, while potentially sparing users in other time zones (UTC-4) who reported the report was present. Confidence: Medium
Qwen saw the mechanism in ticket 3 but described it as a distinct issue from tickets 1 and 5, which it attributed to a “systemic generation outage.” The insight was there. The synthesis was not. It got close to the right answer and then split it incorrectly.
The production cost of the GPT-4o Mini failure is low: a wrong root cause explanation sends the engineering team to the right tickets but the wrong layer. The production cost of the Qwen failure is higher: it splits a single bug into two separate investigations.
Find the right model for your task
Run the same task across 7 models and see where quality diverges before it matters in production.
What this means for model selection
Two tasks, same seven models, same rubric structure. On the summarisation task, every model scored 100 once the rubric was clean. On this task, only Claude Opus 5 was consistent above 96. The task changed. The model ranking changed with it.
The cost difference between Claude Opus 5 and GPT-4.1 Nano is 500x per call. On the summarisation task, that difference bought nothing. On this task, it bought a floor of 96.7 vs a ceiling of 75.
The model selection question stated precisely: not “which model is best” but “which model is best for this task at this complexity level.” Tasks that require multi-step reasoning across ambiguous evidence, cross-document synthesis, or deliberate distractors are the ones where frontier models earn their cost. Tasks with a single correct answer and clean context are not.
GPT-4.1 Nano’s profile is worth noting separately. It never broke 80, but it never collapsed either. Floor of 66.7, ceiling of 75, $0.60/month. For tasks where that range is acceptable and consistency matters more than peak performance, that is the right choice. Not every task in your stack needs to be above 80.
A pipeline running one model uniformly across all tasks is almost certainly overpaying on some tasks and underperforming on others. The benchmark tells you which is which.
This post is part of the ROI Maxxed AI framework: 14 levers for building an LLM stack where every task gets the cheapest model that delivers the quality that task needs.