Cost Per Completed Task: The LLM Metric That Matters

In one week at the end of July, two frontier labs moved in opposite directions on price. Anthropic shipped a new flagship and did not change the rate card. OpenAI cut the price of its cheap tier by roughly 80% and left its flagship alone. Read that as competitive noise and you miss what the two announcements agree on: both companies have stopped selling tokens and started selling completed work.

Claude Opus 5 arrived on 24 July at $5 per million input tokens and $25 per million output — identical to Opus 4.8. A fast mode runs about 2.5x quicker at twice the base price. The price sheet is not the story. The framing is. Nearly every headline claim in the release is normalised by cost rather than stated in absolute terms: more than double Opus 4.8’s performance on Frontier-Bench v0.1 at a lower cost per task; within 0.5% of Fable 5’s peak score on CursorBench 3.2 at half the cost per task; around 1.5x the next-best pass rate on Zapier AutomationBench for the same cost per task; past Fable 5’s best result on OSWorld 2.0 at just over a third of the cost. Only the ARC-AGI 3 number — roughly three times the next-best model — is quoted plain.

Six days later, effective 30 July, OpenAI cut list prices. GPT-5.6 Luna went from $1.00 to $0.20 per million input tokens and from $6.00 to $1.20 per million output. Terra went from $2.50 to $2.00 input and $15.00 to $12.00 output, about 20%. Sol’s pricing did not move, but a new Fast mode replaces Priority Processing in the API. Subscription pricing stayed where it was; Terra and Luna simply consume fewer credits in ChatGPT Work and Codex. The stated cause was serving efficiency — GPU kernel optimisation, speculative decoding, and context management — along with a claim that Sol itself did the production kernel work.

Two bets on where the money actually sits#

Anthropic is betting that buyers will pay a premium per token when the model needs fewer tokens, fewer attempts, and fewer humans in the loop to finish the job. OpenAI is betting that the volume — the vast, boring middle of enterprise workloads — sits at the bottom of the price range, and that an 80% cut there wins more revenue than it gives up.

Both bets are coherent. What matters is that neither treats dollars per million tokens as the unit of value. When a lab’s own marketing normalises its benchmarks by cost, per-token price has stopped being the axis of competition. The axis is cost per completed task — a quiet concession that a token is an input, not an outcome.

The trouble is that their version of that metric is unusable for your planning.

Why vendor cost-normalised benchmarks cannot be planned against#

“Half the cost per task” is a real measurement over a real task distribution. It is just not your task distribution.

CursorBench, OSWorld and AutomationBench each carry an implicit profile: average input length, output length, tool-call depth, how many turns a typical episode runs, and how often a failure is recoverable. Change any of those and the cost ratio moves, sometimes by an order of magnitude.

Take a concrete case. In a Hospital Management System, a claims-coding task looks like this: a long clinical note in, a short structured code set out. That is an input-heavy workload with an input-to-output ratio that might be twenty to one. Now look at the OpenAI cut. An 80% reduction on Luna’s input price does an enormous amount for that task. In a School ERP, the end-of-term report-generation task inverts the shape: a compact set of grades and attendance records in, several paragraphs of narrative per student out. That workload is output-dominated, and output tokens are where the price stays high across every vendor. Same 80% headline, wildly different effect on the bill.

A benchmark cannot know which of those you run. You can.

How to measure cost per completed task in your own pipeline#

This is the part worth doing. It is a week of work in most systems and it changes procurement decisions for the following year.

Define “completed” as something the business already recognises#

Not a benchmark score. A binary your organisation already adjudicates. The claim passed payer validation. The report went to the parent without an edit. The extracted invoice matched the ledger. If your definition of success requires inventing a new rubric, you have picked the wrong unit — find the pass/fail your operations team is already applying by hand and instrument that.

This is the point where most AI implementation work quietly goes wrong. Teams measure model output quality against a rubric written by the team that built the model integration, which is a closed loop with no business signal in it.

Instrument total tokens per task, retries included#

Log tokens at the task level, not the call level. One task equals every call made in service of it: the first attempt, the reformulation after a schema validation failure, each tool-call round trip, the retry after a timeout, and the attempts that were abandoned entirely.

The retry tail is where the real money is, and per-token price lists ignore it completely. A pipeline with a 12% hard-failure rate and two automatic retries before escalation is spending materially more per completed task than its per-call cost implies, and none of that appears on a vendor comparison page. In our experience auditing Operational Automation pipelines, the gap between naive cost-per-call and true cost-per-completed-task is routinely a factor of two or three — and it is entirely invisible until someone joins the token log to a task identifier.

That join is the actual engineering work. Emit a task ID with every model call and land both in the same table in your Data Platforms layer. Everything else here is a query.

Track the input-to-output ratio explicitly#

Store input and output tokens as separate columns and report the ratio per task type. It tells you immediately which price cuts are relevant to you. An 80% reduction on input pricing is close to meaningless for a narrative-generation workload and decisive for a long-document extraction workload. Without the ratio, every vendor announcement is a coin flip.

Test cheap-plus-retries against expensive-first-try empirically#

Here is the trade that per-token pricing hides. Work an illustrative example with the published numbers, using your own failure and labour rates in place of mine.

Say a task sends 8,000 input tokens and returns 1,000 output tokens. On Luna’s new pricing that is about $0.0028 per attempt. On Opus 5 it is about $0.065 — roughly 23x more per attempt. Per token, it is not close.

Now run a thousand of them. Suppose the cheap model averages three attempts and escalates 15% of tasks to a human reviewer, while the expensive model finishes on the first try and escalates 4%. Suppose review costs six minutes at a fully loaded $40 an hour, so $4 a task. The cheap path costs about $8 in tokens and $600 in review. The expensive path costs $65 in tokens and $160 in review. The model that is 23x more expensive per token is roughly 2.7x cheaper per completed task.

Change the escalation rates and the answer flips. That is the point. This is an arithmetic question with a measurable answer, and almost nobody measures it. Teams argue about it from intuition instead, then pick the cheap tier because the number on the pricing page is smaller.

Keep a golden set of 50 to 200 real tasks#

Sampled from production, with adjudicated pass/fail labels, versioned in the repository next to the code. Re-run it whenever a price changes, a model ships, or a prompt is edited. Prices now move monthly — twice in the last week of July alone — so an eval you can only run by hand is an eval you will not run.

Fifty tasks catches a regression. Two hundred is enough to trust a routing decision. Beyond that you are buying precision you will not act on.

Look at caching and context management before you look at models#

Prompt caching and context discipline frequently move cost per completed task further than any model swap. Both labs cited context management as part of their own efficiency gains, which is a fair hint. Restructuring a prompt so the stable prefix stays stable is an afternoon of work with no eval risk attached. We covered the broader shape of that shift in the July 2026 inference price reset — the cache boundary is still the cheapest lever in the stack.

The floor under all of this#

The macro picture says prices keep falling regardless of what demand does.

Microsoft reported capex plus finance leases of roughly $41B for the quarter on 29 July, up 69% year over year, with about two thirds of it going to short-lived assets — CPUs and GPUs. Azure grew 43% and crossed $100B annualised. Alphabet raised 2026 capex guidance on 22 July from $180–190B to $195–205B and said it is renting third-party capacity as a bridge.

That spend is committed and the hardware depreciates whether or not it is busy. Idle accelerators are pure loss, so the rational move is to cut prices until they are full. Expect more cuts, at irregular intervals, landing unevenly across input, output, and cached tokens.

Which leaves one architectural conclusion: switching models must be a configuration change, not a rewrite. One interface, provider adapters behind it, a golden set you can run in an hour. If a price cut takes a quarter of engineering time to act on, it is not a price cut you receive.

One last thing, said plainly. “We used our own model to optimise our GPU kernels” is a recursive self-improvement claim, and it has been made without published verification — no methodology, no baseline, no independent reproduction. It may well be true. Treat it as marketing until someone outside the company measures it.

Cost per token is a price. Cost per completed task is a decision. We build the instrumentation, evals, and routing layer that tells you which model is actually cheaper for the work you run. Talk to us about measuring what your pipeline really costs.