JJ Signal · Benchmark · 8 min read
What an agent actually costs to run
The model bill is rarely the expensive part, and the part that is expensive is rarely budgeted.
Published Retail & E-commerce, Healthcare & Clinics, Professional Services, Logistics & Supply Chain
Agent cost projections are built from token prices, and token prices are the most predictable and usually the smallest component. The costs that actually surprise people are retries on malformed output, retrieval over corpora that grew, and the human review time that the automation created rather than removed.
The evidence
Drawn from our own delivery work. Each claim states its sample and its method, including where the sample is small.
- 01Retry and repair traffic was a material share of spend wherever output was not schema-constrained.Comparing runs before and after forcing structured output at the tool boundary. Constraining the output shape removed most of the repair loop, and with it the cost.
- 02Routing cheap work to a smaller model mattered more than any prompt optimisation we tried.Classification, extraction and triage carry most of the volume and almost none of the difficulty. A router is a larger lever than a better prompt, and it is a one-time change.
- 03Human review time created by an agent is frequently absent from the business case entirely.Where a human gate is required — and in regulated work it is — the reviewing hours belong in the cost model. Several of our own early estimates omitted them, which is why we mention it.
What follows from it
Budget for retries, retrieval growth and review hours, not just tokens. An agent whose output is schema-constrained and whose cheap work is routed to a cheap model costs a fraction of the same agent built without either, and the difference is structural rather than incremental.
What we would do
- 01Force structured output at the tool boundary so a malformed response is retried by the framework, not by a repair prompt.
- 02Route by difficulty before optimising any prompt. Most volume is triage and extraction.
- 03Meter cost per task and per tenant from day one; an agent without a cost attribution is an agent nobody can govern.
- 04Put the human review hours in the business case, and re-check them after go-live rather than at proposal time.
Get JJ Signal by email
Roughly monthly, and only when there is something measured to report. If a month produces nothing worth your attention, you get nothing.
What you are signing up to