What AI features actually cost to run
Most AI projects are approved on build cost and killed by running cost. The bill behaves unlike anything else in a software budget: it scales with usage rather than with headcount or seats, and it is trivially easy to build something that works beautifully and costs more per month than the person it replaced.
Why doesn't the pilot predict the bill?
Because a pilot runs on the happy path with a handful of users, and almost every cost driver in production is absent from it. Retries when a provider is slow. Users pasting a forty-page document into a field sized for a paragraph. A background job re-processing the same records nightly because nobody added a guard. An integration loop that calls itself.
The second reason is that pilots rarely include the work you did not think of as AI. Embedding a document collection to make it searchable is a one-off cost that returns every time the collection changes. Re-processing history after a prompt change can cost more than a month of live traffic.
So a pilot tells you whether the thing works. It tells you very little about what it costs, unless it was deliberately instrumented to.
How is inference actually priced?
Hosted model APIs charge by the token, which is roughly three quarters of a word, and they charge separately for what you send and what comes back. That single fact drives most of the cost surprises, because the part you control least is usually the part you send: if your prompt includes a document, a conversation history and a set of examples, you are paying for all of it on every single call.
Output is normally priced several times higher than input. A feature that returns a paragraph is cheap; one that rewrites a whole document is not, even when the input is identical.
Self-hosting replaces a per-call charge with a per-hour one. You pay for the GPU whether or not anyone uses it, which inverts the economics: hosted APIs suit spiky, low-volume workloads, and self-hosting suits steady high-volume ones. The crossover is a real calculation, not a preference, and it moves whenever either side changes its pricing.
| Cost driver | Effect on the bill | What to do about it |
|---|---|---|
| Prompt length | Charged on every call, so it multiplies | Trim examples, summarise history rather than resending it |
| Output length | Priced above input, often several times | Cap tokens, ask for structured output rather than prose |
| Model size | Larger models cost multiples of smaller ones | Route easy cases to a small model, escalate only hard ones |
| Retries | Invisible in design, real in billing | Cap attempts and log them, or they hide in the total |
| Re-processing | One prompt change can re-bill your whole corpus | Version prompts and re-process deliberately, never on deploy |
| Idle GPU (self-hosted) | Charged whether used or not | Only self-host once volume is steady |
Which controls actually keep the cost flat?
Four, and they are worth building before launch rather than after the first invoice.
Caching. A surprising share of requests in any real system are identical or near-identical, and serving those from cache costs nothing. This is the single largest lever in most integrations and the one most often skipped because it is unglamorous.
Model right-sizing. Most tasks in a workflow are easy, and easy tasks do not need your most capable model. Handling the bulk with a small model and escalating only genuinely hard cases typically changes the bill by a multiple rather than a percentage, with no measurable loss in output quality.
Hard ceilings, per workflow and per tenant. Not alerts, ceilings. An alert tells you about a runaway loop after it has run; a ceiling makes it a logged incident. Any system that can call a paid API in a loop needs one.
Per-feature attribution. Log usage against the feature that caused it, not just the account total. Without that, a bill that doubles is a mystery, and the instinct is to turn everything off rather than the one thing responsible.
What should you measure before committing?
Cost per completed unit of work, whichever unit the business already counts: per ticket resolved, per invoice processed, per document reviewed. That number is comparable to the manual cost it replaces, which is the only comparison that matters when someone asks whether to keep funding it.
Measure it at the ninety-fifth percentile rather than the average. Averages hide the long documents and the difficult cases, and those are what the bill is actually made of.
Then project it at ten times current volume. Anything that only works at pilot volume is a demo, and finding that out on a spreadsheet is considerably cheaper than finding out in production.
Where do the current prices live?
Deliberately not on this page. Per-token rates from the major providers change every few months, generally downward, and a page quoting figures from last year would be actively misleading. Check the provider's own pricing page for current numbers and put them into the model below.
The ratios, though, are stable enough to plan with: output costs more than input, larger models cost multiples of smaller ones, and cached requests cost nothing. Those three relationships have held through every price change so far, and they are what the design decisions hang on.
Common questions
- Why did our AI costs jump after launch?
- Usually one of four things: retries that were invisible in testing, users sending far more text than the pilot did, a background job re-processing records repeatedly, or a prompt change that triggered re-processing of a whole corpus. Per-feature usage logging tells you which within minutes; an account-level total tells you nothing actionable.
- Is it cheaper to self-host an AI model?
- Only at steady, high volume. Self-hosting swaps a per-call charge for a per-hour one, so you pay for the GPU whether it is busy or idle. Hosted APIs are usually cheaper for spiky or low-volume workloads, and self-hosting wins once utilisation is consistently high. It is a calculation from your actual traffic pattern, not a matter of preference.
- How much can caching realistically save?
- It depends entirely on how repetitive your requests are, which is why the first thing to do is measure duplication rather than assume it. In workflows where users ask similar questions of the same documents, the repeat rate is often high enough that caching is the largest single saving available. In workflows where every input is genuinely unique, it saves nothing.
- What is the most common costing mistake?
- Sending more context than the task needs on every call. Prompt length is billed per request, so an unnecessary document or a full conversation history in the prompt is a cost multiplier applied to every user, forever. Summarising history rather than resending it is usually the quickest large saving.
