Where Your AI Bill Actually Goes

I built Guard and forgot about inference costs for a month. Not because I did not care, but because at low volume the bill genuinely does not matter, and then one day it does, and by then the cost structure has calcified into whatever you happened to do first.

Once I actually sat down and broke apart where the money was going, most of it was not model choice. It was four things, and none of them are “pick a cheaper model,” which is the first thing everyone tries.

Model size is the least controllable lever

Yes, bigger models cost more per token. Everyone knows this. But this is also the lever people optimize first and get the least out of, because swapping to a smaller model usually means a quality hit you then have to work around somewhere else, more retries, more prompt engineering, a worse product. It is real, but it is not where most of the waste actually lives.

Prompt length is where the waste actually lives

Every token you send in, not just what comes back, costs money. A system prompt that grew organically over six months of “just add one more instruction” is often the single biggest line item nobody looks at, because it is invisible in the product and repeats on every single request.

I found a system prompt that had accumulated three redundant instructions saying roughly the same thing in different words, left over from three different debugging sessions where I added a rule instead of checking if one already existed. Trimming that cut the input token count on every request without changing behavior at all.

Check your system prompt length in tokens, not characters, and ask honestly whether every sentence in it is still doing something.

Caching is the biggest lever most people skip

If your system prompt and a chunk of your context are identical across many requests, most providers support prompt caching, where the repeated portion is billed at a steep discount on subsequent calls. This is the biggest single change available and the most commonly skipped, because it requires structuring your prompts so the static part comes first and the dynamic part comes last, which is not how most people write prompts by default.

If you have a long, mostly-static system prompt and a short, changing user query, and you are not using caching, you are paying full price for the static part on every single request. That is often the largest single inefficiency in a real production system.

Batching matters more for throughput than for per-request cost

If your workload can tolerate a small delay, batching multiple requests together improves GPU utilization and can lower per-request cost meaningfully, particularly on self-hosted or dedicated capacity. It matters less if you are already using a pay-per-token API, where the provider is doing the batching on their end regardless. Know which situation you are in before you spend time building a batching layer, it is not free effort and it is not always the bottleneck.

Output length is the one nobody budgets for

Output tokens are usually priced higher than input tokens, sometimes 3-4x higher, and an unconstrained model will happily write three paragraphs when one sentence would do. Setting a max token limit and, more importantly, actually prompting for concise output rather than just truncating it, is one of the cheapest changes you can make. Truncation wastes the generation cost of the tokens you throw away, a concise prompt does not generate them in the first place.

What actually happened when I looked

Between trimming a bloated system prompt, turning on caching for the static portion, and capping output length where the product did not need long responses, the bill dropped by more than switching providers ever did. Model choice was maybe a fifth of the total effect. The rest was structural.

That is also roughly why Smart Inference tracks actual cost per request rather than a theoretical rate card number, the theoretical number tells you almost nothing about where your money is actually going. You need the real number, broken down per request, to find the waste, and once you can see it, most of the fix is not a provider switch. It is looking at your own prompts honestly.