How to Benchmark LLM Providers Without Losing a Week
I used to keep a spreadsheet of LLM providers. Column for price, column for latency, column for “seems fine.” It was wrong within a week, every time.
The problem is not that spreadsheets are bad. The problem is that provider pricing and performance are moving targets, and a snapshot is only true for the moment you took it. If you are picking a provider once and never looking again, fine, use a spreadsheet. If you actually care about cost and reliability over time, you need something that measures continuously instead of once.
Here is the methodology I landed on after getting it wrong a few times.
Measure latency as a distribution, not an average
The average latency number on a provider’s marketing page is close to useless. What matters is the tail. A provider that averages 400ms but spikes to 6 seconds on 5% of requests will wreck a user-facing feature even though the average looks great.
So track p50, p95, and p99 latency, not just the mean. p50 tells you the typical experience. p95 and p99 tell you what your unluckiest users see, and those are the numbers that show up in support tickets.
Do this over a sliding window, not a one-time test. An hour-long benchmark run tells you about that hour. Providers have good days and bad days, and infrastructure incidents do not announce themselves in advance.
Track error rates the same way
Errors are not evenly distributed either. A provider might run clean for six days and then have a rough two hours during a deploy. If your benchmark is a single afternoon test, you will never see that, and you will pick the provider that happened to be lucky when you were watching.
Track error rate as a rolling percentage over the last N requests, not a lifetime total. A provider that was reliable last month and flaky this week should show up as flaky now, not as “99.2% reliable since we started measuring.”
Cost per request, not cost per token
This is the one that trips people up most. Providers publish price per million tokens, and it is tempting to compare those numbers directly. But your actual cost per request depends on your prompt length, your output length, whether the provider charges differently for cached tokens, and what model version you actually got routed to.
Two providers with identical published token prices can produce a 3x difference in your real bill because one of them is quietly using a smaller context window or truncating differently. The only number that means anything is what you were actually charged, per request, measured after the fact. Not estimated. Measured.
What actually broke the manual approach
I ran the spreadsheet version of this for a few months on AIVory Guard, my compliance scanner. It worked fine at low volume. Then usage went up, and I found myself opening that spreadsheet every few days because the “best” provider from Tuesday was overloaded by Friday.
At that point the honest move was to stop doing this by hand. I built a router: one line of code, the API base URL changes, and requests get sent to whichever provider is currently cheapest among the ones passing a health check. The health check is the p95/p99 latency and rolling error rate described above, not a static list.
That became Smart Inference. It tracks actual cost per request, not the sticker price, and it prices by the GPU rather than by the VM, which matters more than it sounds like it should once you are running anything at real volume.
The part that will not fit in a spreadsheet
The thing a static comparison always misses is that “best provider” is not a fixed answer. It changes hour to hour based on price and health together, and the only way to actually track that is to keep measuring instead of writing it down once and trusting it.
If you are early and low volume, the spreadsheet is fine. Do it, pick one, move on with your life. But the moment provider choice starts affecting your bill in a way you notice, build the measurement in instead of doing it by hand every few days. You will get tired of it a lot faster than you think.