What GPT-4 Actually Costs vs Open Source Models Over a Year

I spent a weekend with a spreadsheet trying to answer a question a client asked me three times in two months: “should we just self-host instead of paying OpenAI?”

Every time, my gut said “depends on volume.” So this time I actually built the model instead of guessing. Twelve months, three volume tiers, real line items for GPU rental, engineering time, and the boring ops work nobody budgets for. Here’s what came out.

The Setup

I compared two paths for a team running an LLM-backed feature - support triage, content generation, whatever. Not fine-tuning, not training from scratch. Just inference.

Path one: call the GPT-4 API and pay per token. Path two: rent GPUs, deploy an open source model like Llama 3 or Mixtral, and serve it yourself.

Three volume tiers, because the answer changes completely depending on where you sit:

• 10,000 requests/month (a small internal tool)
• 100,000 requests/month (a real product with actual users)
• 1,000,000 requests/month (something that matters to the business)

I assumed an average of 1,000 input tokens and 500 output tokens per request. Reasonable for a support-style prompt with some context stuffed in. Your mileage varies - a RAG pipeline with long documents burns through input tokens fast.

The API Side

GPT-4-class API pricing sits in the range of a few dollars per million input tokens and roughly 3-4x that for output tokens, depending on which tier and model variant you pick. I won’t pin an exact number here because pricing moves every few months and whatever I write will be stale by the time you read it (check the current rate card, always).

What matters is the shape of the cost curve: it’s linear and it’s variable. Every request costs the same fraction of a cent whether you send one or a million. No fixed cost. No idle capacity sitting around burning EUR while nobody’s using it.

At 10,000 requests/month, the bill is small enough that most teams don’t even notice it on the invoice. At 1,000,000 requests/month, it’s a real line item - the kind that shows up in a monthly finance review and gets a raised eyebrow.

The Self-Hosted Side

Self-hosting flips the cost structure. Instead of paying per request, you pay for capacity whether you use it or not.

The GPU rental cost itself is only part of the story, and honestly the part people fixate on the most while ignoring the rest. A single mid-tier GPU rented by the hour, running 24/7 for a month, adds up to a few hundred to low four figures in EUR depending on the card and the provider. That’s before anything else.

Then the parts nobody puts in the spreadsheet:

Engineering time. Someone has to set up the serving stack (vLLM, TGI, whatever), tune batch sizes, handle model loading, and debug why throughput dropped 40% after a driver update. Budget at least a few days of senior engineer time up front, and ongoing maintenance after that. In Germany that’s not cheap - a contractor at a reasonable day rate eats the “savings” fast if you’re not careful.

Ops overhead. Monitoring, autoscaling (or the lack of it), handling GPU node failures, patching. This is real work that either you do or someone on your team does instead of something else.

Idle capacity. Unless your traffic is perfectly flat 24/7 (it never is), you’re either overprovisioned and paying for GPU time nobody uses, or underprovisioned and users hit timeouts during peak hours. Autoscaling helps but doesn’t eliminate this - GPU cold starts are slow, often 30-90 seconds, so you can’t scale as reactively as you’d like.

Model quality gap. Open source models have closed a lot of ground, but for complex reasoning tasks, GPT-4-class models still tend to win. If your use case needs that ceiling, swapping to Llama isn’t free even if the infra is cheaper - you might need more retries, more prompt engineering, or a bigger model that costs more to serve.

Where the Crossover Actually Sits

Here’s the part that surprised me. I expected self-hosting to win somewhere around 100,000 requests/month. It doesn’t, not once you include engineering and ops time honestly.

At 10,000 requests/month, self-hosting loses badly. You’re paying for a GPU sitting mostly idle plus the setup time, against an API bill that’s practically a rounding error. Don’t self-host at this volume. Nobody should, and yet every few months someone tries.

At 100,000 requests/month, it’s closer, but the API still usually wins once you count engineering time honestly. The GPU utilization is better, but you still need someone maintaining the stack, and that person’s time is worth more per hour than the GPU savings you’re capturing.

At 1,000,000 requests/month, self-hosting starts to make real sense - roughly in the range of half to a third of the API cost once utilization is decent and the engineering cost is amortized over the volume. This is also where it’s worth it to actually invest in the ops work, because the absolute EUR at stake justifies a dedicated person or team owning it.

The honest takeaway: the crossover point is a lot higher than the “just self-host, it’s cheaper!” crowd on Hacker News implies. Below six figures of monthly requests, just pay the API. Your engineering time is worth more doing something else.

The “Why Not Both” Option

The real answer for most teams isn’t “API or self-hosted.” It’s routing requests to whichever backend makes sense for that specific call - cheap open models for simple classification and extraction, GPT-4-class models for the requests that actually need the reasoning.

That’s the whole premise behind Smart Inference, the router I built after running into this exact spreadsheet problem for a client. It routes each request to the cheapest capable model - sometimes a rented GPU running an open model, sometimes the frontier API - instead of forcing an all-or-nothing bet on one path. One honest limitation: it adds a routing decision to your request path, so if your use case is latency-critical to the millisecond, that overhead is worth testing before you commit.

What I’d Tell the Client

If I had to boil three volume tiers and a weekend of spreadsheet work into one sentence: stay on the API until the bill genuinely hurts, and even then, look at routing before you look at running your own GPU fleet.

Self-hosting isn’t wrong. It’s just a bigger commitment than the “cheaper GPU hours” headline suggests, and most teams asking the question aren’t at the volume where it pays off yet.