Effort Dials Broke My Model Routing Rules

I spent an afternoon in July rewriting a routing table that had worked fine for over a year, because it had quietly stopped meaning what I thought it meant.

The table was simple. Model name in, provider and cost out. Claude goes here, GPT goes there, cheap tasks get the small model. It worked because for a long time a model name was one thing: one price, one latency profile, one capability level. That stopped being true this year, and the routing table did not notice on its own, because nothing about the table’s shape would have told it to.

What tipped me off was a cost report that stopped making sense. The same model name showed up with a five times spread in per-request cost across two weeks, and the first thing I checked was whether a provider had quietly changed rates on me. Nobody had. The requests were going to the same model at three different effort levels, and my logging only ever recorded the model name, so the report was averaging three products together and presenting the average as one number.

The second axis nobody priced in

Every frontier vendor now offers an effort or thinking dial alongside the model itself. Claude Opus 5 launched on 24 July 2026 with low, medium and high effort settings, priced at $5 per million input tokens and $25 per million output, on a model with a May 2026 knowledge cutoff. OpenAI has carried a reasoning effort parameter across the GPT-5 family, and GPT-5.6, which rolled out globally on 9 July 2026, stacks a second layer on top: three separate tiers named Luna, Terra and Sol, from fast and cheap to flagship, each a different model at a different price. Gemini 3.1 Pro, released on 19 February 2026, carries its own three thinking levels, Low, Medium and High, plus a 1,048,576 token input window and 65,536 output tokens.

Three vendors, three different names for the same idea: the model you call is no longer one cost and latency profile, it is three or more, and which one you get depends on a parameter that lives next to the model name, not inside it.

A routing table keyed on model name alone was built for a world with one axis. Cost and latency now vary across two: which model, and which effort level. “Route cheap tasks to the small model” was never wrong exactly. It is just an answer to half the question.

What the key actually needs to carry

The fix is mechanical once you see it. The routing key stops being just a model identifier and starts carrying the effort level as a first-class field:

Two requests to the same model name are no longer comparable unless you also know the effort level each one asked for. Any dashboard, log line, or cost report that groups by model name alone is now averaging over several different products and calling it one number. Latency behaves the same way. A high-effort call to a model can run several times longer than a low-effort call to the identical model name, so a p95 latency chart keyed on model name alone is quietly blending a fast product and a slow one into a figure that describes neither.

Why the cheap-tier default fails

The obvious cost-cutting move is to pin everything to the low effort tier and call it done. It works right up until it does not: a low-effort pass on a task that actually needed multi-step reasoning does not fail loudly, it just gives you a worse answer, confidently, and the routing layer has no way to know that happened unless something downstream catches it.

The tasks where effort matters most are exactly the tasks where you are least equipped to detect a wrong-but-plausible answer automatically. That is the trap in “always cheap”: the failure mode is silent, not an error you can retry on.

I saw this on a document classification pipeline that pinned every request to the low effort tier to save money. Most documents classified fine, because most classification is genuinely easy and the low tier was never the bottleneck for them. The documents that needed a second pass through a longer chain of reasoning, ambiguous ones with conflicting signals, got a confident, wrong label instead of a slow, right one, and nothing in the pipeline flagged the difference between the two cases. The average accuracy number looked fine for weeks before anyone noticed the pattern in the errors.

The override I actually built

What I run now picks a default effort level from the request shape. A short classification prompt gets low. A long document summary gets medium. An open-ended planning task gets high. Then the caller can override that default explicitly on the call, an effort field the request can set alongside the task itself, which takes precedence over whatever the heuristic would have guessed. The router’s guess is a starting point, not a verdict. It will be wrong sometimes, because “is this task actually hard” is a judgment the caller usually has more context for than a heuristic looking at token count and a task label.

Smart Inference routes across providers with this same two-axis problem in mind, and it tracks the actual cost of each request rather than a rate-card estimate, so the effort level you picked shows up as a real number afterward instead of a guess. It does not know your task is hard before you tell it. Nothing does. The override field is still yours to set.