Model Routing 101 - Fallback, Cost, and Latency

Model Routing 101 - Fallback, Cost, and Latency

This is what Smart Inference does, and building it yourself is viable if your scale is small. Here is roughly how it works underneath, in case you want to build the simple version yourself before deciding whether you need the full thing.

The problem in one sentence

You have more than one place a request could go, more than one provider, more than one model, and you need to pick one per request based on some mix of cost, health, and latency, without hardcoding a single choice that goes stale the moment a provider has a bad day.

Step one: fallback chains

The simplest useful version is a fallback chain. Try provider A. If it fails or times out, try provider B. If that fails, try provider C.

f u n c t f r i o a o r i n s p t e e r r r x o o y c A u v : e l t i r r p c l e d e e t o P ( e s t n r r r p u ( t o e o r T i v q i n n i n i u n s m u d e e r e e e s [ e o r t A = s u s ) , p t F : p o , a B r n i , o s P l v e r e C i o d ] d v E : e i r r d r . e o c r r a E l r l r ( o r r e ) q : u e s t , t i m e o u t = 5 s )

This alone is worth building even if you do nothing else in this post. A single hardcoded provider with no fallback means one outage becomes your outage. A three-line fallback chain removes most of that risk for very little code.

Step two: health scoring instead of a fixed order

A static fallback chain has a problem: it always tries A first, even if A has been failing for the last hour. You end up eating A’s timeout on every single request before falling through to B, which is slow and wasteful.

Track a rolling health score per provider instead of a fixed order.

s f t u r n u c c e p i t h i r t r 9 s i e f e r 5 _ o a t P o _ h n l h u r r l e t e h r o _ a a p h a e n v r t l i y l a i a e t c t l s d t n h k = h t o e e c y _ y h r r : y : p [ y t H _ r p i _ e f m b o s = b a l s o v f y l o : o i o e p ( t a l d r m r h h t f e p o e : l r p t v a o ( y i l a p i : d t t r n e h o r y v p s , i r # # # d o k e v # e r r e r i y o o r s d d = l l r ) e e h l l o : r g e i i r s r a n n _ a l g g r i d t , , a f e h t _ l l e p g s a a . r c s s < h a o t t e c r t a e e N N h l f ) r t u [ r r e h l 0 e e s . l ] q q h i y u u o s , e e l _ s s d h t t t e r s s a y l t a h n y y ] w a y

health_score can be as simple as error_rate * weight_a + p95_latency_ms * weight_b. The point is that the ordering itself becomes dynamic, a provider having a bad afternoon drops down the list without anyone touching a config file.

Step three: cost-weighted routing

Once you have health scoring, layer cost on top. Among the providers passing the health bar, pick the cheapest, not the first.

f u n c t h i # r r i e f a e o a c n t n l h o k u t e h s e r p h a e t d n i y l a c t l e = r k = h t s a _ y h t s n p [ y i o k r p i m r e o s = a t d v f t _ [ i o e p e b 0 d r m r d y ] e p o ( r p t v f h ( y i r e p i : d o a r n e m l o r t v p s r h i r e y d o q , e v u r i e k s d s e , e t y r = r s s l e i a q i z m u f e b e d s p a a t . n ) h d p : e : a p l r e t o s h v t . i i i d m s e a _ r t h ' e e s d a _ l r c t a o h t s y e t ] ( p , r e q u e s t ) )

The trap here is estimating cost from a stale rate card. Providers change prices, sometimes hourly on the GPU side. If your estimated_cost function reads from a config file you update manually, you are back to the spreadsheet problem, a static number that goes wrong quietly. Pull current pricing from the provider’s API where possible, or measure actual billed cost from recent requests and use that as your estimate.

Step four: measuring what actually happened, not just what you predicted

This is the part that separates a routing prototype from something you can trust with real spend. Every response should get logged with what it actually cost, not what you expected it to cost.

f u n c t p s r l a r r i r t e a c e e o o a s t t c t n v r p e u o u i t o n a r r r d n c l d n o e = s y _ _ u r e c m r t n = o e e e = o = s t s ( w n t r p r p ( p o i o e i ) r w = c n q c o ( ( s u k v ) p p e e _ i r r s p d - o o t r e v v ) o r s i i : v . t d d i c a e e d a r r r e l t . , r l c ( ( o l p r m a r e p t o q u e v u t n i e e c d s _ y e t a , r ) c s t a , u c a t r l u e _ a q c l u o _ e s c s t o t ( s ) r t e , s p s o u n c s c e e ) s s = t r u e )

Feed record_metric back into the health score and cost estimate for the next request. This closes the loop, the system learns from what actually happened instead of running on a fixed assumption forever.

When to build this yourself vs. not

If you have two providers and modest volume, the fallback chain from step one, maybe fifty lines of code, covers most of the real risk. Build it. You do not need a platform for this.

Where it gets genuinely annoying to maintain yourself: once you have more than a couple of providers, need per-GPU rather than per-VM cost tracking for self-hosted inference, or want the health scoring and cost data to survive a redeploy instead of resetting every time your process restarts. That is the point where Smart Inference starts saving real time, it is this same logic, already built, already tracking actual per-request cost instead of estimated, with the health and pricing data persisted instead of living in a process that restarts.

Either way, understand the mechanism before you reach for a tool that does it for you. It is not complicated. It is just tedious to get right and easy to let drift once it is working well enough to ignore.