Model Routing: Split the Expensive Work from the Cheap Work

Not every question needs to be thrown at the most expensive model. Model routing sends simple tasks to a cheap small model and hard ones to a big model, cutting overall cost and latency by a large margin without dropping quality.

Why you can’t use a big model for everything

Big models are expensive and slow, but many everyday requests — classification, extraction, rewriting — are handled fine by a small model. Using a big model for all of them is like mincing scallions with a cleaver; money and latency both burn on simple work. Worse is concurrency: with everything on big models, peak compute can’t hold up, forcing rate limits or paying to scale up. Route the simple work elsewhere and both throughput and the cost curve get much healthier.

The core idea of routing

The routing layer sits between requests and a set of models: first it judges whether this job is hard, then decides which model to hand it to. Simple goes to a small model, complex to a big one, boundary cases to the middle tier. Think of courier services: letters go ordinary mail, urgent packages go express — not everything flies air freight. Routing puts every bit of compute where it counts; it’s tiered service in engineering.

How to judge difficulty

The simplest way is rules: by input length, whether reasoning is needed, whether it involves sensitivity — write a few checks to split traffic. Rules are easy to understand and tune but can’t cover the fuzzy cases that look simple but hide traps. The advanced step is using a small model purely for difficulty classification: it quickly and accurately scores each request’s difficulty, then routes on that. The small classifier itself is cheap, and in exchange the overall routing quality improves.

How to set the tiers

A common setup has three tiers: a small model handles high-frequency simple work like summaries and classification, a middle model handles medium work like rewriting and extraction, and a big model handles the rare and hard — complex reasoning, long-text synthesis. More tiers aren’t necessarily better; three is enough and manageable. Choosing tiers must match the real task distribution: if 80% of your traffic is simple work, the small-model tier needs to be strong and fast enough; hard work is rare, so the big-model tier can be pricier and still hold. Design tiers around the traffic profile.

How to do the cost accounting

Routing’s benefit shows in two accounts: the unit price saved (simple work no longer burns a big model) and the latency saved (small models are fast). Pull out the bills and response times before and after splitting, and the value is obvious at a glance. Also count the routing layer’s own overhead: the classifier and scheduling logic consume resources too, but usually far less than what it saves. With the ledger clear, routing goes from sounding nice to actually saving money.

How to keep quality from dropping

Routing’s biggest fear is misclassification: a simple job sent to a big model is waste; a hard job sent to a small model drops quality. So you need fallback: when a small model isn’t sure, escalate to the big model rather than forcing through an error. Also monitor each tier’s accuracy; if the small-model tier errors often, the threshold is too low or its capability is insufficient — either tune the threshold or swap in a stronger tier. Quality is watched, not set once and forgotten.

Working with caching

High-frequency repeated questions like common Q&A can hit a cache first, and on a hit return directly — saving even the routing and model calls. Caching is the second cost-reduction gate after routing, and the two stack for a stronger effect. Caching needs invalidation care: answers change with price and policy, and an expired cache returning a stale answer does harm. Give the cache a time-to-live or version, so routing doesn’t hand back stale goods. Cost reduction and fidelity must be managed together.

Dynamic scaling

If a certain tier stays under high load, traffic is biased toward that tier — add instances to that tier or temporarily upgrade it; tiers idle long-term get scaled down to save money. Routing lets each tier scale independently, finer than scaling the whole thing. Tie scaling to the cost curve: adding machines to the small-model tier is cheap, so expand boldly; the big-model tier is expensive, so route traffic down to the minimum before expanding. Resources follow real difficulty, and nothing is wasted.

Its relationship with agents

Agents also route internally: planning uses a big model, small execution steps use a small model, retrieval judgment uses a middle model. Routing thinking extends from external service to internal orchestration, and it’s the core idea behind agent cost savings. Understand this layer and you won’t call the most expensive model for every agent action — you call by action difficulty. An agent’s intelligence doesn’t come from piling on big models everywhere, but from routing that allocates brainpower well.

Three common pitfalls

Pit one: routing rules too coarse, so fuzzy jobs all get misrouted. Pit two: the small-model tier’s quality isn’t watched, quietly degrading. Pit three: counting only model costs, not the routing layer’s own overhead, so the ledger is crooked. All three are resolved by a more accurate classifier, watching small-tier quality, and a clear total ledger. Routing isn’t magic once you wire a judgment; tuned well it’s a cost-cutting weapon, tuned wrong it’s both pricier and worse.

A small-step path to rollout

Don’t touch the big-model tier first; only divert the most obvious simple work — classification, format conversion — to the small model, verify it saves money without dropping quality, then gradually bring more types into routing. Small scope means light pit-stepping. Build a routing dashboard alongside: real-time visibility into each tier’s traffic, cost, and accuracy. Once the dashboard is up, which tier is overloaded, which is wasteful, and which is dropping quality all show themselves, so routing keeps improving instead of being set once and abandoned.

Measuring whether routing is worth it

Watch two things: how much overall per-request cost dropped, and how much the model tiers’ average latency dropped. Both dropping together means routing actually works; if only one drops, the other end may be eating the gains. Then check whether the quality red line holds: hard-work accuracy didn’t drop, and simple work isn’t wrong from misrouting. Cost reduction at the expense of quality isn’t good routing.