Model Distillation: Cut 90% of Cost with a Small Model

Not every task needs a big model thrown at it. Distill a big model’s capability into a small one and you can cut inference cost by up to 90% with lower latency — and many business scenarios work perfectly well with that.

What distillation is actually doing

Ordinary training only teaches the small model “what the standard answer is.” Distillation additionally teaches it “the big model’s confidence distribution across options” — say the big model judges A at 90% and B at 8%. These soft labels carry information about “why,” so the small model learns to imitate the teacher’s judgment logic. An intuitive analogy: memorizing the answer is memorizing the conclusion; watching the teacher work through a problem is learning the reasoning. Distillation hands over the reasoning, so at the same data volume a distilled small model often generalizes better than one trained purely on hard labels — especially on borderline samples.

When to use a small model

High-frequency, simple, latency-sensitive tasks fit best: classification, tagging, simple extraction, intent recognition. A big model can do these, but it’s using a sledgehammer — cost and latency don’t pay off. A small model is fast and cheap on them. Conversely, open-ended generation, complex reasoning, and tasks needing broad knowledge make a small model show its limits. The judgment is one line: does this task require “thinking many steps”? If it thinks a lot, keep the big model; if it thinks little, push it to a small model.

How to calculate the cost ledger

In scenarios billed per token, routing 60-70% of the simple requests to a small model cuts the bill substantially. A small model’s per-inference cost is often a tenth of a big model’s or less, and the more volume, the more you save. There’s also a latency ledger: small models have fewer parameters and often respond several times faster, which noticeably improves real-time interactions like instant suggestions in an input box. Cost plus experience, the two ledgers together make distillation’s return very real.

Distillation isn’t just about accuracy

Staring only at accuracy lands you in a pit: a small model can match the teacher on average accuracy while failing badly on a few long-tail classes. The validation set must cover the business’s real distribution, and metrics should be viewed per scenario layer instead of being fooled by one total number. Also look at error types. In classification, misjudging an “important customer” as “ordinary” costs far more than two ordinary classes swapping. When distilling, weight the important classes so the small model first protects the expensive-to-get-wrong part.

The extra benefit of local deployment

A small model can run on a laptop or even a phone, keeping data on the device. For privacy-sensitive scenarios like local document processing, this is far more compliant than calling a cloud big model, and it doesn’t depend on the network. Localization also brings stability: no dependence on external service availability, and it runs offline. For industrial sites and intranet environments, this is a hard advantage of small over big models, regardless of cost.

Common approaches to distillation

Practice offers two routes. One is using the teacher model to label the training set with soft labels, then training the small model on them. The other is having the small model imitate the teacher’s hidden states or intermediate outputs directly — learns faster but is more complex to implement. For starters, take the soft-label route: run existing labeled data through the big model to get probability distributions, then train the small model on this augmented data. Small changes, clear gains — the most cost-effective entry point.

How it pairs with a big model

The steady architecture is “small model first, big model as fallback”: the small model handles it first, high confidence means it outputs directly, low confidence escalates to the big model. That keeps cost low without losing accuracy on complex cases. The routing itself needs design too: a confidence threshold or a simple classifier decides who takes it. Tune the routing well and you find the right cost-quality balance instead of going all-small or all-big.

Three easy pits

Pit one: distillation data and the teacher’s training data have different distributions, and the small model learns skewed. Pit two: optimizing only total accuracy, ignoring long-tail and important-class errors. Pit three: chasing the smallest blindly, then key scenarios escalate to the big model constantly and you save nothing. All three are resolved by “validate with real business data, view metrics per scenario layer, route as a fallback.” Distillation is an engineering job, not train-and-done.

Selection checklist

Ask four questions before starting: does the task need multi-step reasoning? How much volume? How strict are the latency requirements? Can the data leave the domain? Answering those four makes it fairly clear whether a small model fits, without blindly jumping into distillation. If the answers lean “simple, high-volume, fast, local,” commit to a small model with distillation; otherwise keep the budget on a big model — don’t save for the sake of saving.

Launch and monitoring

After a small model ships, monitor two kinds of drift: data distribution changing and accuracy slipping, and the teacher upgrading while the student hasn’t kept up. Build a pipeline of “teacher re-labels periodically, student retrains periodically.” Also keep rollback: if a new distilled version’s metrics degrade, you can one-click revert to the old version. Models age; monitoring and rollback are the floor for distillation to run stably long-term.

A real case of distillation results

A common landing is customer-service intent recognition: originally a big model judged each one, costly and slow; after distilling a small model, nine-tenths of the simple inquiries classify locally in milliseconds, and only the few complex cases escalate to the big model. Another scenario is content-moderation pre-screening: the small model blocks obviously violating content first, and the big model only rechecks borderline samples. Cost drops while people and the big model both spend their effort where it matters — overall throughput is actually higher.