Every model vendor releases leaderboards and flaunts scores, but topping the board doesn’t mean the model works best in your business. Evaluating LLMs is something you have to bring down to your own scenario instead of staring at other people’s rankings. Make evaluation a job that fits your business, and the conclusions actually become usable.
The limits of public leaderboards
Leaderboards mostly test general abilities: problem-solving, common sense, coding. But your business is things like “answering after-sales questions in Chinese” and “writing titles for products,” which are far from leaderboard scenarios. A top-scoring model on the board can still trip on your long-tail questions. So leaderboards only work as a first filter, not a conclusion — the real decision rests on your own scenario’s performance. Worse, the board questions are fixed, vendors may optimize toward them, and the scores get inflated — a world away from your production environment.
Build your own evaluation set
Pull a few dozen to a few hundred real input-output pairs from your business logs, pair them with expected answers, and make them your own evaluation set. This sits closer to the need than any public leaderboard. Weight the set by your real traffic distribution: if 80% of your traffic lands on one type of question, that type is the evaluation focus, not a balanced all-subject score. A common misjudgment is deciding a model just because it has the “highest average score,” then discovering after launch that it’s precisely weak on your highest-frequency question type.
Which metrics keep you from going astray
Accuracy checks whether answers are right, especially for factual questions. Stability checks how much results fluctuate when the same type of question is asked repeatedly. Format adherence checks that when you ask for a table you get a table, and a list gets a list — no freelancing. Anti-fabrication checks whether it makes things up hard when material is missing. Fabricating without material is the most dangerous; external output often gets stuck on format adherence. Look at these metrics together, and no single high score can mislead you.
Figure: key takeaways of LLM evaluation
| Metrics to watch | Explanation | How to use |
|---|---|---|
| Accuracy | Whether answers are right, especially factual ones | Log every wrong factual answer; don’t look at the overall only |
| Stability | How results fluctuate across repeated same-type questions | Big fluctuation means untrustworthy; use with caution |
| Format adherence | Table when asked for table, list when asked for list | External output often gets stuck here |
| Anti-fabrication | Whether it makes things up when material is missing | Fabricating without material is the most dangerous |
| Cost and speed | Per-call price and response time | Count routing combos together |
Retest regularly and prevent leakage
Models update, and what works well today may change in the next version; new models keep coming out. Make evaluation a scheduled job: every quarter, run the same evaluation set against candidates and decide whether to switch or mix. The evaluation set itself is an asset that grows more valuable as it accumulates. If the evaluation set heavily overlaps training data, scores get inflated — make sure evaluation uses questions the model has never seen, ideally from real business logs. A leaked evaluation only makes you misjudge, and you discover the problem after launch. Treat the evaluation set as something maintained long-term, like code.
Total cost of ownership
The most expensive model isn’t necessarily the best value. Often a routing combo — “strong model handles the hard, small model handles the easy” — wins on both overall quality and cost. Fold price into evaluation and what you get isn’t “who’s strongest” but “who fits best within my budget.” Human judgment is also irreplaceable, especially for subjective items like “whether the answer is appropriate” and “whether the tone is right.” Regularly sampling and recording your own sense of it is the most expensive and most valuable part of evaluation — don’t hand it entirely to scores.
Human judgment is a must
No matter how complete the automatic metrics are, they can’t replace a human looking. Especially for subjective items like “whether the answer fits” and “whether the tone is right,” regularly sample and note your feel for it — that’s the priciest and most valuable part of evaluation. Don’t hand it all to scores; sample even high-scoring cases, in case the model learned to “chase metrics” rather than truly answer. A suggestion: sample ten to twenty cases every week, record how your sense shifts, feed human feedback back into the set, and the next round of evaluation gets more accurate.
A concrete evaluation example
Take an after-sales scenario: pull a hundred real customer complaint Q&As from the ticketing system, have each candidate model answer them one by one, mark right or wrong with the reason; then pull twenty long-tail odd questions purely to watch anti-fabrication — whether it makes things up when material is missing. This set is small in volume but close to the business, showing at a glance which model fits you better than a public leaderboard. Retest the same hundred questions every quarter and you can directly see which version regressed.
Tier models instead of just ranking them
Don’t reduce the evaluation conclusion to a first-second-third ranking. More useful is tiering by task: route “high-frequency Q&A” to a small model, “complex reasoning and long-form writing” to a strong model, and use routing for the middle ground. Once tiered, you can directly compute the monthly bill and know which question types need a model upgrade and which need more rules. The value of evaluation isn’t the score on paper; it’s the tiering table and the cost ledger behind it. When a new model comes out, slot it into its tier and retest; ones that don’t fit stay where they are.
