Build Your Own AI Eval Dataset: From Question Selection to Scoring

To objectively know whether a model is good enough for your business, you need your own evaluation set. Assorted questions scraped from anywhere don’t tell you much — building a reusable dataset from real scenarios is the real foundation of evaluation.

Why you can’t rely only on public leaderboards

Public leaderboard questions are set by others, weighted toward general ability, and often have little to do with the work you do every day. The top model failing in your scenario isn’t surprising at all. To truly know which model suits you, you have to test your work with your questions. A self-built set pulls evaluation from someone else’s standards back to your own business — only then are the conclusions usable.

Where questions come from

The best source is real life: customer-service logs, user questions, historical tickets — pick out the high-frequency and the error-prone ones. Real questions measure real ability, not pretty but detached samples. Also cover two types: typical questions for daily level, and edge questions (vague, long, tricky) for the floor. If you only test typical ones, the model looks shaky on odd questions and you’d never know.

You need both typical and edge

Typical questions mostly come from high-frequency requests and represent the everyday; edge questions come from anomalies and hard cases and represent risk. With both sets in, the model’s strengths and weaknesses are both visible. Edge questions are especially valuable — they’re often the source of incidents. Treat edges as a focus in collection, so evaluation isn’t just “fine in normal times” but “steady when it gets hard”.

How to define reference answers

Objective questions (is the format right, is the field present) are easy to judge; subjective ones (logic, writing style, strategy) need clear scoring criteria. Keep answers and criteria separate so scoring isn’t arbitrary. For open questions, give a reference answer plus score points rather than a single solution. Setting the standard gives scoring a ruler; with a unified ruler, scores from different people become comparable.

The scoring rubric

A rubric is the detailed scoring rule: what performance each point corresponds to, where the boundaries are. Written clearly, human scoring is consistent and automated scoring has a basis. Avoid giving points by feel. Break good performance into observable dimensions (accuracy, completeness, compliance, tone) and score each. With clear dimensions, the score stops being mysticism.

Combine human and automation

Whatever can be auto-judged (format, keywords, whether a field is present) runs in a script — time-saving; what needs quality judgment (style, logic) gets human review on a sample — fidelity. Combining the two is both fast and stable. Human review must also follow the rubric, not impression. Periodically have two people score back-to-back and compute agreement; when disagreement is large, extend the rubric.

Prevent data leakage

If the evaluation set leaks into training data, scores inflate and you fool yourself. Isolate strictly: evaluation questions never enter training, never go public, and get checked for duplicates regularly. Isolation is the bottom line of honest evaluation. Also guard against the model memorizing questions: use dynamic generation or refresh with new questions periodically, so the model scores on ability rather than memory.

Scale and cost

The set doesn’t need to be huge — start small and grow: a few dozen to a hundred real questions gets you going, and running the process through matters more than building a big set. Scale grows with demand; don’t stall for completeness. Maintenance has a cost too: labeling, review, and updates all spend money. Treat the set as an asset to run, with investment matching its contribution to decisions — not an endless pile of questions.

Version management

The evaluation set itself needs versioning: what questions were added, what standards changed — keep records. Scores across versions aren’t directly comparable; compare within the same version to be fair. After a model upgrade, running the same version’s set is what makes score rises or falls meaningful. Versioning makes evaluation reproducible and traceable — the key to turning it from one-off into routine.

Connecting to model selection

A self-built set directly serves selection: run candidate models on the same set, compare across dimensions, and who suits which task is clear at a glance. It also guides prompts and fine-tuning: whichever question type loses points, adjust the prompt or add data there. An evaluation set isn’t just scoring — it’s a compass for optimization.

Three common pitfalls

Pitfall one: questions detached from reality, testing equals not testing. Pitfall two: unclear rubric, human scores incomparable. Pitfall three: evaluation set mixed into training, scores inflated. All three are resolved by real selection, a clear rubric, and strict isolation.

How to craft edge questions

Edge questions can be reverse-engineered from failure cases: things the model got wrong, things users complained about — distill them into questions. Failure is the best question-writer; the weakness it exposes is exactly what should be tested. You can also construct deliberately: long inputs, ambiguity, prompts designed to mislead, cross-language. Systematize the situations that easily crash into questions, so evaluation pokes at weaknesses instead of only testing easy giveaways.

Relation to online monitoring

The offline evaluation set decides whether something should go live; online monitoring decides whether it stays stable after launch. The two complement each other: offline gates, online guards the daily. New error types found online flow back in as new questions, so the set evolves with reality. Only when offline and online are connected does evaluation stay alive, instead of being built once and abandoned.

Measuring whether the set itself is good

Look at two things: whether the questions cover real high/low frequency and edges, and whether the scores can distinguish good from bad models. If it can distinguish and fits reality, the set stands. Then check whether it drove decisions: did selection, prompt changes, and data additions actually reference it? An evaluation set’s value isn’t in looking good — it’s in being used, being trusted, and guiding action.

Small teams start light

Don’t build a platform up front. A spreadsheet plus a few dozen real questions is enough to start: one row per question with a standard and a rubric, score by human or script after running the model, and manage versions with the spreadsheet’s history. Lightweight is what makes it sustainable. Script it only after the process runs smoothly and the question count grows. A small team’s advantage is speed — getting self-built evaluation moving at minimum cost is more likely to survive than building a big system.