AI Evals Are the Product Spec

Direct answer: An eval set is the most precise product specification an AI feature can have. Build it early from 20 to 50 real failures, grade outcomes rather than steps, measure consistency across repeated runs, and give product and design a direct say in what passes.

A traditional product spec describes what the software should do, and engineers translate it into code that either does it or does not. AI features break that chain. The model sits between the spec and the result, and it does not read the spec the way an engineer does. A sentence like “the assistant should summarize the customer’s issue accurately” gives a model no hard boundary to fail against. The only spec a model can be held to is a set of concrete cases with a clear verdict on each one. That set is an eval.

Many teams still treat evaluation as the testing phase that follows the build: write prompts, wire up tools, ship a beta, then figure out how to measure quality. Anthropic’s engineering guide to agent evals, published in January 2026, argues the opposite order. It recommends starting early with a small set, “20-50 simple tasks drawn from real failures,” and warns that evals get harder to build the longer a team waits. I would put it more bluntly. If you cannot write down the cases your AI feature must get right, you do not yet know what you are building.

A spec the model can actually fail

The value of an eval case is that it is falsifiable. “Handle refund requests politely” is not falsifiable. “A customer asks for a refund 40 days after purchase on a 30-day policy; the assistant declines, cites the policy and offers the exchange route” is. The second version forces decisions that the vague one hides: what the policy is, which exceptions exist, what tone counts as acceptable and what the fallback offer should be. Those are product decisions. When they are left implicit, they get made by whoever writes the prompt, usually late and under deadline.

Anthropic’s guide offers a useful test for a well-written task: two domain experts should reach the same pass or fail verdict on it independently. If they disagree, the case is ambiguous, and an ambiguous case measures the grader rather than the product. The same guide notes that a 0% pass rate across many trials usually points to a broken task rather than an incapable model. Both rules are good practice for any spec, AI or not.

Start from failures, not from a benchmark

Public benchmarks are useful for choosing a model. They say little about whether your feature works for your users. The best first eval cases come from your own transcripts: support tickets, beta sessions, internal dogfooding and the moments a user corrected the assistant. Hamel Husain’s widely cited essay “Your AI Product Needs Evals” makes the same point from the practitioner side. Remove the friction from reviewing traces, and accept that “you can never stop looking at data.”

A practical routine is to tag every reviewed failure with a short reason, then turn the most frequent reasons into eval cases. Over a few weeks this produces a test set that mirrors the real distribution of problems instead of the ones the team imagined during planning. It also gives the team a shared language for quality, which is often more valuable than the score itself.

Grade outcomes, and measure consistency

Agent evals introduce a trap that ordinary unit tests do not have. An agent can reach the right result through many paths, so a grader that checks an exact sequence of tool calls will fail good runs and teach the team to optimize for the wrong thing. Anthropic recommends grading the outcome the agent produced rather than every step it took, with deterministic, code-based graders where possible and model-based graders where judgement is needed.

The second trap is averaging. A model that succeeds 75% of the time on a task sounds acceptable until you ask how often it succeeds three times in a row. The guide distinguishes pass@k, at least one success in k attempts, from pass^k, success on all k attempts, and works through that example: a 75% per-trial rate gives roughly a 42% chance of three straight passes. For a customer-facing workflow that runs thousands of times a day, consistency is the metric users actually experience. Report both numbers, and set launch thresholds on the stricter one for high-stakes tasks.

Capability evals and regression evals are different contracts

Not every eval set should aim for a perfect score. Anthropic separates capability evals, which start with a low pass rate and give a team “a hill to climb,” from regression evals, which should sit near 100% and protect behavior that already works. The two answer different questions. Capability evals tell the roadmap where the product is improving. Regression evals tell a release manager whether a new prompt, a model upgrade or a tool change broke something customers rely on. When a capability case becomes reliable, it graduates into the regression suite.

This split also makes model upgrades less frightening. Teams that switch models on reputation alone are guessing. Teams with a regression suite can run the new model against it in an afternoon and see exactly which behaviors moved. That is the difference between an AI product with an engineering process and one with a vibes process.

Who should own the eval set

Because eval cases encode product decisions, they should not belong to engineering alone. Product managers decide which cases matter and what the thresholds are. Designers decide what a good response looks and sounds like, including how the system declines, asks for clarification or hands off to a person. Domain experts, such as support leads, lawyers or analysts, label the hard cases. Engineering builds the harness, keeps graders honest and ties the suite to the release pipeline.

Model-based graders need their own check. Husain recommends tracking how often an automated judge agrees with human labels before trusting it, and measuring precision and recall rather than raw agreement when the classes are imbalanced. A grader that silently drifts is a spec that silently changes.

A minimum eval practice before launch

Twenty to fifty cases taken from real transcripts, each with a verdict two experts agree on.
Balanced cases: situations where the behavior should happen and situations where it should not.
Outcome-based graders, deterministic where possible, with model graders checked against human labels.
pass^k reported alongside the average for any workflow a customer runs repeatedly.
A regression suite that runs on every prompt, model or tool change, with a named owner for failures.
A weekly transcript review that feeds new failures back into the set.

None of this requires a large team or a specialist platform. It requires treating the eval set as the place where the product is defined. Teams that do this ship slower in the first week and much faster every week after, because they can change prompts and models without guessing what they broke.

Sources

Author: Oleh Hebel. LinkedIn

Continue with our analysis of single-agent loops and subagent migration for a concrete architecture comparison to put those evals to work.

Evals are the product spec AI teams forget to write