How many examples do you need in an LLM eval set?

Around 100 cases is a workable starting point, and 30 is not enough to make decisions on. At 100 cases with a 90 percent pass rate, the 95 percent confidence interval is roughly plus or minus 6 percentage points; at 30 cases it widens to roughly plus or minus 11, so a 30-case suite cannot distinguish an 85 percent system from a 92 percent one. Running two variants on the same inputs and comparing them pairwise cancels most of the per-question difficulty variance, which lets a paired 100-case suite detect much smaller differences than 100 independently scored runs. Build the cases from real production traces rather than invented examples, and report each category separately instead of averaging everything into one number.

Is LLM-as-a-judge reliable enough to gate a deployment?

It is reliable enough to be useful and not reliable enough to trust unchecked. The 2023 MT-Bench paper by Zheng et al. found GPT-4 agreeing with human experts more than 80 percent of the time, roughly the rate at which human raters agree with each other, while documenting position, verbosity and self-enhancement biases in the same judges. A later study of 15 judges across more than 150,000 evaluation instances found that position bias is systematic rather than random and is strongest when the two candidate answers are close in quality, which is exactly when the decision matters most. Practical mitigations are to swap the order and run each comparison twice, count only self-consistent verdicts, never let a model judge outputs from its own family, and score the judge itself against a hand-labelled sample before trusting it.

How do you detect when a model provider silently changes a model?

Run your eval suite on a schedule against the exact model endpoint you call in production, not only when someone commits code, and alert when any category moves further than your measured noise floor. This is a documented failure mode: a 2023 study by Chen, Zaharia and Zou found GPT-4's accuracy at identifying prime numbers fell from 97.6 percent in March 2023 to 2.4 percent in June 2023 under the same product name. Pinning a dated model snapshot gives you stable behaviour plus a known retirement date, and OpenAI announced on 11 June 2026 that six GPT-5 and o3 snapshots leave its API on 11 December 2026. A floating alias gives you neither a deadline nor any notification when the behaviour underneath changes.

LLM Evals Are Regression Tests for AI Systems

Most teams I meet can describe what their AI system does. Very few can tell me whether last Tuesday's prompt change made it better or worse. That gap is the entire problem. LLM evals close it, and almost nothing else does. A team that cannot measure its system cannot improve it, so every change becomes a matter of opinion, settled by whoever argues hardest in the review.

Deterministic software has regression tests. Probabilistic software has evals. Same job, different maths. You are not asserting that a function returns 42. You are asserting that a distribution of outputs stays inside a boundary you defined, on inputs you chose on purpose. That is harder. It is also the highest-leverage thing an under-resourced team can build, because it is the only artefact that makes every future decision cheaper.

A hundred rows is enough to start, and thirty is not

The first objection to building an eval set is always the same. We do not have labelled data. You need less than you think. You also need more than the twenty examples somebody pasted into a spreadsheet last quarter.

Do the arithmetic. If your suite has 100 cases and your system passes 90 of them, the 95 percent confidence interval on that rate is roughly plus or minus 6 percentage points. At 30 cases it is plus or minus 11. A 30-case suite cannot distinguish an 85 percent system from a 92 percent system, which means it cannot tell you whether the change you just shipped helped. Evan Miller's November 2024 paper for Anthropic works this out properly and adds the parts people skip: paired tests when you run two variants on the same questions, cluster-robust standard errors when your questions arrive in related groups, and power analysis before you build the set rather than after you stop trusting it. Paired testing is the cheapest win on that list. Comparing two variants on identical inputs cancels most of the per-question difficulty variance, so a paired set of 100 detects differences that would need several hundred cases if you scored the two runs independently.

Build the set from your own traces, not from your imagination. Read production logs until you stop discovering new failure modes, which in practice lands somewhere between 30 and 100 traces for a young system. Group what you find. Then write cases for each group and keep the groups separate in the report. One averaged number across easy cases, known failure modes and adversarial inputs hides exactly the movement you care about.

Every change passes an offline gate before a small share of live traffic sees it, and the failures that show up in production traces are minimised into new cases so the same regression cannot ship twice.

A suite that never fails is not a safety net. It is decoration.

The judge has a thumb on the scale

For anything with an open-ended output you will end up using a model to grade a model. This works better than sceptics expect and worse than the marketing suggests. Zheng et al., the 2023 NeurIPS paper that introduced MT-Bench and Chatbot Arena, found GPT-4 as a judge agreeing with human experts more than 80 percent of the time, which is about how often the human raters agreed with each other. The same paper named the failure modes: position bias, verbosity bias and self-enhancement bias.

Position bias is the one that bites first. A 2024 study, later published at IJCNLP-AACL 2025, ran 15 LLM judges over roughly 40 solution-generating models for more than 150,000 evaluation instances across MTBench and DevBench. Position bias was not random noise. It varied by judge and by task, was only weakly related to the length of the prompt components and was strongly related to the quality gap between the two candidate answers. Read that last clause again. The judge is least stable precisely when the two answers are close, which is precisely when you are trying to make a decision.

So do the boring things. Run every pairwise comparison twice with the positions swapped, count only the cases where the judge agrees with itself and treat the flips as ties. Never let a model judge its own family. Label 50 to 100 cases by hand and score your judge against them before you let it score anything else. Then re-score the judge whenever you change its prompt, because the judge prompt is production code and almost nobody treats it that way.

Pairwise questions beat five-point scores

Asking a judge to rate an answer from 1 to 5 produces a number that drifts with the prompt, the model version and the phase of the moon. Asking which of two answers is better produces something far more stable, because the comparison is anchored to a concrete alternative. Public preference leaderboards fit those pairwise votes with a Bradley-Terry model rather than averaging absolute scores, for the same reason.

You do not need Elo for a product. You need a simple question. On these 140 cases, did the new version beat the old one, and where did it lose. Then wire in the metrics that need no judge at all. Does the generated SQL run, and does it return the expected row count. Are the extracted fields exactly equal to ground truth. Is the cited chunk actually present in the retrieved set. Did the tool call parse as valid JSON against its schema. These checks are cheap, deterministic and unambiguous, and they catch most of what breaks. Reach for a judge only where you genuinely cannot write an assertion.

A high score on a public benchmark tells you a model is generally competent. It tells you nothing about whether it reads your invoices.

What I put in an eval set that no benchmark contains

I read Chinese model releases the morning they publish, before the English write-up exists. The pattern repeats. The technical report describes how the eval set was constructed, and the English summary keeps the headline number and drops that paragraph. The number survives translation. The conditions do not. A vision model reporting high accuracy on clean scanned documents is making no claim at all about a product label photographed at an angle under a flickering fluorescent tube in a workshop, which is the input the deployed system will actually receive.

Every set I build carries a fixed block of Nigerian names, Lagos street addresses and Naira amounts. Not for representation. Because models that score well on English benchmarks still split Chukwuemeka across token boundaries and hand back a mangled entity, still read a figure written as 2.5m Naira as a dollar amount, still collapse Ikeja and Ikoyi into each other somewhere in a retrieval step. No leaderboard contains these cases. My users are made of them. The general lesson is that the twenty inputs your system sees most often, in the exact form your users type them, are worth more than any public benchmark you could bolt on.

Offline gates, online traces and the loop between them

Offline evaluation runs a fixed set on every change, in CI, blocking the merge when the numbers move the wrong way. Fast, cheap, repeatable and blind to everything you did not anticipate. Online evaluation samples real production traffic, scores it and tells you about the inputs nobody imagined. You need both, and the loop between them is where the value sits. A production failure gets read, gets minimised into a case and joins the offline set, so that failure can never ship twice.

The tooling consolidated fast. ClickHouse acquired Langfuse in January 2026 alongside a $400 million Series D at a $15 billion valuation, putting the most widely deployed open-source LLM tracing tool inside a database company. Braintrust raised an $80 million Series B led by ICONIQ at an $800 million post-money in February 2026. OpenAI announced its acquisition of Promptfoo on 9 March 2026, a tool with more than 350,000 developers and 130,000 monthly active users, largely for its red-teaming work, with a stated commitment to keep the open-source suite alive. Arize Phoenix went the other direction and stayed self-hostable, adding native support for OpenTelemetry's GenAI semantic conventions so traces are portable by construction.

One detail from the same period is worth more than all of that funding news. On 3 June 2026 OpenAI announced that its own Evals platform, dashboard and API, shuts down on 30 November 2026. The place where you look at your evals can itself be deprecated. Your dataset and your scoring functions belong in your repository, version controlled next to the code they test. The platform renders them. It does not own them.

The provider moves, and nobody tells you

Chen, Zaharia and Zou compared the March 2023 and June 2023 versions of GPT-4 on identical tasks. Accuracy at identifying prime numbers went from 97.6 percent to 2.4 percent. Same product name, same endpoint, same code calling it. Whatever you make of the benchmark design, the operational lesson survived intact. The thing behind the API is not a fixed object.

Three years on, the mechanics are clearer and no more comfortable. OpenAI announced on 11 June 2026 that six snapshots, including gpt-5-2025-08-07 and o3-2025-04-16, leave the API on 11 December 2026. Its published policy is at least six months of notice for GA models, at least three months for chat, codex and deep research variants, and as little as two weeks for anything with preview in the name. Pin a dated snapshot and you get stable behaviour plus a hard expiry date. Use a floating alias and you get no expiry date and no notification when the behaviour underneath shifts.

Both choices need the same defence. Run the suite on a schedule against the model you are actually calling in production, not only when a human pushes a commit. A drift alarm is not exotic infrastructure. It is a diff on numbers you already compute, running at 3am, posting into a channel when a category moves further than your measured noise floor.

The case against evals is stronger than most teams admit

Three objections, all correct. First, suites rot. A golden set written in March describes March's product, and by September half the cases test behaviour nobody ships any more while none of them cover the feature that went live in July. Second, evals get gamed by the very optimisation they are meant to guide. If your judge quietly rewards structured answers, your prompts learn to emit headings, and the score climbs while quality goes sideways. Goodhart's law with a CI badge. Third, false confidence. A green suite is the most dangerous artefact in the building, because it silently converts "we did not test that" into "we tested it and it passed".

None of this argues for having no evals. It argues for evals with a maintenance budget and a written list of known blind spots. Concretely: hold out a slice you never optimise against and open it only once a quarter. Rotate fresh production traces in every sprint so the set ages with the product. Score the judge against human labels on a schedule, not just the system against the judge. Delete cases that have passed unchanged for six months, because a case that has never failed has stopped carrying information. And put the failure rate of the suite itself on the dashboard. A suite that never fails is not a safety net. It is decoration.

The alternative to a flawed eval suite is rarely a better one. It is vibes, plus whoever in the room holds the strongest opinion and the most seniority.

The switching cost is the real number

Here is the organisational argument, and it is the one that gets budget approved. A team with a working suite can point its system at a different model on a Thursday afternoon and have a defensible answer by Friday morning. Three categories regressed, here they are, here is what it costs to fix them. A team without one holds a meeting instead, then another, and the model they integrated eighteen months ago stays where it is because nobody can say what would break.

That is the true cost of skipping evals in 2026, with open-weight models from Chinese labs shipping on a cadence that makes an annual vendor review look quaint. The integration work to swap a model is usually a day. The confidence gap is a quarter.

The part that stays uncomfortable is that your suite is itself a model of what you think matters, written by the same people who wrote the system, carrying the same blind spots in the same places. It will be confidently green about the failure you have not imagined yet. Somebody will find that failure. The only thing you control is whether it is you, on a Tuesday, from a sampled trace, or a customer, on a Friday, in public.

Tools referenced

Langfuse, reviewed here: Langfuse review.

Promptfoo, reviewed here: Promptfoo review.

Braintrust, reviewed here: Braintrust review.

Arize Phoenix, reviewed here: Arize Phoenix review.

Weights & Biases, reviewed here: Weights & Biases review.

Label Studio, reviewed here: Label Studio review.

Sources

Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (NeurIPS 2023): https://arxiv.org/abs/2306.05685

Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge: https://arxiv.org/abs/2406.07791

Evan Miller, Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations (Anthropic, Nov 2024): https://arxiv.org/abs/2411.00640

Chen, Zaharia and Zou, How Is ChatGPT's Behavior Changing over Time?: https://arxiv.org/abs/2307.09009

OpenAI API Deprecations (GPT-5/o3 snapshot retirement, Evals platform shutdown, notice policy): https://developers.openai.com/api/docs/deprecations

ClickHouse acquires Langfuse, open-source LLM observability (January 2026): https://clickhouse.com/blog/clickhouse-acquires-langfuse-open-source-llm-observability

Promptfoo is joining OpenAI (9 March 2026): https://www.promptfoo.dev/blog/promptfoo-joining-openai/

Braintrust's Series B: building the infrastructure for production AI (February 2026): https://www.braintrust.dev/blog/announcing-series-b

LLM Evals Are the Regression Tests for Probabilistic Software