pass^k vs pass@k agent reliability: your user only gets one run

An agent that works nine times out of ten looks finished in a demo. Put it in front of a thousand people and roughly a hundred of them get a broken answer, and not one of them re-runs the request twenty times to fish out the good attempt. That gap has a name in the literature. pass^k vs pass@k agent reliability is the whole of it: one metric asks whether a success exists somewhere in k attempts, the other asks whether every one of the k attempts worked.

The first is a search metric. The second is a service metric. Almost every agent number in circulation is the first, presented as though it were the second.

I have spent years shipping systems where the answer goes straight to a person, and the question that arrives in every review is the same. Give me one number. There is no honest single number. There are two, they are computed from identical data and they point in opposite directions.

pass^k vs pass@k agent reliability turns on one word: any, or every

pass@k reaches the field through the Codex paper, Mark Chen and 57 co-authors, July 2021, which credits the metric itself to Kulal and colleagues in 2019 and contributes the unbiased estimator everyone now uses. The framing is explicit: k samples are generated per problem, and a problem counts as solved if any sample passes the unit tests. The unbiased estimator they publish is pass@k = E[1 - C(n-c, k) / C(n, k)], where n is the number of samples actually generated, c is the number that were correct and k is the number you are pretending to draw. They also warn, in that same paper, that the tempting shortcut 1 - (1 - p)^k using an empirical pass@1 is biased.

pass^k, pronounced pass-hat-k after the caret, comes from tau-bench, Shunyu Yao, Noah Shinn, Pedram Razavi and Karthik Narasimhan, June 2024. Their definition is the chance that all k independent trials of a task succeed, averaged across tasks. The estimator is pass^k = E_task[C(c, k) / C(n, k)].

Set the two side by side and the symmetry is the point. Same denominator, C(n, k), the number of ways to draw k trials out of the n you ran. pass@k's numerator excludes the draws that are entirely failures. pass^k's numerator counts only the draws that are entirely successes. Draw k trials at random: pass@k is the probability that at least one succeeded, pass^k is the probability that none failed.

One word apart in English. Orders of magnitude apart in what they report.

The metric you pick decides who absorbs the variance. pass@k assigns it to a machine that can retry overnight. pass^k assigns it to a person who gets one attempt.

The two numbers separate before k reaches five

Take an agent with a per-attempt success probability p and work the closed forms. At p = 0.9, pass@5 is essentially 1.000 and pass^5 is 0.591. At p = 0.8, pass@5 is 0.9997 and pass^5 is 0.328. At p = 0.612, which is exactly the pass^1 tau-bench reports for GPT-4o on its retail domain, pass@5 is 0.991 and pass^5 is 0.086.

Read that last pair again. Same agent, same trials. One metric says the task is solved 99 times in 100. The other says the agent gets a clean run of five 9 times in 100.

tau-bench found this in its own results table: the best function-calling agent they tested, GPT-4o, averaged 48.2 percent task success across the two domains and its pass^8 on retail still fell below 25 percent. tau2-bench, June 2025, carried the metric forward and showed how far it moves across domains. GPT-4.1 posts pass^1 of 74 percent on retail, 56 percent on airline and 34 percent on telecom. The headline number is not a property of the model alone.

The same arithmetic is what quietly kills long-horizon agents, because a multi-step trajectory is pass^k in disguise. Twenty tool calls that each have to land is twenty independent chances to fall over. To reach 95 percent end to end across 20 steps you need 99.744 percent per step. Across 50 steps, 99.897 percent. Almost nobody measures per-step reliability to three nines, let alone holds it under load, which is why long agent runs fail far more often than their per-step evaluation suggests they should.

Nobody re-runs their refund request two hundred times

The hinge is not the formula. It is whether anything in the loop can tell a good attempt from a bad one.

When something can, pass@k is measuring a genuine capability. Brown and colleagues, in Large Language Monkeys, July 2024, took DeepSeek-Coder-V2-Instruct from 15.9 percent of SWE-bench Lite issues solved with one sample to 56 percent with 250 samples, past the 43 percent single-sample state of the art at the time. That is real, and pass@k is the right way to report it, because the unit tests do the picking. The same paper is careful about the other case: where no automatic verifier exists, the usual selection methods, majority voting and reward models, flatten out past a few hundred samples. Coverage keeps climbing. Your ability to cash it in does not.

Most agent work sits on the wrong side of that line. Rebooking a flight, issuing a refund, filing a change against a ticketing system: the first attempt has already written to a system of record. There is no oracle standing by to announce that attempt three was the good one, and no free undo. The person who asked is not going to ask nineteen more times and diff the answers.

So when the harness cannot pick the winner, extra attempts do not help the user. They help the benchmark. Reporting pass@k for that product is reporting a capability nobody in the loop can spend.

The (c/n)^k shortcut overstates your agent by about half

The practitioner write-ups in circulation tend to hand you (c/n)^k as the working formula, sometimes after correctly stating the binomial version first and calling the power form an approximation. It is a worse approximation than it looks. It is a worse approximation than it looks.

I evaluated both estimators in closed form, taking the expectation of each over a binomial distribution of successes rather than sampling. The hypergeometric form, C(c, k) / C(n, k), comes out exactly unbiased for p^k at every setting tested. The plug-in (c/n)^k comes out biased upward at every setting tested. With n = 8 trials, p = 0.6 and k = 4, the plug-in averages 0.195 against a true value of 0.130, an overstatement of roughly 50 percent. Push n to 50 and the gap narrows to 0.140 against 0.130. The bias runs in the direction that makes an agent look steadier than it is, which is the direction bias in evaluation reporting reliably runs.

There is a second trap that bites harder. At k = n the estimator has no resolution left. Run eight trials and pass^8 for a task is C(c, 8) / C(8, 8), which is 1 if all eight succeeded and 0 otherwise. pass@8 on those same eight trials is 1 the moment a single trial succeeds. Both collapse to a binary at the edge, and averaging binaries across tasks hides that you measured almost nothing.

If pass^8 is going to carry weight, run considerably more than eight trials per task. I treat n of at least four times the largest k as a floor, and I state n next to the number.

The shape of the pass^k curve tells you which problem you have

Averaging pass^k across tasks conceals which failure you are actually looking at. Consider two agents that both post pass^1 of 0.5. Agent A is a coin flip on every task. Agent B solves half the tasks every single time and never solves the other half.

A's curve runs 0.500, 0.250, 0.063, 0.004 as k goes 1, 2, 4, 8. B's curve sits at 0.500 forever. Identical headline, completely different engineering. A has a variance problem, which scaffolding, verification and determinism can attack. B has a capability problem, which only a better model or a redesigned task will move. A single pass^k value cannot tell them apart. The slope can.

And the variance is not only sampling temperature. Horace He and Thinking Machines Lab, September 2025, generated 1000 completions of the same prompt at temperature zero on Qwen3-235B-A22B-Instruct-2507 and got 80 distinct completions, first diverging at token 103. The cause is not the sampler. Matmul, RMSNorm and attention kernels are not batch invariant, so the floating-point reduction order for your request depends on how many other people's requests were batched alongside it. Their framing is that concurrent users behave less like an input to the system and more like a source of nondeterminism inside it.

Which means your production agent's pass^k is partly a function of server load at the hour it was measured. Worth establishing before you spend a week rewriting the prompt.

pass@k is the correct metric when a verifier does the picking

The strongest case against everything above is that pass@k is not dishonest, it is answering a different and legitimate question: can this model reach the solution at all, given enough draws. That is exactly what you want to know when you are picking a base model to post-train, when a coding agent runs inside a test suite that adjudicates for you, or when a human reviews every output anyway and only needs one usable draft out of five. In those settings the product genuinely re-rolls, and pass@k is the user-facing number. Reporting pass^k there would understate a system that works.

pass^k also has defects worth saying out loud. It decays exponentially on every task the agent does not solve every single time, so at large enough k most systems converge toward zero and the metric stops ranking anything. It charges environment flakiness to the agent: a rate-limited API or a sandbox that times out reads as model unreliability. It penalises a hard task and a flaky task identically, which is the heterogeneity problem all over again. And k is a free parameter, so whoever reports it can tune k until the story lands where they wanted.

Neither number stands alone. What stands is both of them, with the trial count and the interval attached, and an explicit statement of whether anything in the product can tell a good run from a bad one. That last sentence is the one most evaluation write-ups are missing, and it is the one that decides which metric was ever appropriate.

Report the curve, the trial count and the price

Concretely, for an agent whose output reaches a person. Report pass^1 with a task-level bootstrap interval rather than a bare point. Run n at least four times your largest k and print n next to every figure. Publish the pass^k curve for k = 1 through 8 instead of the single k that flattered you. Publish per-task success counts, because variance concentrates in a handful of unstable tasks rather than spreading evenly, and the reader needs to see where.

Publish pass@k only beside the verifier that gives it meaning, and name the verifier. If the product has no verifier and no re-roll, leave pass@k out entirely. Use the binomial estimators, not (c/n)^k and not 1 - (1 - p)^k. State the temperature and whether your serving stack uses batch-invariant kernels. Report cost per attempt: Kapoor and colleagues, AI Agents That Matter, July 2024, argued that accuracy-only benchmarking has pushed state-of-the-art agents toward needless complexity and expense, and a pass^k bought with 40 retries is a different product from one bought with none.

The field has not internalised much of this. APEX-Agents, published January 2026 with 480 long-horizon tasks written by investment banking analysts, management consultants and corporate lawyers, still runs its leaderboard on Pass@1. And an August 2026 paper, Beyond Pass@k, found agentic code-generation work setting n to the number of unit tests inside a single submission rather than the number of independent rollouts. Correcting that one wiring error moved reported scores from a range of 0.96 to 0.98 down to a range of 0.00 to 0.12.

The estimators have been correct and public since 2021. Feeding them the wrong quantity is where the damage happens.

The metric names who is doing the retrying

Choosing between these two is not a statistical preference. It is a decision about who absorbs the variance.

pass@k assigns it to the machine, which can run 250 attempts overnight for the price of a coffee and let a test suite sort the wreckage. pass^k assigns it to the person, who gets one attempt, at the moment they needed it, on an afternoon when the batch size on some GPU in a data centre happened to run large.

So publish the number that names who is actually doing the retrying. If the answer is nobody, publish pass^k and take the beating. The alternative is that your users discover the real figure on your behalf, one at a time, and the ones who leave never file a bug report.

Tools referenced

vLLM, reviewed here: vLLM review.

Sources

Chen et al., Evaluating Large Language Models Trained on Code (Codex), arXiv:2107.03374. Original pass@k definition and unbiased estimator: https://arxiv.org/abs/2107.03374

Yao, Shinn, Razavi, Narasimhan, tau-bench, arXiv:2406.12045. Introduces pass^k and reports GPT-4o pass^1 of 61.2% retail, 35.2% airline, pass^8 below 25%: https://arxiv.org/abs/2406.12045

tau2-Bench: Evaluating Conversational Agents in a Dual-Control Environment, arXiv:2506.07982. Pass^k across retail, airline and telecom domains: https://arxiv.org/abs/2506.07982

Brown et al., Large Language Monkeys: Scaling Inference Compute with Repeated Sampling, arXiv:2407.21787. Coverage scaling and the verifier requirement: https://arxiv.org/abs/2407.21787

Kapoor, Stroebl, Siegel, Nadgir, Narayanan, AI Agents That Matter, arXiv:2407.01502. Cost-blind benchmarking and reproducibility: https://arxiv.org/abs/2407.01502

Horace He and Thinking Machines Lab, Defeating Nondeterminism in LLM Inference, September 2025. 1000 temperature-zero completions produced 80 unique outputs: https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/

Jiang, Zheng, Vidra, Setty, Beyond Pass@k: Measuring Reliability and Security of Agentic Code Generation, arXiv:2608.14711. Misapplied n inflating scores from 0.00-0.12 to 0.96-0.98: https://arxiv.org/abs/2608.14711

Vidgen et al., APEX-Agents, arXiv:2601.14242. January 2026 long-horizon agent benchmark still leaderboarding on Pass@1: https://arxiv.org/abs/2601.14242