pass@k
In plain wordsA score for code-writing AI: the chance that at least one of k attempts at a problem is correct, as checked by actually running tests on the code.
A metric giving the probability that at least one of k sampled outputs for a problem passes its correctness check — usually the problem's unit tests — averaged over problems. Because drawing exactly k samples gives a noisy estimate, practice is to sample n ≥ k outputs per problem, note how many of them (c) are correct, and derive from n and c an unbiased, lower-variance estimate of pass@k (formula below). pass@1 approximates single-attempt accuracy; larger k measures what a system could achieve given a perfect way to pick among its attempts, and so overstates practical performance. Related measures are cons@k or maj@k (accuracy of the most common answer among k samples) and pass^k, introduced with τ-benchτ-bench, prop. n.: A test of AI customer-service agents in which a simulated customer, played by another AI, chats with the agent, which must use company tools and follow written rules to resolve the request.: the probability that all k independent trials succeed, a measure of consistency rather than coverage.
The model's pass@1 on HumanEval is 92%, but its pass^5 on our agent tasks is under 40%.
Origin
Mark Chen and colleagues' July 2021 OpenAI Codex paper, which introduced HumanEvalHumanEval, prop. n.: A set of 164 small programming exercises, each described in plain English and checked by hidden tests, used to see whether an AI can write working Python code., popularized the name pass@k and its unbiased estimator. They credited the evaluation idea to Sumith Kulal and colleagues' 2019 SPoC paper, which scored program synthesis by success within a budget of attempts.
Usage note
Scores are comparable only when k, the sampling temperature and n are reported. Do not confuse pass@k, which rises with k, with pass^k, which falls as k rises.
Related
- HumanEvalHumanEval, prop. n.: A set of 164 small programming exercises, each described in plain English and checked by hidden tests, used to see whether an AI can write working Python code.
- τ-benchτ-bench, prop. n.: A test of AI customer-service agents in which a simulated customer, played by another AI, chats with the agent, which must use company tools and follow written rules to resolve the request.
- code generationcode generation, n.: Having an AI write computer code from a description in plain language, from finishing a single line as you type to building whole programs.
- self-consistencyself-consistency, n.: Asking an AI the same question many times, letting it reason differently each time, then going with the answer it reaches most often — a vote among its own attempts.
- best-of-n samplingbest-of-n sampling, n.: Asking an AI the same question several times and keeping whichever answer a scoring model or checker rates highest — a simple way to trade extra computing for better answers.
- exact matchexact match, n.: The simplest way to grade an AI's answer: it counts as right only if it is identical to the reference answer, after minor tidying such as ignoring capital letters.
Sources
- Chen, M., Tworek, J., Jun, H. et al. (2021). Evaluating Large Language Models Trained on Code. arXiv:2107.03374. arxiv.org/abs/2107.03374
- Kulal, S., Pasupat, P., Chandra, K. et al. (2019). SPoC: Search-based Pseudocode to Code. NeurIPS 2019. arxiv.org/abs/1906.04908
Mentioned in
benchmarkbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time., best-of-n samplingbest-of-n sampling, n.: Asking an AI the same question several times and keeping whichever answer a scoring model or checker rates highest — a simple way to trade extra computing for better answers., code generationcode generation, n.: Having an AI write computer code from a description in plain language, from finishing a single line as you type to building whole programs., confidence intervalconfidence interval, n.: A range of plausible values for something being estimated, such as 'accuracy 82 percent, give or take 2 points', reflecting the uncertainty that comes from measuring only a sample., exact matchexact match, n.: The simplest way to grade an AI's answer: it counts as right only if it is identical to the reference answer, after minor tidying such as ignoring capital letters., HumanEvalHumanEval, prop. n.: A set of 164 small programming exercises, each described in plain English and checked by hidden tests, used to see whether an AI can write working Python code., reinforcement learning with verifiable rewardsreinforcement learning with verifiable rewards, n.: Training an AI by letting it attempt problems whose answers can be checked automatically — a math result, a passing program — and rewarding it only when the check succeeds., RL scalingRL scaling, n.: The practice, and the finding behind it, of giving AI models far more trial-and-error practice after their initial training, especially on problems with checkable answers, which has so far improved their reasoning fairly predictably., samplingsampling, n.: Picking some items at random from a larger set or from a range of possibilities — surveying 1,000 voters, say, or letting an AI choose its next word by a weighted lottery., τ-benchτ-bench, prop. n.: A test of AI customer-service agents in which a simulated customer, played by another AI, chats with the agent, which must use company tools and follow written rules to resolve the request.