Skip to content
Last Dictionary
Plane
Evaluation & BenchmarksTraining & Post-training

win rate

nounAlso: pairwise win rate, length-controlled win rate, AlpacaEval

In plain wordsThe share of head-to-head comparisons in which judges — people or another AI — prefer one model's answer over a rival's for the same request.

  1. The proportion of pairwise comparisons in which a system's output is preferred to a reference system's output on the same inputs, with ties counted as half a win or reported separately. Preferences come from human ratershuman evaluation, n.: Judging an AI's outputs by asking people to rate or compare them, because for many tasks — writing, conversation, translation — no automatic score captures quality well. or an LLM-as-a-judgeLLM-as-a-judge, n.: Using one AI language model to grade or compare the answers of another, as a cheaper and faster stand-in for human reviewers.. Win rates appear in reinforcement learning from human feedbackreinforcement learning from human feedback, n.: A way of training AI using people's judgments: humans compare pairs of answers, a scoring model learns their preferences, and the AI is then trained to produce answers that score well. papers (a tuned policy against a baseline), in automatic benchmarks such as AlpacaEval, which compares a model's answers with those of a fixed reference model, and in GDPvalGDPval, prop. n.: An OpenAI test that pits AI against seasoned professionals on genuine workplace assignments, from reports and spreadsheets to slide decks, in 44 occupations important to the US economy., which reports how often experts judge AI work as good as or better than professionals' work. Because judges favor longer answers, AlpacaEval 2.0 introduced a length-controlled win rate that adjusts for output length by regression. A win rate depends on the chosen baseline and is not an absolute measure of quality.

    Against the previous release, the new model has a 61% win rate on coding prompts, counting ties as half.

Origin

Pairwise win rates were standard in human-feedback studies such as OpenAI's summarization work (Nisan Stiennon and colleagues, 2020) and InstructGPT (Long Ouyang and colleagues, 2022). AlpacaEval (Stanford, 2023) automated the measure with model judges, and Yann Dubois and colleagues added length control in 2024.

Usage note

A 50% win rate means parity with the baseline, not mediocre quality. Win rates against different baselines are not comparable, and verbosity or a style the judges happen to like can inflate them.

Related

See also
  • LLM-as-a-judgeLLM-as-a-judge, n.: Using one AI language model to grade or compare the answers of another, as a cheaper and faster stand-in for human reviewers.
  • human evaluationhuman evaluation, n.: Judging an AI's outputs by asking people to rate or compare them, because for many tasks — writing, conversation, translation — no automatic score captures quality well.
  • Elo ratingElo rating, n.: A number that ranks players — or AI models — from the results of head-to-head contests, rising when you beat a strong opponent and falling when you lose to a weak one.
  • Bradley–Terry modelBradley–Terry model, n.: A simple statistical rule for turning head-to-head comparisons into scores: each item gets a strength, and the chance one beats another depends only on the difference in their strengths.
  • LMArenaLMArena, prop. n.: A public website that ranks AI chatbots by blind taste test: visitors put one question to two unnamed models, pick the reply they prefer, and millions of such votes produce the rankings.
  • GDPvalGDPval, prop. n.: An OpenAI test that pits AI against seasoned professionals on genuine workplace assignments, from reports and spreadsheets to slide decks, in 44 occupations important to the US economy.

Sources

  1. Dubois, Y., Galambosi, B., Liang, P. & Hashimoto, T. B. (2024). Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators. arXiv preprint.

Mentioned in

LMArenaLMArena, prop. n.: A public website that ranks AI chatbots by blind taste test: visitors put one question to two unnamed models, pick the reply they prefer, and millions of such votes produce the rankings., Elo ratingElo rating, n.: A number that ranks players — or AI models — from the results of head-to-head contests, rising when you beat a strong opponent and falling when you lose to a weak one., GDPvalGDPval, prop. n.: An OpenAI test that pits AI against seasoned professionals on genuine workplace assignments, from reports and spreadsheets to slide decks, in 44 occupations important to the US economy., human evaluationhuman evaluation, n.: Judging an AI's outputs by asking people to rate or compare them, because for many tasks — writing, conversation, translation — no automatic score captures quality well., LLM-as-a-judgeLLM-as-a-judge, n.: Using one AI language model to grade or compare the answers of another, as a cheaper and faster stand-in for human reviewers., MT-BenchMT-Bench, prop. n.: A set of 80 two-part questions — in writing, reasoning, math, coding and more — whose answers are graded by a strong AI model, used to compare chat assistants quickly.