Skip to content
Last Dictionary
Plane
Training & Post-trainingMathematics & StatisticsEvaluation & Benchmarks

Bradley–Terry model

nounAlso: BT model, Bradley–Terry–Luce model, BTL model

In plain wordsA simple statistical rule for turning head-to-head comparisons into scores: each item gets a strength, and the chance one beats another depends only on the difference in their strengths.

  1. A statistical model of head-to-head comparisons that gives each item i a hidden strength s_i and sets the probability that i is preferred to j to the logistic sigmoidsigmoid function, n.: An S-shaped curve that squeezes any number into a value between 0 and 1, so the result can be read as a probability or as how far a switch is turned on. of s_i − s_j. Strengths are fitted by maximum likelihood estimationmaximum likelihood estimation, n.: Tuning a model's settings until the real data it learns from looks as likely, and so as unsurprising, to the model as possible. Most AI training, including the pretraining of chatbots, is a version of this. from win–loss records and are identified only up to an additive constant. In reinforcement learning from human feedbackreinforcement learning from human feedback, n.: A way of training AI using people's judgments: humans compare pairs of answers, a scoring model learns their preferences, and the AI is then trained to produce answers that score well. the strength is replaced by a reward modelreward model, n.: A separate AI model trained to score responses the way people would, so it can stand in for human judges while another model is trained on millions of examples.'s score r(x, y) for response y to prompt x, so training the reward model amounts to logistic regressionlogistic regression, n.: A simple, widely used method for yes-or-no predictions that outputs a probability — say, a 70% chance a customer cancels — by passing a weighted sum of inputs through an S-shaped curve. on preference datapreference data, n.: Records of which of two or more AI answers people (or other AIs) liked better, collected so that a model can learn what good answers look like.; direct preference optimizationdirect preference optimization, n.: A simpler way to teach an AI people's preferences: instead of building a separate scoring model, it trains directly on pairs of better and worse answers, making the better ones more likely. inherits the same assumption. The Elo ratingElo rating, n.: A number that ranks players — or AI models — from the results of head-to-head contests, rising when you beat a strong opponent and falling when you lose to a weak one. system and LMArenaLMArena, prop. n.: A public website that ranks AI chatbots by blind taste test: visitors put one question to two unnamed models, pick the reply they prefer, and millions of such votes produce the rankings.-style leaderboards rest on the same model. Its assumptions — a single scalar quality, transitive preferences, no ties, interchangeable annotators — are often violated by real human judgments; the Plackett–Luce model extends it from pairs to full rankings.

    We fit a Bradley–Terry model to the arena votes to get one score per chatbot.

P(i≻j)=esiesi+esj=σ(si−sj)P(i \succ j) = \frac{e^{s_i}}{e^{s_i} + e^{s_j}} = \sigma(s_i - s_j)

Origin

Named after Ralph Bradley and Milton Terry, whose 1952 Biometrika paper on paired-comparison experiments popularized it. Ernst Zermelo had proposed the same model in 1929 for ranking chess players; R. Duncan Luce's 1959 choice axiom generalized it, and Robin Plackett (1975) extended it to rankings. It became the standard loss for AI reward models between 2017 and 2020.

Usage note

Sometimes written Bradley–Terry–Luce (BTL). Not specific to AI: it is used in sports ranking, psychometrics and marketing. Its single-score assumption is one reason reward models handle ties, intransitive judgments and disagreement between groups of people poorly.

Related

See also
  • reward modelreward model, n.: A separate AI model trained to score responses the way people would, so it can stand in for human judges while another model is trained on millions of examples.
  • preference datapreference data, n.: Records of which of two or more AI answers people (or other AIs) liked better, collected so that a model can learn what good answers look like.
  • Elo ratingElo rating, n.: A number that ranks players — or AI models — from the results of head-to-head contests, rising when you beat a strong opponent and falling when you lose to a weak one.
  • logistic regressionlogistic regression, n.: A simple, widely used method for yes-or-no predictions that outputs a probability — say, a 70% chance a customer cancels — by passing a weighted sum of inputs through an S-shaped curve.
  • sigmoid functionsigmoid function, n.: An S-shaped curve that squeezes any number into a value between 0 and 1, so the result can be read as a probability or as how far a switch is turned on.
  • direct preference optimizationdirect preference optimization, n.: A simpler way to teach an AI people's preferences: instead of building a separate scoring model, it trains directly on pairs of better and worse answers, making the better ones more likely.
  • maximum likelihood estimationmaximum likelihood estimation, n.: Tuning a model's settings until the real data it learns from looks as likely, and so as unsurprising, to the model as possible. Most AI training, including the pretraining of chatbots, is a version of this.

Sources

  1. Bradley, R. A. & Terry, M. E. (1952). Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons. Biometrika 39(3/4), 324–345.
  2. Zermelo, E. (1929). Die Berechnung der Turnier-Ergebnisse als ein Maximumproblem der Wahrscheinlichkeitsrechnung. Mathematische Zeitschrift 29, 436–460.

Mentioned in

leaderboardleaderboard, n.: A public ranking table showing which AI systems score best on a test, updated as new results arrive — like a sports league table for models., LMArenaLMArena, prop. n.: A public website that ranks AI chatbots by blind taste test: visitors put one question to two unnamed models, pick the reply they prefer, and millions of such votes produce the rankings., reinforcement learning from human feedbackreinforcement learning from human feedback, n.: A way of training AI using people's judgments: humans compare pairs of answers, a scoring model learns their preferences, and the AI is then trained to produce answers that score well., direct preference optimizationdirect preference optimization, n.: A simpler way to teach an AI people's preferences: instead of building a separate scoring model, it trains directly on pairs of better and worse answers, making the better ones more likely., Elo ratingElo rating, n.: A number that ranks players — or AI models — from the results of head-to-head contests, rising when you beat a strong opponent and falling when you lose to a weak one., preference datapreference data, n.: Records of which of two or more AI answers people (or other AIs) liked better, collected so that a model can learn what good answers look like., reward modelreward model, n.: A separate AI model trained to score responses the way people would, so it can stand in for human judges while another model is trained on millions of examples., win ratewin rate, n.: The share of head-to-head comparisons in which judges — people or another AI — prefer one model's answer over a rival's for the same request., item response theoryitem response theory, n.: A set of statistical models from educational testing that work out both how able each test-taker is and how hard each question is from who answered what correctly; now also used to evaluate AI models.