Kahneman–Tversky optimization
In plain wordsA way to tune an AI's behavior that needs only a thumbs-up or thumbs-down on single answers, rather than side-by-side comparisons, borrowing ideas from the psychology of how people weigh gains and losses.
A preference-optimization method for language models that learns from unpaired binary feedback — each output labeled simply desirable or undesirable — instead of the chosen/rejected pairs required by direct preference optimizationdirect preference optimization, n.: A simpler way to teach an AI people's preferences: instead of building a separate scoring model, it trains directly on pairs of better and worse answers, making the better ones more likely.. It treats the log-probability ratio between policy and reference modelreference model, n.: A frozen copy of an AI model, saved before further training, that the new version is compared against so it does not stray too far from where it started. as an implicit reward and passes it through a value function modeled on prospect theory: gains and losses are measured relative to a reference point (estimated from the KL divergence between policy and reference), and losses can be weighted more heavily than gains. Separate weights for desirable and undesirable examples let it cope with imbalanced data. The authors cast DPO and PPO-style objectives as 'human-aware losses' (HALOs) that implicitly encode such biases, and report that KTO matches or beats pairwise methods from 1 to 30 billion parameters. Its practical appeal is that thumbs-up/thumbs-down logs are far cheaper to collect than comparisons.
We only had thumbs-up and thumbs-down logs from the app, so we trained with KTO instead of DPO.
Origin
Introduced by Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky and Douwe Kiela in February 2024 and published at ICML 2024. The name honors the psychologists Daniel Kahneman and Amos Tversky, whose 1979 prospect theory describes loss-averse human valuation.
Usage note
Despite the name, KTO does not model human decision-making; prospect theory only supplies the shape of its loss. Not to be confused with Kahneman's System 1 and System 2System 1 and System 2, phr.: A popular way of describing two modes of thinking: fast, automatic and intuitive (System 1) versus slow, effortful and deliberate (System 2). AI researchers borrow it to contrast instant answers with step-by-step reasoning..
Related
- direct preference optimizationdirect preference optimization, n.: A simpler way to teach an AI people's preferences: instead of building a separate scoring model, it trains directly on pairs of better and worse answers, making the better ones more likely.
- preference datapreference data, n.: Records of which of two or more AI answers people (or other AIs) liked better, collected so that a model can learn what good answers look like.
- reference modelreference model, n.: A frozen copy of an AI model, saved before further training, that the new version is compared against so it does not stray too far from where it started.
- reinforcement learning from human feedbackreinforcement learning from human feedback, n.: A way of training AI using people's judgments: humans compare pairs of answers, a scoring model learns their preferences, and the AI is then trained to produce answers that score well.
- direct alignment algorithmdirect alignment algorithm, n.: Any method that tunes an AI directly on examples of preferred and dispreferred answers, skipping the separate scoring model and trial-and-error training used in classic RLHF.
Sources
- Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D. & Kiela, D. (2024). KTO: Model Alignment as Prospect Theoretic Optimization. ICML 2024. arXiv:2402.01306. arxiv.org/abs/2402.01306v4
- Kahneman, D. & Tversky, A. (1979). Prospect Theory: An Analysis of Decision under Risk. Econometrica 47(2), 263–291.
Mentioned in
Daniel KahnemanDaniel Kahneman, prop. n.: An Israeli-American psychologist who showed how human judgment systematically errs and popularized the idea of fast, intuitive versus slow, deliberate thinking, now used to describe AI reasoning., direct preference optimizationdirect preference optimization, n.: A simpler way to teach an AI people's preferences: instead of building a separate scoring model, it trains directly on pairs of better and worse answers, making the better ones more likely., System 1 and System 2System 1 and System 2, phr.: A popular way of describing two modes of thinking: fast, automatic and intuitive (System 1) versus slow, effortful and deliberate (System 2). AI researchers borrow it to contrast instant answers with step-by-step reasoning., direct alignment algorithmdirect alignment algorithm, n.: Any method that tunes an AI directly on examples of preferred and dispreferred answers, skipping the separate scoring model and trial-and-error training used in classic RLHF.