quantilizer
In plain wordsAn AI decision rule that, instead of picking the single highest-scoring action, picks at random among the top few percent of ordinary, human-like actions — limiting how badly a flawed goal can backfire.
An agent that chooses by filtering rather than maximizing: it starts from a trusted base distribution γ over actions (for example, the actions a human might plausibly take), keeps only the slice of γ's probability mass, of size q, that a utility function U scores highest, and samples from that slice in proportion to γ. As q approaches 0 it approaches a maximizer; at q = 1 it simply imitates the base distribution. Its key property is a bound on harm: for any non-negative cost function, a q-quantilizer's expected cost is at most 1/q times that of the base distribution, so if ordinary behavior is rarely catastrophic, quantilizing cannot be much worse — even when U is badly wrong. Quantilizers thus trade some performance for robustness to errors in the objective, and they are only as safe as their base distribution.
A 1% quantilizer can be at most a hundred times riskier than the human behavior it samples from.
Origin
Introduced by Jessica Taylor of the Machine Intelligence Research Institute in 'Quantilizers: A Safer Alternative to Maximizers for Limited Optimization' at the AAAI 2016 Workshop on AI, Ethics, and Society. The name blends 'quantile' and 'optimizer'.
Usage note
The best-known formal proposal for mild optimizationmild optimization, n.: Building an AI that settles for a solid result instead of squeezing out every last bit of score, since the most extreme ways to max out a goal are often loopholes or excesses nobody wanted.. Best-of-n samplingbest-of-n sampling, n.: Asking an AI the same question several times and keeping whichever answer a scoring model or checker rates highest — a simple way to trade extra computing for better answers. from a language model behaves somewhat like a quantilizer with q ≈ 1/n. The guarantee weakens if ordinary actions are themselves risky or if a learned imitation of human behavior is unreliable.
Related
- mild optimizationmild optimization, n.: Building an AI that settles for a solid result instead of squeezing out every last bit of score, since the most extreme ways to max out a goal are often loopholes or excesses nobody wanted.
- Goodhart's lawGoodhart's law, n.: The rule of thumb that once a number is used as a target, people or AIs find ways to push it up that no longer reflect what it was meant to measure.
- impact measureimpact measure, n.: A way of scoring how much an AI changes the world, so that it can be penalized for doing more than its task requires.
- imitation learningimitation learning, n.: Teaching an AI by showing it examples of an expert doing a task, so it learns to copy the behavior instead of discovering it through trial and error.
- best-of-n samplingbest-of-n sampling, n.: Asking an AI the same question several times and keeping whichever answer a scoring model or checker rates highest — a simple way to trade extra computing for better answers.
- optimizationoptimization, n.: Finding the best possible choice according to some score, such as the lowest error or the highest reward. Training an AI is optimization: adjusting its settings until its mistakes are as small as possible.
Sources
- Taylor, J. (2016). Quantilizers: A Safer Alternative to Maximizers for Limited Optimization. AAAI-16 Workshop on AI, Ethics, and Society.
Mentioned in
Goodhart's lawGoodhart's law, n.: The rule of thumb that once a number is used as a target, people or AIs find ways to push it up that no longer reflect what it was meant to measure., satisficingsatisficing, n.: Choosing the first option that is good enough instead of searching for the very best one — a sensible strategy when time, information or brainpower are limited., mild optimizationmild optimization, n.: Building an AI that settles for a solid result instead of squeezing out every last bit of score, since the most extreme ways to max out a goal are often loopholes or excesses nobody wanted.