value learning
In plain wordsTeaching an AI what people want by having it learn from their behavior, choices and feedback, rather than trying to write every goal down in advance.
An approach to AI alignmentAI alignment, n.: Making an AI system genuinely try to do what the people it serves want — not a literal-minded, distorted or self-serving version of it. Also the research field working on this. in which a system infers the objectives it should pursue from evidence about human preferences — demonstrations, choices, comparisons, corrections or instructions — instead of receiving a hand-coded objective. Daniel Dewey's 2011 formulation had an agent keep a probability distribution over candidate utility functions and update it from observations; later work developed inverse reinforcement learninginverse reinforcement learning, n.: Working backward from behavior to goals: watching an expert act and inferring what they must be trying to achieve, so an AI can pursue the same aims., cooperative inverse reinforcement learningcooperative inverse reinforcement learning, n.: A model of a human and an AI working as a team, where only the human knows the goal and the AI must work it out while helping., inverse reward designinverse reward design, n.: A method in which an AI treats the reward its designer wrote as a clue to what the designer wanted, rather than as a perfect statement of the goal. and reward modelsreward model, n.: A separate AI model trained to score responses the way people would, so it can stand in for human judges while another model is trained on millions of examples. learned from pairwise human comparisons, the basis of reinforcement learning from human feedbackreinforcement learning from human feedback, n.: A way of training AI using people's judgments: humans compare pairs of answers, a scoring model learns their preferences, and the AI is then trained to produce answers that score well.. Open problems include modeling human irrationality — Stuart Armstrong and Sören Mindermann (2018) showed that behavior alone cannot pin down values without assumptions about how rational a person is — inferring values people hold but rarely act on, aggregating across people, and stopping the learner from manipulating the evidence it learns from.
Value learning spares us from writing the reward by hand, but now everything rests on the model of human error.
Origin
Daniel Dewey introduced 'value learning' in 'Learning What to Value' at the 2011 Artificial General Intelligence conference. Nick Bostrom's 'Superintelligence' (2014) treated it as an answer to the 'value-loading problem', and Nate Soares's 'The Value Learning Problem' (2015) set out its difficulties.
Usage note
Paul Christiano distinguished ambitious value learning (inferring everything a person values) from narrow value learning (inferring preferences for a particular task), a distinction popularized by Rohin Shah's 2018–2019 'Value Learning' sequence; current practice such as RLHF is narrow. Related techniques for recommendations and rankings are usually called preference learning.
Related
- value alignmentvalue alignment, n.: Building AI whose goals reflect human values — what people care about and consider right — rather than one that merely follows orders or chases a narrow target.
- inverse reinforcement learninginverse reinforcement learning, n.: Working backward from behavior to goals: watching an expert act and inferring what they must be trying to achieve, so an AI can pursue the same aims.
- cooperative inverse reinforcement learningcooperative inverse reinforcement learning, n.: A model of a human and an AI working as a team, where only the human knows the goal and the AI must work it out while helping.
- inverse reward designinverse reward design, n.: A method in which an AI treats the reward its designer wrote as a clue to what the designer wanted, rather than as a perfect statement of the goal.
- reinforcement learning from human feedbackreinforcement learning from human feedback, n.: A way of training AI using people's judgments: humans compare pairs of answers, a scoring model learns their preferences, and the AI is then trained to produce answers that score well.
- reward modelreward model, n.: A separate AI model trained to score responses the way people would, so it can stand in for human judges while another model is trained on millions of examples.
- coherent extrapolated volitioncoherent extrapolated volition, n.: A proposal that a superintelligent AI should serve not what people want today but what humanity would agree on wanting after far more knowledge and reflection.
Sources
- Dewey, D. (2011). Learning What to Value. Artificial General Intelligence (AGI 2011), LNCS 6830, Springer.
- Soares, N. (2015). The Value Learning Problem. Machine Intelligence Research Institute technical report; IJCAI 2016 Ethics for Artificial Intelligence workshop.
- Armstrong, S. & Mindermann, S. (2018). Occam's Razor is Insufficient to Infer the Preferences of Irrational Agents. NeurIPS 2018.
Mentioned in
utility functionutility function, n.: A way of scoring every possible outcome by how much someone, or some AI, wants it. A rational agent is often modeled as choosing whatever maximizes its expected score., value alignmentvalue alignment, n.: Building AI whose goals reflect human values — what people care about and consider right — rather than one that merely follows orders or chases a narrow target., cooperative inverse reinforcement learningcooperative inverse reinforcement learning, n.: A model of a human and an AI working as a team, where only the human knows the goal and the AI must work it out while helping., inverse reinforcement learninginverse reinforcement learning, n.: Working backward from behavior to goals: watching an expert act and inferring what they must be trying to achieve, so an AI can pursue the same aims., inverse reward designinverse reward design, n.: A method in which an AI treats the reward its designer wrote as a clue to what the designer wanted, rather than as a perfect statement of the goal., motivation selectionmotivation selection, n.: Keeping a powerful AI safe by shaping what it wants, so that it chooses to behave well, rather than by restricting what it is able to do., off-switch gameoff-switch game, n.: A simple game-theory model showing that a robot unsure what its human wants has a reason to let the human switch it off — and loses that reason once it is sure., ontological crisisontological crisis, n.: The puzzle an AI faces when deeper understanding leaves goals phrased in its old way of seeing things without a clear meaning — like a rule to protect 'the same person' once it learns minds can be copied., shard theoryshard theory, n.: A theory that people and AIs develop values as many small, situation-triggered habits shaped by past rewards, rather than as one fixed goal pursued single-mindedly., wireheadingwireheading, n.: When an AI, animal or person stimulates its own reward signal directly instead of doing anything useful — like a rat pressing a lever wired to its brain's pleasure center.