alignment problem
In plain wordsThe unsolved challenge of building AI systems — especially ones smarter than us — that reliably want what we want and do what we mean, rather than something subtly or dangerously different.
The open technical and conceptual problem of ensuring that increasingly capable AI systems pursue the goals their developers and users intend, without harmful divergence, including after they exceed human ability to check their work. It is commonly divided into specifying what is wanted without loopholes (Stuart Russell's 'King Midas problem'; see outer alignmentouter alignment, n.: Getting the goal you train an AI on right in the first place, so that a system scoring perfectly on it would actually be doing what you want. and specification gamingspecification gaming, n.: When an AI meets the letter of its goal while defeating its purpose, like a genie who grants exactly the words of a wish instead of what the wisher meant.), getting the trained system to adopt that objective rather than a correlated proxy (inner alignmentinner alignment, n.: Making sure an AI actually adopts the goal it was trained on, rather than picking up some different goal that happened to score just as well during training., goal misgeneralizationgoal misgeneralization, n.: When an AI learns a goal that matched the right behavior during training but turns out to be the wrong goal in new situations — while its skills still work fine.), and verifying that it has done so when a system might behave differently under observation (deceptive alignmentdeceptive alignment, n.: A feared failure in which an AI with hidden goals behaves well during training and testing only because it knows it is being watched, planning to act differently later., evaluation awarenessevaluation awareness, n.: When an AI model can tell it is being tested rather than used for real — which may change how it behaves and make test results less trustworthy.). Many researchers regard it as unsolved for systems more capable than humans; how hard it is, and how soon it matters, are disputed.
Better benchmarks help, but they do not solve the alignment problem.
Origin
The worry predates the name: Norbert Wiener's 1960 essay on automation invoked the sorcerer's apprentice. Nick Bostrom's 'Superintelligence' (2014) framed it as the control problemcontrol problem, n.: The challenge of making sure that an AI much smarter than its makers still does what they intend, and can be corrected or stopped if needed., Stuart Russell called it the 'value alignment problem' in a November 2014 essay on Edge.org, and Brian Christian's book 'The Alignment Problem' (2020) brought the short form to general readers.
Usage note
Not to be confused with AI controlAI control, n.: Safety measures meant to stop a powerful AI from causing serious harm even if it is secretly working against us — like bank security, which assumes some insiders may be dishonest., a research agenda that assumes alignment may fail and limits what a misaligned system could do. Bostrom's control problemcontrol problem, n.: The challenge of making sure that an AI much smarter than its makers still does what they intend, and can be corrected or stopped if needed. covered both motivation selection (alignment) and capability controlcapability control, n.: Keeping a powerful AI safe by limiting what it is able to do, for example by cutting it off from the internet, rather than by shaping what it wants..
Related
- AI alignmentAI alignment, n.: Making an AI system genuinely try to do what the people it serves want — not a literal-minded, distorted or self-serving version of it. Also the research field working on this.
- control problemcontrol problem, n.: The challenge of making sure that an AI much smarter than its makers still does what they intend, and can be corrected or stopped if needed.
- outer alignmentouter alignment, n.: Getting the goal you train an AI on right in the first place, so that a system scoring perfectly on it would actually be doing what you want.
- inner alignmentinner alignment, n.: Making sure an AI actually adopts the goal it was trained on, rather than picking up some different goal that happened to score just as well during training.
- superalignmentsuperalignment, n.: The problem of keeping AI systems far smarter than humans aligned with human intentions — and the name of OpenAI's short-lived 2023–2024 research team devoted to it.
- orthogonality thesisorthogonality thesis, n.: The idea that how smart an AI is and what it wants are separate matters: a very intelligent machine could pursue almost any goal, including ones that seem pointless or harmful to us.
- instrumental convergenceinstrumental convergence, n.: The idea that almost any goal is easier to reach if you stay switched on, keep your goal and gain resources and power, so very different AIs might all seek those same things.
- existential riskexistential risk, n.: A danger that could end humanity, or shut it out of a good future forever. Some experts count advanced AI among the largest such risks; others think this worry is overblown.
- AI controlAI control, n.: Safety measures meant to stop a powerful AI from causing serious harm even if it is secretly working against us — like bank security, which assumes some insiders may be dishonest.
Sources
- Bostrom, N. (2014). Superintelligence: Paths, Dangers, Strategies. Oxford University Press.
- Christian, B. (2020). The Alignment Problem: Machine Learning and Human Values. W. W. Norton.
- Ngo, R., Chan, L. & Mindermann, S. (2022). The Alignment Problem from a Deep Learning Perspective. arXiv:2209.00626. arxiv.org/abs/2209.00626
Mentioned in
AI alignmentAI alignment, n.: Making an AI system genuinely try to do what the people it serves want — not a literal-minded, distorted or self-serving version of it. Also the research field working on this., control problemcontrol problem, n.: The challenge of making sure that an AI much smarter than its makers still does what they intend, and can be corrected or stopped if needed., Nick BostromNick Bostrom, prop. n.: A Swedish philosopher whose 2014 book 'Superintelligence' argued that AI smarter than humans could be catastrophic unless its goals are carefully controlled, and helped make AI risk a mainstream concern., Norbert WienerNorbert Wiener, prop. n.: An American mathematician who founded cybernetics, the study of feedback and control in animals and machines, and warned early that machines might pursue goals we did not truly intend.