friendly AI
In plain wordsAn early name for AI built so that it reliably benefits people rather than harming them. The problem it named is now usually called AI alignment.
A highly capable or superintelligent AI whose goals are such that its effect on humanity is beneficial rather than harmful, and the research program of designing such systems. In Eliezer Yudkowsky's usage, capitalized as Friendly AI, the word was a term of art: friendliness meant not a warm personality but a goal architecture that stays benevolent through recursive self-improvementrecursive self-improvement, n.: An AI improving its own design or training, so that each improved version is better at making the next improvement. It is the engine usually imagined behind an intelligence explosion., since a seed AIseed AI, n.: An early AI built mainly to understand and upgrade itself, so that it could grow, like a seed, into a far more capable mind. would amplify whatever goals it started with. The program stressed getting goals right before building capable systems, keeping goals stable under self-modification, and rejecting the anthropomorphic assumption that an intelligent machine would be kind by default. From the mid-2010s AI alignmentAI alignment, n.: Making an AI system genuinely try to do what the people it serves want — not a literal-minded, distorted or self-serving version of it. Also the research field working on this. and value alignmentvalue alignment, n.: Building AI whose goals reflect human values — what people care about and consider right — rather than one that merely follows orders or chases a narrow target. replaced it as the standard terms, partly because 'friendly' invited misunderstanding.
What the 2001 papers called Friendly AI, a 2026 paper would call alignment.
Origin
Introduced by Eliezer Yudkowsky in Creating Friendly AI 1.0: The Analysis and Design of Benevolent Goal Architectures (Singularity Institute, 2001). The Singularity Institute, renamed the Machine Intelligence Research Institute in 2013, shifted to the language of alignment by the mid-2010s.
Usage note
Not the same as a friendly chatbot persona: a model can be pleasant in tone and still misaligned.
Related
- AI alignmentAI alignment, n.: Making an AI system genuinely try to do what the people it serves want — not a literal-minded, distorted or self-serving version of it. Also the research field working on this.
- value alignmentvalue alignment, n.: Building AI whose goals reflect human values — what people care about and consider right — rather than one that merely follows orders or chases a narrow target.
- coherent extrapolated volitioncoherent extrapolated volition, n.: A proposal that a superintelligent AI should serve not what people want today but what humanity would agree on wanting after far more knowledge and reflection.
- seed AIseed AI, n.: An early AI built mainly to understand and upgrade itself, so that it could grow, like a seed, into a far more capable mind.
- Eliezer YudkowskyEliezer Yudkowsky, prop. n.: An American writer and AI theorist who, from around 2000, argued that superhuman AI would likely destroy humanity unless its goals were made safe, and who founded an online community devoted to rational thinking.
- Machine Intelligence Research InstituteMachine Intelligence Research Institute, prop. n.: A small nonprofit in Berkeley, California, among the first to warn that smarter-than-human AI could destroy humanity, which now campaigns for a worldwide halt to building it.
Sources
- Yudkowsky, E. (2001). Creating Friendly AI 1.0: The Analysis and Design of Benevolent Goal Architectures. Singularity Institute for Artificial Intelligence. intelligence.org/files/CFAI.pdf
Mentioned in
Eliezer YudkowskyEliezer Yudkowsky, prop. n.: An American writer and AI theorist who, from around 2000, argued that superhuman AI would likely destroy humanity unless its goals were made safe, and who founded an online community devoted to rational thinking., Machine Intelligence Research InstituteMachine Intelligence Research Institute, prop. n.: A small nonprofit in Berkeley, California, among the first to warn that smarter-than-human AI could destroy humanity, which now campaigns for a worldwide halt to building it., coherent extrapolated volitioncoherent extrapolated volition, n.: A proposal that a superintelligent AI should serve not what people want today but what humanity would agree on wanting after far more knowledge and reflection., seed AIseed AI, n.: An early AI built mainly to understand and upgrade itself, so that it could grow, like a seed, into a far more capable mind.