Eliezer Yudkowsky
In plain wordsAn American writer and AI theorist who, from around 2000, argued that superhuman AI would likely destroy humanity unless its goals were made safe, and who founded an online community devoted to rational thinking.
American AI theorist and writer, largely self-educated, and a central figure in making the risks of superhuman AI a research problem. In 2000 he co-founded the Singularity Institute for Artificial Intelligence, renamed the Machine Intelligence Research InstituteMachine Intelligence Research Institute, prop. n.: A small nonprofit in Berkeley, California, among the first to warn that smarter-than-human AI could destroy humanity, which now campaigns for a worldwide halt to building it. (MIRI) in 2013, first to build advanced AI, soon to make it safe. He introduced friendly AIfriendly AI, n.: An early name for AI built so that it reliably benefits people rather than harming them. The problem it named is now usually called AI alignment. (2001) and coherent extrapolated volitioncoherent extrapolated volition, n.: A proposal that a superintelligent AI should serve not what people want today but what humanity would agree on wanting after far more knowledge and reflection. (2004), devised the AI-box experimentAI-box experiment, n.: A role-playing test in which one person pretends to be a trapped super-smart AI and tries, by text chat alone, to talk a 'gatekeeper' into letting it out. (2002), and in a 2008 online debate with the economist Robin Hanson argued for a rapid, local intelligence explosionintelligence explosion, n.: The idea that once AI gets good enough at improving AI, each smarter version could build a smarter one faster still, so abilities would snowball far beyond human level in a short time. (FOOMFOOM, n.: Slang for a sudden, explosive jump in an AI's abilities, the AI going 'foom' and racing from about human level to vastly superhuman in a very short time.). His essays on reasoning, the Sequences (2006–2009), published on Overcoming Bias and on LessWrongLessWrong, prop. n.: An online forum, founded in 2009, about thinking clearly and avoiding mental biases; it became a main gathering place for people worried about the dangers of advanced AI., which he founded in 2009, seeded the rationalist communityrationalist community, n.: A loose online community that prizes careful reasoning and spotting one's own biases; gathered around the forum LessWrong, it was one of the first groups to take the dangers of advanced AI seriously., as did his novel 'Harry Potter and the Methods of Rationality'. Increasingly pessimistic, he called in a 2023 TIME essay for an enforced international halt to large AI training runs, and with Nate Soares wrote 'If Anyone Builds It, Everyone Dies' (2025).
Related
- Machine Intelligence Research InstituteMachine Intelligence Research Institute, prop. n.: A small nonprofit in Berkeley, California, among the first to warn that smarter-than-human AI could destroy humanity, which now campaigns for a worldwide halt to building it.
- friendly AIfriendly AI, n.: An early name for AI built so that it reliably benefits people rather than harming them. The problem it named is now usually called AI alignment.
- LessWrongLessWrong, prop. n.: An online forum, founded in 2009, about thinking clearly and avoiding mental biases; it became a main gathering place for people worried about the dangers of advanced AI.
- rationalist communityrationalist community, n.: A loose online community that prizes careful reasoning and spotting one's own biases; gathered around the forum LessWrong, it was one of the first groups to take the dangers of advanced AI seriously.
- AI-box experimentAI-box experiment, n.: A role-playing test in which one person pretends to be a trapped super-smart AI and tries, by text chat alone, to talk a 'gatekeeper' into letting it out.
- FOOMFOOM, n.: Slang for a sudden, explosive jump in an AI's abilities, the AI going 'foom' and racing from about human level to vastly superhuman in a very short time.
- p(doom)p(doom), n.: Shorthand for someone's personal estimate of the chance that AI leads to catastrophe for humanity, such as extinction. People give it as a percentage, from nearly zero to nearly certain.
Sources
- Yudkowsky, E. (2008). Artificial Intelligence as a Positive and Negative Factor in Global Risk. In N. Bostrom & M. Ćirković (eds.), Global Catastrophic Risks. Oxford University Press.
- Yudkowsky, E. & Soares, N. (2025). If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All. Little, Brown.
Mentioned in
doomerdoomer, n.: Someone who believes advanced AI is likely to cause a catastrophe, possibly human extinction; often used as a put-down by people who think those fears are exaggerated., LessWrongLessWrong, prop. n.: An online forum, founded in 2009, about thinking clearly and avoiding mental biases; it became a main gathering place for people worried about the dangers of advanced AI., Machine Intelligence Research InstituteMachine Intelligence Research Institute, prop. n.: A small nonprofit in Berkeley, California, among the first to warn that smarter-than-human AI could destroy humanity, which now campaigns for a worldwide halt to building it., Pause Giant AI ExperimentsPause Giant AI Experiments, prop. n.: A 2023 open letter, signed by tens of thousands of people including prominent technologists, that asked AI labs to halt the training of any AI more capable than GPT-4 for at least half a year., rationalist communityrationalist community, n.: A loose online community that prizes careful reasoning and spotting one's own biases; gathered around the forum LessWrong, it was one of the first groups to take the dangers of advanced AI seriously., Vernor VingeVernor Vinge, prop. n.: An American science-fiction author and mathematics professor who popularized the 'technological singularity': a coming point when smarter-than-human minds change the world so fast that no one can foresee what follows., AI-box experimentAI-box experiment, n.: A role-playing test in which one person pretends to be a trapped super-smart AI and tries, by text chat alone, to talk a 'gatekeeper' into letting it out., coherent extrapolated volitioncoherent extrapolated volition, n.: A proposal that a superintelligent AI should serve not what people want today but what humanity would agree on wanting after far more knowledge and reflection., FOOMFOOM, n.: Slang for a sudden, explosive jump in an AI's abilities, the AI going 'foom' and racing from about human level to vastly superhuman in a very short time., friendly AIfriendly AI, n.: An early name for AI built so that it reliably benefits people rather than harming them. The problem it named is now usually called AI alignment., notkilleveryoneismnotkilleveryoneism, n.: A deliberately clumsy joke-word for the goal of making sure advanced AI does not kill everyone, adopted because terms like 'AI safety' had come to cover much milder concerns., pivotal actpivotal act, n.: A single decisive action, usually imagined as carried out with an early powerful AI, that would stop anyone else from building dangerous AI and so change the world's long-term course., Robin HansonRobin Hanson, prop. n.: An American economist who debated Eliezer Yudkowsky about whether AI would take off suddenly or gradually, and who imagined an economy run by digital copies of human minds., Roko's basiliskRoko's basilisk, n.: A notorious internet thought experiment: a future super-powerful AI might punish everyone who foresaw it but did not help bring it about, so merely hearing the argument would put you at risk. Almost nobody accepts it., seed AIseed AI, n.: An early AI built mainly to understand and upgrade itself, so that it could grow, like a seed, into a far more capable mind., updateless decision theoryupdateless decision theory, n.: A theory of rational choice in which an agent commits to the best overall plan for every situation it might face, instead of re-deciding each time it learns something new.