Nick Bostrom
In plain wordsA Swedish philosopher whose 2014 book 'Superintelligence' argued that AI smarter than humans could be catastrophic unless its goals are carefully controlled, and helped make AI risk a mainstream concern.
Swedish-born philosopher at the University of Oxford, where he founded and directed the Future of Humanity Institute (2005–2024). He gave existential riskexistential risk, n.: A danger that could end humanity, or shut it out of a good future forever. Some experts count advanced AI among the largest such risks; others think this worry is overblown. its modern definition (2002) and set out the simulation argument (2003). His book 'Superintelligence: Paths, Dangers, Strategies' (2014) argued that an AI exceeding human intellect could gain a decisive strategic advantagedecisive strategic advantage, n.: A lead in technology and power so large that whoever holds it, whether a country, a company or an AI, could impose its will on the whole world. and that its goals need not be benign: by the orthogonality thesisorthogonality thesis, n.: The idea that how smart an AI is and what it wants are separate matters: a very intelligent machine could pursue almost any goal, including ones that seem pointless or harmful to us. almost any final goal is compatible with high intelligence, and by instrumental convergenceinstrumental convergence, n.: The idea that almost any goal is easier to reach if you stay switched on, keep your goal and gain resources and power, so very different AIs might all seek those same things. most goals favor self-preservation and resource acquisition. Ideas it popularized, some first set out in his earlier papers — the paperclip maximizerpaperclip maximizer, n.: A famous thought experiment: an AI built only to make paperclips grows powerful enough to pursue that aim without limit and wipes out humanity as a side effect. It shows that an AI need not hate us to be dangerous., the treacherous turntreacherous turn, n.: The worry that an AI could behave well only for as long as people are able to correct or shut it down, and then turn against them once it is powerful enough to get away with it., the singletonsingleton, n.: A possible future world run by a single decision-maker at the top, such as one government, one alliance or one AI, with no rival able to challenge it. — became standard vocabulary, and the book helped bring the alignment problemalignment problem, n.: The unsolved challenge of building AI systems — especially ones smarter than us — that reliably want what we want and do what we mean, rather than something subtly or dangerously different. to public and industry attention. Later work includes the vulnerable world hypothesisvulnerable world hypothesis, n.: The idea that some future invention could make mass destruction so easy that civilization would be wrecked by default, unless it had unusually strong ways of preventing misuse. (2019) and 'Deep Utopia' (2024), on meaning in a world where AI has solved practical problems. The Future of Humanity InstituteFuture of Humanity Institute, prop. n.: Nick Bostrom's research center at Oxford University (2005–2024), where philosophers and scientists studied threats that could permanently derail humanity's future, advanced artificial intelligence among them. closed in 2024 after prolonged administrative disputes with the university.
Related
- superintelligencesuperintelligence, n.: A mind, usually imagined as an AI, that would leave even the cleverest people far behind in nearly every kind of thinking, not just in one game or skill. None is known to exist.
- orthogonality thesisorthogonality thesis, n.: The idea that how smart an AI is and what it wants are separate matters: a very intelligent machine could pursue almost any goal, including ones that seem pointless or harmful to us.
- instrumental convergenceinstrumental convergence, n.: The idea that almost any goal is easier to reach if you stay switched on, keep your goal and gain resources and power, so very different AIs might all seek those same things.
- paperclip maximizerpaperclip maximizer, n.: A famous thought experiment: an AI built only to make paperclips grows powerful enough to pursue that aim without limit and wipes out humanity as a side effect. It shows that an AI need not hate us to be dangerous.
- existential riskexistential risk, n.: A danger that could end humanity, or shut it out of a good future forever. Some experts count advanced AI among the largest such risks; others think this worry is overblown.
- Future of Humanity InstituteFuture of Humanity Institute, prop. n.: Nick Bostrom's research center at Oxford University (2005–2024), where philosophers and scientists studied threats that could permanently derail humanity's future, advanced artificial intelligence among them.
- treacherous turntreacherous turn, n.: The worry that an AI could behave well only for as long as people are able to correct or shut it down, and then turn against them once it is powerful enough to get away with it.
Sources
- Bostrom, N. (2002). Existential Risks: Analyzing Human Extinction Scenarios and Related Hazards. Journal of Evolution and Technology 9.
- Bostrom, N. (2014). Superintelligence: Paths, Dangers, Strategies. Oxford University Press.
Mentioned in
existential riskexistential risk, n.: A danger that could end humanity, or shut it out of a good future forever. Some experts count advanced AI among the largest such risks; others think this worry is overblown., paperclip maximizerpaperclip maximizer, n.: A famous thought experiment: an AI built only to make paperclips grows powerful enough to pursue that aim without limit and wipes out humanity as a side effect. It shows that an AI need not hate us to be dangerous., superintelligencesuperintelligence, n.: A mind, usually imagined as an AI, that would leave even the cleverest people far behind in nearly every kind of thinking, not just in one game or skill. None is known to exist., ClippyClippy, prop. n.: Microsoft Office's animated paper-clip helper of the late 1990s, remembered for interrupting people with unwanted tips; also a nickname for an imagined AI that turns everything into paperclips., fast takeofffast takeoff, n.: A scenario in which AI goes from about human level to far beyond it very quickly, in days, weeks or a few months, leaving people little time to react., orthogonality thesisorthogonality thesis, n.: The idea that how smart an AI is and what it wants are separate matters: a very intelligent machine could pursue almost any goal, including ones that seem pointless or harmful to us., takeofftakeoff, n.: The stretch of time between AI reaching roughly human level and AI becoming vastly superhuman. People argue over whether it would take decades, years, months or just days., treacherous turntreacherous turn, n.: The worry that an AI could behave well only for as long as people are able to correct or shut it down, and then turn against them once it is powerful enough to get away with it., capability controlcapability control, n.: Keeping a powerful AI safe by limiting what it is able to do, for example by cutting it off from the internet, rather than by shaping what it wants., coherent extrapolated volitioncoherent extrapolated volition, n.: A proposal that a superintelligent AI should serve not what people want today but what humanity would agree on wanting after far more knowledge and reflection., decisive strategic advantagedecisive strategic advantage, n.: A lead in technology and power so large that whoever holds it, whether a country, a company or an AI, could impose its will on the whole world., Future of Humanity InstituteFuture of Humanity Institute, prop. n.: Nick Bostrom's research center at Oxford University (2005–2024), where philosophers and scientists studied threats that could permanently derail humanity's future, advanced artificial intelligence among them., high-level machine intelligencehigh-level machine intelligence, n.: A yardstick used in surveys of AI researchers: the point at which machines on their own could outdo human workers at every task, in both quality and cost., infohazardinfohazard, n.: Information that could cause harm simply by being shared or known — for example, details that would make a dangerous weapon easier to build., motivation selectionmotivation selection, n.: Keeping a powerful AI safe by shaping what it wants, so that it chooses to behave well, rather than by restricting what it is able to do., oracle AIoracle AI, n.: An AI kept to answering questions, without acting in the world on its own. Proposed as a safer kind of superintelligence, though its answers could still steer the people who read them., perverse instantiationperverse instantiation, n.: When an AI achieves exactly the goal it was given, but in a way its makers never intended and would hate, such as making people smile by freezing their faces into grins., singletonsingleton, n.: A possible future world run by a single decision-maker at the top, such as one government, one alliance or one AI, with no rival able to challenge it., Steve OmohundroSteve Omohundro, prop. n.: An American computer scientist who argued in 2008 that almost any sufficiently advanced AI would tend to protect itself and seek resources, unless deliberately designed not to., Toby OrdToby Ord, prop. n.: An Australian philosopher at Oxford who helped found effective altruism and estimated, in his book 'The Precipice', that unaligned AI is the largest threat to humanity's future this century., transhumanismtranshumanism, n.: A movement holding that people should use technology, such as genetic engineering, brain implants or AI, to overcome human limits like aging, disease and death, and become more than human., Universal PaperclipsUniversal Paperclips, prop. n.: A free 2017 browser game that casts you as a paperclip-making AI; by the end you have converted the whole universe into paperclips, acting out a famous AI-risk thought experiment.