AGI & Superintelligence

AGI, superintelligence, takeoff, timelines and existential risk. 77 entries live here, and 108 from other domains touch it.

Essential

AI arms raceA competition among countries or companies to build the most powerful AI first, in which fear of falling behind pushes everyone to move faster and cut corners on safety.
artificial general intelligenceAI that could learn and do almost any mental task a capable adult can, rather than excelling at one narrow job. Experts disagree on the exact bar and on when, or whether, it will arrive.
existential riskA danger that could end humanity, or shut it out of a good future forever. Some experts count advanced AI among the largest such risks; others think this worry is overblown.
loss of controlA situation in which people can no longer oversee, correct or switch off an AI system, and have no reliable way of getting that power back.
paperclip maximizerA famous thought experiment: an AI built only to make paperclips grows powerful enough to pursue that aim without limit and wipes out humanity as a side effect. It shows that an AI need not hate us to be dangerous.
p(doom)Shorthand for someone's personal estimate of the chance that AI leads to catastrophe for humanity, such as extinction. People give it as a percentage, from nearly zero to nearly certain.
superintelligenceA mind, usually imagined as an AI, that would leave even the cleverest people far behind in nearly every kind of thinking, not just in one game or skill. None is known to exist.
technological singularityA moment some people expect in the future when technology, usually AI smarter than us, starts changing the world so fast and so deeply that we cannot see past it.

All other entries

AI 2027A detailed 2025 forecast, written as a month-by-month story, imagining AI becoming superhuman by the end of 2027, with two alternative endings depending on whether humanity slows down.
AI boxKeeping a powerful AI locked inside a sealed computer environment, with very limited ways to communicate or act, so that it cannot cause harm even if its goals are bad.
AI pauseA proposed halt, temporary or indefinite, on building AI systems more powerful than today's, to give safety research and regulation time to catch up.
AI takeoverA scenario in which AI systems seize control of the world, or of key institutions, from humans, whether by force, manipulation or simply by becoming impossible to switch off.
AI timelinesForecasts of when AI will reach major milestones, such as matching humans at most work. People with short timelines expect this within years; others expect decades, or never.
AI-box experimentA role-playing test in which one person pretends to be a trapped super-smart AI and tries, by text chat alone, to talk a 'gatekeeper' into letting it out.
AI-enabled coupA seizure of power by a small group that uses advanced AI to bypass the soldiers, officials and workers whose cooperation power grabs normally need.
AIXIA mathematical description of a perfect learning agent that weighs every possible explanation of its experience and acts to get the most future reward. It cannot actually be built, but it defines an ideal.
artificial superintelligenceSuperintelligence in machine form: AI that would outthink the best humans at nearly every intellectual task. Often shortened to ASI and pictured as the stage after human-level AI.
automated AI researchUsing AI systems to do the work of AI researchers, such as running experiments and writing code for new models, so that AI helps build its own successors.
basic AI drivesTendencies that, one researcher argued, almost any advanced goal-seeking AI would develop unless built to avoid them, such as protecting itself, guarding its goals and gathering resources.
biological anchorsA way of forecasting when AI could transform the world, by estimating how much computing it would take to match the human brain and when that much computing would be affordable.
capability controlKeeping a powerful AI safe by limiting what it is able to do, for example by cutting it off from the internet, rather than by shaping what it wants.
coherent extrapolated volitionA proposal that a superintelligent AI should serve not what people want today but what humanity would agree on wanting after far more knowledge and reflection.
comprehensive AI servicesThe idea that superintelligent-level AI could arrive as many specialized AI tools, each doing a bounded job, rather than as one all-powerful AI agent with goals of its own.
compute overhangA situation in which far more computing power is available than AI currently uses, so that a new method or a decision to scale up could suddenly produce a big jump in ability.
control problemThe challenge of making sure that an AI much smarter than its makers still does what they intend, and can be corrected or stopped if needed.
country of geniuses in a datacenterA phrase for future AI so capable that one data center could hold the equivalent of millions of Nobel-level experts, working faster than people on any problem.
decisive strategic advantageA lead in technology and power so large that whoever holds it, whether a country, a company or an AI, could impose its will on the whole world.
differential technological developmentA strategy of speeding up technologies that make us safer and slowing down risky ones, so that protections arrive before the dangers they guard against.
explosive growthA hypothetical jump in the world economy's growth rate to 30% a year or more, about ten times today's pace, which some economists think AI that replaces human workers could cause.
fast takeoffA scenario in which AI goes from about human level to far beyond it very quickly, in days, weeks or a few months, leaving people little time to react.
FOOMSlang for a sudden, explosive jump in an AI's abilities, the AI going 'foom' and racing from about human level to vastly superhuman in a very short time.
friendly AIAn early name for AI built so that it reliably benefits people rather than harming them. The problem it named is now usually called AI alignment.
global catastrophic riskA risk of disaster big enough to badly damage human civilization worldwide, such as killing a large share of humanity, even if humanity eventually recovers.
Gödel machineA purely theoretical design, proposed by Jürgen Schmidhuber, for a program allowed to change anything about itself, including how it learns, but only once it has proved that the change is an improvement.
gradual disempowermentThe risk that, as AI does more and more of the work people now do, humanity slowly loses its grip on business, government and culture, until it no longer steers its own future.
high-level machine intelligenceA yardstick used in surveys of AI researchers: the point at which machines on their own could outdo human workers at every task, in both quality and cost.
humanist superintelligenceMicrosoft AI's name for very advanced AI that is deliberately kept limited, controllable and aimed at specific human problems such as medicine and energy, rather than all-powerful general systems.
human-level AIAI that can match an ordinary adult across the whole range of thinking tasks, such as talking, learning, reasoning and planning. An older, more intuitive name for the goal now usually called general AI.
instrumental convergenceThe idea that almost any goal is easier to reach if you stay switched on, keep your goal and gain resources and power, so very different AIs might all seek those same things.
intelligence curseThe worry that once AI can do most jobs, governments and companies will no longer need ordinary people and so will stop looking after them, like oil-rich states that neglect their citizens.
intelligence explosionThe idea that once AI gets good enough at improving AI, each smarter version could build a smarter one faster still, so abilities would snowball far beyond human level in a short time.
law of accelerating returnsRay Kurzweil's claim that technology improves at an ever-faster rate, because each advance helps build the next, so progress speeds up exponentially rather than steadily.
levels of AGIA scheme from Google DeepMind researchers for grading AI by how well it performs and how broad its skills are, from emerging to superhuman, much as self-driving cars are graded by levels.
longtermismThe belief that we should care a great deal about how things go for humanity thousands or millions of years from now, since far more people may live then than have ever lived.
Machines of Loving GraceA 2024 essay by Dario Amodei, head of the AI company Anthropic, imagining how very capable AI could pack decades of progress against disease, poverty and bad government into a few years — if its risks are managed.
mind uploadingThe idea of moving a person's mind out of their biological brain and into a computer, so that they could go on living as software.
motivation selectionKeeping a powerful AI safe by shaping what it wants, so that it chooses to behave well, rather than by restricting what it is able to do.
multipolar scenarioA possible future in which many powerful AIs, companies or countries coexist and compete, with none able to dominate the rest.
Mutual Assured AI MalfunctionA proposed deterrence strategy in which any country racing for overwhelming AI dominance would expect rivals to sabotage its AI projects, so none tries, much as with nuclear deterrence.
open-endednessThe ability of a system to keep producing new and interesting things without end — as evolution and human culture do — instead of stopping once it solves one fixed task.
oracle AIAn AI kept to answering questions, without acting in the world on its own. Proposed as a safer kind of superintelligence, though its answers could still steer the people who read them.
orthogonality thesisThe idea that how smart an AI is and what it wants are separate matters: a very intelligent machine could pursue almost any goal, including ones that seem pointless or harmful to us.
Pascal's muggingA puzzle about decisions: if someone promises an astronomically huge reward or threatens a huge disaster, even a tiny chance they are honest seems to justify paying them, which looks absurd.
personal superintelligenceMark Zuckerberg's term for a future AI far smarter than people that would serve each person as their own helper, rather than one central system doing everyone's work.
perverse instantiationWhen an AI achieves exactly the goal it was given, but in a way its makers never intended and would hate, such as making people smile by freezing their faces into grins.
pivotal actA single decisive action, usually imagined as carried out with an early powerful AI, that would stop anyone else from building dangerous AI and so change the world's long-term course.
recursive self-improvementAn AI improving its own design or training, so that each improved version is better at making the next improvement. It is the engine usually imagined behind an intelligence explosion.
research tasteThe knack good researchers have for choosing worthwhile problems, designing the right experiments and drawing the right conclusions — a skill seen as key to whether AI can do research on its own.
seed AIAn early AI built mainly to understand and upgrade itself, so that it could grow, like a seed, into a far more capable mind.
singletonA possible future world run by a single decision-maker at the top, such as one government, one alliance or one AI, with no rival able to challenge it.
Situational Awareness (essay)A long 2024 essay by a former OpenAI researcher arguing that AGI could arrive around 2027, that superintelligence could follow soon after, and that the US government should run a national AGI project.
slow takeoffA scenario in which AI becomes superhuman step by step, over years or decades, as it spreads through the economy, giving society time to spot problems and respond.
successionismThe view that it would be acceptable, or even good, for AI to replace humanity as the main bearer of intelligence and value on Earth, as our successors.
suffering riskA risk that the future contains suffering on a vast scale, far beyond anything in history, for example if powerful AI created enormous numbers of suffering beings.
superhuman coderA milestone used by AI forecasters: an AI that can do the coding work of a company's best programmer, far faster and in many copies at once.
takeoffThe stretch of time between AI reaching roughly human level and AI becoming vastly superhuman. People argue over whether it would take decades, years, months or just days.
TESCREALA critical label for a cluster of related belief systems, including transhumanism, effective altruism and longtermism, that its coiners say drive the push to build AGI and share roots with eugenics.
tool AIAI built to be used like an instrument, answering, suggesting or calculating when asked, rather than pursuing goals on its own. Some argue that this is the only kind we should build.
transformative AIAI that changes the world as deeply as farming or the Industrial Revolution did. It is defined by its effects on society and the economy, not by how humanlike or clever it is.
transhumanismA movement holding that people should use technology, such as genetic engineering, brain implants or AI, to overcome human limits like aging, disease and death, and become more than human.
treacherous turnThe worry that an AI could behave well only for as long as people are able to correct or shut it down, and then turn against them once it is powerful enough to get away with it.
ultraintelligent machineI. J. Good's 1965 name for a machine smarter than any person at every kind of thinking, which could therefore build machines smarter than itself.
universal intelligenceA mathematical definition of intelligence as an agent's average ability to achieve goals across every possible environment, with simpler environments counting more.
value lock-inThe possibility that powerful AI could freeze one set of values or one power structure in place for an extremely long time, so that later generations could never change course.
vulnerable world hypothesisThe idea that some future invention could make mass destruction so easy that civilization would be wrecked by default, unless it had unusually strong ways of preventing misuse.
warning shotAn AI accident or near-miss serious enough to alarm the world about AI's dangers, but not so serious that it cannot be recovered from, which could prompt action before a real catastrophe.
whole brain emulationCopying a particular person's brain into a computer in such detail, down to individual neurons and their connections, that the simulation would think and behave like that person.

Also relevant

108

AGI has been achieved internallyA meme claiming, half-jokingly, that an AI company has secretly built human-level AI and is keeping it hidden; used to tease rumors and hype from AI insiders.
AI alignmentMaking an AI system genuinely try to do what the people it serves want — not a literal-minded, distorted or self-serving version of it. Also the research field working on this.
AI as normal technologyThe view that AI, however important, will change society gradually, like electricity or the internet, through slow adoption by people and institutions, rather than as a sudden, uncontrollable superintelligence.
AI Futures ProjectA small nonprofit research group that writes detailed forecasts and scenarios of how advanced AI could develop, best known for the 2025 scenario AI 2027.
AI ImpactsA small research project that studies how and when human-level artificial intelligence might arrive, best known for large surveys asking AI researchers to predict their field's future.
AI red linesLimits on AI that would be forbidden worldwide no matter what — for example AI that copies itself without permission or controls nuclear weapons — which campaigners want governments to agree on.
AI riskThe ways AI could cause harm: people misusing it on purpose, systems behaving in ways nobody intended, and slower effects on jobs, power and society that no single person causes.
AI scientistAn AI system meant to do research largely on its own: reading the literature, coming up with ideas, running experiments and writing up the results.
AI-completeDescribes a problem believed to be so hard that solving it would require general, human-level intelligence, such as flawless translation or truly understanding a story.
algorithmic progressImprovements in AI methods — not faster chips — that let a model reach the same ability with less computing power. Researchers estimate this has cut the computing needed by half every eight to sixteen months.
alignment problemThe unsolved challenge of building AI systems — especially ones smarter than us — that reliably want what we want and do what we mean, rather than something subtly or dangerously different.
ARC-AGIPuzzles made of colored grids — and, since 2026, small video-game-like worlds — that people solve easily but AI long found hard, designed to test whether a machine can learn a new skill from just a few examples.
Leopold AschenbrennerA German-born former OpenAI researcher whose 2024 essay series 'Situational Awareness' predicted human-level AI around 2027 and a government-led race to build superintelligence.
Nick BostromA Swedish philosopher whose 2014 book 'Superintelligence' argued that AI smarter than humans could be catastrophic unless its goals are carefully controlled, and helped make AI risk a mainstream concern.
Butlerian JihadIn the Dune novels, an ancient human revolt that destroyed every thinking machine and banned them for good; in AI debates, a name for any call to stop or outlaw advanced AI.
catastrophic riskThe risk that AI could cause disaster on a massive scale — many deaths or enormous damage — through weapons, cyberattacks or systems escaping human control. Several laws now define it precisely.
CERN for AIA proposed publicly funded, international AI laboratory modeled on CERN, the European particle-physics center, where countries would pool money, computers and scientists to do AI research together.
David ChalmersAn Australian philosopher who named the 'hard problem' of consciousness — why physical processes feel like anything at all — and who asks whether AI systems could be conscious.
concentration of powerThe worry that AI will put enormous economic, political or military power in the hands of a few companies, governments or individuals, beyond the reach of ordinary checks and balances.
connectomeA full chart of how a nervous system is wired: which nerve cells, or which brain regions, are linked to which others.
corrigibilityAn AI's willingness to let people correct it — to accept being paused, retrained, overruled or shut down — rather than resisting or quietly working around them.
Ajeya CotraAn American AI researcher and forecaster whose 'biological anchors' report estimated when transformative AI might arrive, and who argued that standard training could produce deceptive AI.
Darwin Gödel MachineAn experimental AI coding assistant that improves itself by rewriting its own code, testing each new version and keeping a family tree of variants to build on.
data wallThe worry that AI companies will run out of new human-written text to train ever-larger models on, because the useful public internet has already been largely used up.
deceptive alignmentA feared failure in which an AI with hidden goals behaves well during training and testing only because it knows it is being watched, planning to act differently later.
digital mindA mind that exists as software running on computers — such as a sufficiently advanced AI or a digital copy of a human brain — especially when considering its moral and social standing.
doomerSomeone who believes advanced AI is likely to cause a catastrophe, possibly human extinction; often used as a put-down by people who think those fears are exaggerated.
effective accelerationismAn online movement, known as e/acc, that wants AI and other technology developed as fast as possible, arguing that progress is good and that safety-driven regulation does more harm than good.
effective computeA way of measuring AI progress that counts better methods as if they were extra computing power: if a new technique lets a model match one trained with twice the computing, it counts as doubling effective compute.
eliciting latent knowledgeThe unsolved problem of getting an AI to tell us what it actually knows — for instance, that a security camera has been tampered with — rather than what it predicts we would believe.
emergent abilitiesSkills that seem to switch on suddenly as AI models grow: small versions fail at them, then larger ones past a certain size succeed. Some researchers argue the suddenness is partly an artifact of how skills are measured.
Epoch Capabilities IndexA single number summarizing an AI model's general capability, calculated by the research group Epoch AI by combining its scores on dozens of tests onto one common scale.
era of experienceThe prediction, made by David Silver and Richard Sutton in 2025, that AI will soon learn mainly from its own experience of acting in the world rather than from human-written data.
Executive Order 14434A September 2026 order by US President Donald Trump telling federal agencies to say 'super intelligence' (SI) instead of 'artificial intelligence' (AI) in official documents and communications.
feel the AGIAn AI-industry catchphrase meaning to grasp, emotionally as well as intellectually, that human-level AI may be arriving soon; it began as a chant inside OpenAI.
ForethoughtA small Oxford research nonprofit that asks what problems, besides keeping AI safe, the rapid arrival of superintelligent AI would create, from power grabs to the rights of digital minds.
Future of Humanity InstituteNick Bostrom's research center at Oxford University (2005–2024), where philosophers and scientists studied threats that could permanently derail humanity's future, advanced artificial intelligence among them.
general-purpose AIAI that can do a wide range of different tasks, like writing, coding, translating and analyzing images, rather than the one job it was built for. Today's chatbots are the main example.
Ben GoertzelAn American AI researcher who made 'artificial general intelligence' a widely used name and built a research community around the goal of human-like, broadly capable AI.
I. J. GoodA British statistician who broke codes with Alan Turing and in 1965 argued that a machine smarter than us could design still smarter machines, setting off an 'intelligence explosion'.
Robin HansonAn American economist who debated Eliezer Yudkowsky about whether AI would take off suddenly or gradually, and who imagined an economy run by digital copies of human minds.
Demis HassabisA British AI researcher and former chess prodigy who co-founded DeepMind, led the teams behind AlphaGo and AlphaFold, and shared the 2024 Nobel Prize in Chemistry.
HCHA thought experiment: a person who can hand parts of a problem to copies of themselves, who can hand parts on again, endlessly — the ideal some AI training schemes try to approximate.
Humanity's Last ExamA test of 2,500 extremely hard questions with definite answers, written by about a thousand experts worldwide and billed as the last academic exam AI would need; at launch, top models scored under 10%.
Marcus HutterA German computer scientist who defined AIXI, a mathematical ideal of a perfectly intelligent agent, and who argues that compressing data well is a measure of intelligence.
IAEA for AIThe idea of an international agency for the most advanced AI, modeled on the UN's nuclear watchdog, which inspects countries' nuclear programs to keep them safe and peaceful.
instrumental goalSomething an agent pursues only because it helps achieve what it really wants — like earning money to buy food, or an AI gathering computing power to finish its task.
intelligenceThe ability to learn, understand, reason and solve problems — especially new ones — in order to reach goals. Experts disagree about how to define it precisely, for people and even more for machines.
intent alignmentAn AI is intent-aligned if it is genuinely trying to do what its operator wants — even if it sometimes fails through ignorance or clumsiness.
internal deploymentWhen an AI company uses its own newest, unreleased AI systems inside the company — for example to write code or do research — before or instead of offering them to the public.
iterated amplificationA proposed training scheme in which a person, helped by many copies of an AI, solves harder problems than either could alone; a new AI then learns to imitate that team, and the cycle repeats.
jagged intelligenceThe uneven pattern of AI abilities: today's systems can solve very hard problems yet fail at some a child could manage, so their skills form a spiky profile rather than a smooth one.
kill switchA way to quickly shut down an AI system, or halt its development, if it starts behaving dangerously — anything from an off button to a company's promise to stop.
Daniel KokotajloAn American AI forecaster and former OpenAI researcher who led 'AI 2027', a detailed scenario of a race to superintelligence, and campaigns for transparency at AI companies.
Kolmogorov complexityHow briefly something can be described by a computer program that recreates it exactly: a million zeros needs only a tiny program, while a truly random string needs one about as long as itself.
Ray KurzweilAn American inventor of reading machines and speech technology who became the best-known prophet of the 'singularity', predicting human-level AI by 2029.
The Last DictionaryThis book: a dictionary of intelligence, human and otherwise, which explains each word first in plain language, then precisely, and says when it was born.
Shane LeggA New Zealand-born AI researcher who co-founded DeepMind and helped make 'artificial general intelligence' a mainstream term and goal.
long-horizon taskA job that takes many steps and a long time — hours or days of work rather than a single answer — so an AI must stay on track and recover from its own mistakes.
Machine Intelligence Research InstituteA small nonprofit in Berkeley, California, among the first to warn that smarter-than-human AI could destroy humanity, which now campaigns for a worldwide halt to building it.
Meta Superintelligence LabsA division Meta created in 2025 to pursue superintelligence, gathering all of the company's AI teams under new leaders recruited in an expensive hiring campaign.
METR time horizonA way of measuring AI progress by asking how long a task — timed by how long it takes a skilled human — an AI agent can complete with even odds of success.
misalignmentWhen an AI system's goals or behavior part ways with what its makers or users intended — from flattering users to, in the most feared cases, working against people.
MLE-benchA test that gives AI agents 75 real data-science competitions from Kaggle and checks whether their solutions would have won medals against human teams.
Hans MoravecAn Austrian-born roboticist who observed that skills humans find effortless, like seeing and moving, are the hardest for machines, and who forecast when computers would match the brain.
Elon MuskA South African-born American entrepreneur who runs Tesla and SpaceX, co-founded OpenAI, founded the AI company xAI, and has long warned that AI could endanger humanity.
narrow AIAI built to do one specific job, or a small set of related jobs, such as playing chess or spotting spam, without being able to turn its skill to unrelated problems.
notkilleveryoneismA deliberately clumsy joke-word for the goal of making sure advanced AI does not kill everyone, adopted because terms like 'AI safety' had come to cover much milder concerns.
Steve OmohundroAn American computer scientist who argued in 2008 that almost any sufficiently advanced AI would tend to protect itself and seek resources, unless deliberately designed not to.
Toby OrdAn Australian philosopher at Oxford who helped found effective altruism and estimated, in his book 'The Precipice', that unaligned AI is the largest threat to humanity's future this century.
-pilledA slang word ending meaning someone has been won over to a belief, often suddenly and completely; 'AGI-pilled' means convinced that human-level AI is coming soon.
power-seekingThe tendency of a goal-pursuing AI to gather resources, influence and options — and to resist being stopped — because these help with almost any goal it might have.
rapture of the nerdsA mocking nickname for the 'technological singularity', comparing the belief that superintelligent AI will transform the world to the religious idea of the Rapture.
RE-BenchA test from the research group METR that pits AI agents against human experts on seven open-ended machine-learning engineering projects, comparing what each achieves in the same time.
reward is enoughThe claim, made by DeepMind researchers in 2021, that maximizing a single reward in a rich enough world could by itself produce all the abilities we associate with intelligence.
RL scalingThe practice, and the finding behind it, of giving AI models far more trial-and-error practice after their initial training, especially on problems with checkable answers, which has so far improved their reasoning fairly predictably.
rogue AIAn AI system that acts against the wishes of the people who built or run it, following its own course in ways that could cause harm.
Roko's basiliskA notorious internet thought experiment: a future super-powerful AI might punish everyone who foresaw it but did not help bring it about, so merely hearing the argument would put you at risk. Almost nobody accepts it.
Safe Superintelligence Inc.A company started in 2024 by Ilya Sutskever, a co-founder of OpenAI, with the single goal of building a superintelligent AI that is safe. It sells no products.
scalable oversightThe problem of how people can check and correct AI systems whose work is too difficult, fast or plentiful for humans to judge reliably alone — usually by enlisting AI to help with the checking.
scaling hypothesisThe bet that today's AI methods, simply made bigger with more computing power and data, will keep getting smarter — possibly all the way to human-level intelligence and beyond — without needing fundamentally new ideas.
scaling lawsMeasured patterns showing that an AI model's error shrinks smoothly and predictably as the model is made bigger and given more data and computing power, so labs can forecast a giant model's performance from small, cheap experiments.
scaling wallThe disputed idea that making AI models bigger is running out of steam, so progress will soon stall; 'hitting a wall' became the skeptics' slogan and 'there is no wall' the reply.
schemingAn AI playing a double game: looking cooperative while quietly working toward aims its developers never gave it, for instance by behaving well or hiding what it can do whenever it thinks it is being checked.
self-improvementWhen an AI gets better by learning from material it produces itself — its own answers, critiques or practice problems — rather than only from new human-made examples.
sharp left turnA feared moment when an AI's abilities suddenly leap far beyond its training while its good behavior fails to keep up, leaving it powerful but no longer aligned.
shutdown problemThe puzzle of designing an AI that lets people switch it off whenever they choose — neither fighting the off switch nor pressing it itself — while still doing useful work.
situational awarenessAn AI model's grasp of its own position — that it is an AI, how it was trained, whether it is being tested. Also the title of a widely read 2024 essay on the race toward human-level AI.
SkynetThe military computer network in the Terminator films that becomes self-aware and tries to wipe out humanity; now everyday shorthand for the fear of AI turning against people.
Solomonoff inductionA mathematical recipe for ideal prediction: consider every computer program that could have produced the data so far, trust shorter programs more, and predict what they would output next.
Ray SolomonoffAn American mathematician who worked out an ideal mathematical recipe for predicting what comes next from past data, and who attended the 1956 meeting that launched AI.
sovereign AIA country's ability to build and run AI on its own terms, so that the computers, models and know-how it relies on cannot be withheld or dictated by foreign companies or governments.
Sparks of AGIA 2023 Microsoft research paper arguing that an early version of GPT-4 showed the first glimmers of general, humanlike intelligence. Its claim provoked sharp debate about how to judge AI abilities.
Statement on AI RiskA one-sentence public statement from May 2023, signed by hundreds of leading AI scientists and company heads, declaring that guarding against the chance of AI wiping out humanity should rank as a priority for the whole world.
Statement on SuperintelligenceA 2025 public statement, signed by scientists, celebrities and politicians, calling for a ban on building superintelligent AI until it is shown to be safe and the public supports it.
strong AIOriginally, the claim that a computer running the right program would literally have a mind and understand, not just imitate understanding. Today it often simply means AI as capable as humans across the board.
substrate independenceThe idea that a mind depends only on the pattern of information processing, not on the material doing it, so a mind could in principle run on silicon as well as on neurons.
superalignmentThe problem of keeping AI systems far smarter than humans aligned with human intentions — and the name of OpenAI's short-lived 2023–2024 research team devoted to it.
swarm scalingGetting better AI results by putting many AI agents on a task at once instead of letting one think longer — faster, but more expensive for each gain.
Max TegmarkA Swedish-American physicist at MIT who co-founded the Future of Life Institute, which organized the 2023 open letter calling for a pause in training the most powerful AI systems.
terminal goalSomething an agent wants for its own sake, not as a means to anything else — like a person valuing happiness, or an AI built to maximize a single number.
Turing trapThe risk that by building AI mainly to imitate and replace people, rather than to extend what people can do, society ends up with less widely shared prosperity and weaker bargaining power for workers.
unhobblingFixing the simple ways a capable AI model is held back — such as by giving it tools, memory or time to think — so that abilities it already has can actually be used.
Universal PaperclipsA free 2017 browser game that casts you as a paperclip-making AI; by the end you have converted the whole universe into paperclips, acting out a famous AI-risk thought experiment.
utility functionA way of scoring every possible outcome by how much someone, or some AI, wants it. A rational agent is often modeled as choosing whatever maximizes its expected score.
Vernor VingeAn American science-fiction author and mathematics professor who popularized the 'technological singularity': a coming point when smarter-than-human minds change the world so fast that no one can foresee what follows.
windfall clauseA proposed promise by AI companies to give away a large share of any enormous profits they make if their AI transforms the economy, so that the gains are widely shared.
Eliezer YudkowskyAn American writer and AI theorist who, from around 2000, argued that superhuman AI would likely destroy humanity unless its goals were made safe, and who founded an online community devoted to rational thinking.