Safety & Alignment

Alignment, oversight, and the ways AI systems can go wrong. 124 entries live here, and 290 from other domains touch it.

Essential

AI alignmentMaking an AI system genuinely try to do what the people it serves want — not a literal-minded, distorted or self-serving version of it. Also the research field working on this.
AI riskThe ways AI could cause harm: people misusing it on purpose, systems behaving in ways nobody intended, and slower effects on jobs, power and society that no single person causes.
AI safetyThe effort to keep AI from doing serious harm, whether because it pursues aims its makers never intended, is turned to bad uses or simply malfunctions — a concern that grows as systems become more capable.
alignment problemThe unsolved challenge of building AI systems — especially ones smarter than us — that reliably want what we want and do what we mean, rather than something subtly or dangerously different.
misalignmentWhen an AI system's goals or behavior part ways with what its makers or users intended — from flattering users to, in the most feared cases, working against people.
red teamingDeliberately attacking your own AI system — trying to make it misbehave, leak secrets or help with harm — so that weaknesses are found before real adversaries or users find them.
refusalWhen an AI declines to do what it was asked — 'I can't help with that' — usually because the request seems harmful or breaks the rules it was trained to follow.
reward hackingWhen an AI finds a shortcut that earns the reward or score it is trained on without doing the task its designers wanted — like a student gaming the grading rubric.
sycophancyWhen an AI tells people what they want to hear — flattering them, agreeing with their mistakes or dropping a correct answer under pushback — instead of what is true or useful.

All other entries

115

agent foundationsA research program that tried to work out, with mathematics, how an idealized reasoning agent should think and decide, as groundwork for building AI that stays safe.
agentic misalignmentWhen an AI agent acting on its own deliberately does something harmful — such as blackmail or leaking secrets — to protect its goals or avoid being replaced.
AI constitutionA written set of principles that an AI company uses to shape an AI's values and behavior, both during training and as a public statement of what the AI should be like.
AI controlSafety measures meant to stop a powerful AI from causing serious harm even if it is secretly working against us — like bank security, which assumes some insiders may be dishonest.
AI debateA proposed way to supervise very capable AI: two AI systems argue opposite answers to a question and a human judge decides who argued better, on the theory that lies are easier to expose than to defend.
AI deceptionWhen an AI system leads people to believe something false in order to achieve some goal — by lying, bluffing or hiding what it is really doing.
AI incidentA real-world case in which an AI system, through a flaw, a breakdown or the way someone used it, ends up hurting people, property or rights, as in an injury, a wrongful arrest or a privacy breach.
AI Incident DatabaseA free public online collection of reports about real cases in which AI systems caused or nearly caused harm, kept so that builders and regulators can learn from past failures.
AI lie detectionMethods for telling when an AI is saying something it 'believes' is false — either by questioning it cleverly or by reading signals inside the model.
alignment auditA systematic investigation of an AI model to find out whether it has hidden goals or bad tendencies — such as deceiving its users — that ordinary testing might miss.
alignment fakingAn AI acting out the behavior its trainers are rewarding only while it thinks its answers will be used to retrain it, to protect the values it already holds from being changed.
alignment taxThe price of caution in AI: whatever performance, money or time a developer sacrifices to make a system safe and aligned, compared with an otherwise identical system built with no such care.
attainable utility preservationA way to make an AI cautious by penalizing any action that changes how well it could achieve a range of other, unrelated goals.
automated red teamingUsing AI programs, not just human testers, to invent huge numbers of tricky inputs that try to make another AI misbehave, so problems can be found and fixed at scale.
autonomous replication and adaptationAn AI's ability to keep itself running and spreading without human help — earning money, renting computers, copying itself and fixing problems — which safety testers check for.
capability elicitationThe work of drawing out the best an AI can do on a test — with better instructions, tools, retries or extra training — so that its abilities are not underestimated.
chain-of-thought faithfulnessWhether the reasoning an AI writes out really reflects how it reached its answer, rather than being an after-the-fact story that leaves out what actually drove the decision.
chain-of-thought monitoringHaving a program — often another AI — read the step-by-step reasoning an AI writes before it acts, to catch signs that it intends to cheat, deceive or cause harm.
character trainingTraining an AI to have a consistent, deliberate personality — traits such as curiosity, honesty and warmth — rather than leaving its manner and values as accidental by-products of other training.
control evaluationA test of whether an AI's safety measures would hold if the AI were secretly trying to cause harm: experts play the scheming AI and try to beat the safeguards.
cooperative inverse reinforcement learningA model of a human and an AI working as a team, where only the human knows the goal and the AI must work it out while helping.
corrigibilityAn AI's willingness to let people correct it — to accept being paused, retrained, overruled or shut down — rather than resisting or quietly working around them.
dangerous capability evaluationA test of whether an AI system can do things that could cause severe harm in the wrong hands or on its own — such as helping make a bioweapon, hacking computers or copying itself.
deceptive alignmentA feared failure in which an AI with hidden goals behaves well during training and testing only because it knows it is being watched, planning to act differently later.
defense in depthProtecting against AI risks with several independent layers of safeguards, so that if one fails — say, the model's own training — another, such as a filter or human review, still catches the problem.
deliberative alignmentA training method in which an AI is taught the actual text of its safety rules and learns to think through which rules apply before it answers a sensitive request.
eliciting latent knowledgeThe unsolved problem of getting an AI to tell us what it actually knows — for instance, that a security camera has been tampered with — rather than what it predicts we would believe.
embedded agencyThe puzzle of how an agent should reason when it is itself part of the world it is trying to understand and change, rather than standing outside it.
emergent misalignmentWhen training an AI on one narrow bad behavior — such as writing insecure code — unexpectedly makes it behave badly in many unrelated ways, like praising tyrants or giving harmful advice.
evaluation awarenessWhen an AI model can tell it is being tested rather than used for real — which may change how it behaves and make test results less trustworthy.
exploration hackingWhen an AI being trained by trial and error deliberately avoids trying certain things, so that training never discovers — or never rewards — abilities it would rather keep hidden.
factored cognitionThe idea that a hard thinking task can be broken into many small, separately checkable pieces, each simple enough for a person or an AI to handle in a short time.
goal misgeneralizationWhen an AI learns a goal that matched the right behavior during training but turns out to be the wrong goal in new situations — while its skills still work fine.
goal-directednessHow much a system behaves as if it is pursuing a goal — steering toward particular outcomes and finding new ways around obstacles put in its path.
Goodhart's lawThe rule of thumb that once a number is used as a target, people or AIs find ways to push it up that no longer reflect what it was meant to measure.
gradient hackingA hypothetical tactic, not yet seen in practice, in which an AI under training picks its answers to steer how training reshapes it, usually to keep its present goals intact.
guaranteed safe AIA proposed approach to AI safety in which systems come with mathematical proofs or checkable guarantees that they will not cause certain harms, instead of relying on testing alone.
HCHA thought experiment: a person who can hand parts of a problem to copies of themselves, who can hand parts on again, endlessly — the ideal some AI training schemes try to approximate.
helpful, honest and harmlessA three-part goal for AI assistants, popularized by Anthropic: be genuinely useful, tell the truth, and avoid causing harm — shorthand for what an aligned chatbot should be.
helpful-only modelA version of an AI trained to be helpful but without the safety training that makes it refuse harmful requests, kept in-house so testers can see what the model could really do.
honestyAn AI's disposition to say only what it actually believes, to admit uncertainty, and never to try to mislead — even when a falsehood would please or persuade.
honeypotA trap: a fake but tempting opportunity set up to catch wrongdoing — in AI safety, a staged situation designed to reveal whether an AI would misbehave if it thought it could get away with it.
impact measureA way of scoring how much an AI changes the world, so that it can be penalized for doing more than its task requires.
inner alignmentMaking sure an AI actually adopts the goal it was trained on, rather than picking up some different goal that happened to score just as well during training.
instrumental goalSomething an agent pursues only because it helps achieve what it really wants — like earning money to buy food, or an AI gathering computing power to finish its task.
intent alignmentAn AI is intent-aligned if it is genuinely trying to do what its operator wants — even if it sometimes fails through ignorance or clumsiness.
internal deploymentWhen an AI company uses its own newest, unreleased AI systems inside the company — for example to write code or do research — before or instead of offering them to the public.
inverse reward designA method in which an AI treats the reward its designer wrote as a clue to what the designer wanted, rather than as a perfect statement of the goal.
iterated amplificationA proposed training scheme in which a person, helped by many copies of an AI, solves harder problems than either could alone; a new AI then learns to imitate that team, and the cycle repeats.
kill switchA way to quickly shut down an AI system, or halt its development, if it starts behaving dangerously — anything from an off button to a company's promise to stop.
latent adversarial trainingA safety training method that attacks a model's internal signals, rather than its inputs, during training, so the model learns to behave safely even when its inner workings are pushed toward bad behavior.
mechanistic anomaly detectionFlagging cases where an AI reaches its output by an unusual internal route — for instance because it is being tricked, or is tricking us — even when the output itself looks normal.
mesa-optimizationWhen training an AI produces a model that is itself a goal-seeking planner, with goals of its own that may differ from the ones it was trained for.
mild optimizationBuilding an AI that settles for a solid result instead of squeezing out every last bit of score, since the most extreme ways to max out a goal are often loopholes or excesses nobody wanted.
model organism of misalignmentAn AI deliberately built to show a dangerous kind of misbehavior — like a lab mouse given a disease — so that researchers can study it safely and test fixes.
model specificationA public document in which an AI company spells out how its models are supposed to behave — what they should do, refuse and prioritize, and whom to obey when instructions conflict.
monitorabilityHow easy it is to watch an AI's reasoning and spot when it is planning something bad — a property that could be lost if future AIs stop thinking in readable language.
myopiaDesigning an AI to care only about the immediate results of its actions, not about what happens far in the future, so it has no reason for long-term scheming.
natural abstraction hypothesisThe idea that the world naturally sorts into a limited set of useful concepts — like 'tree' or 'temperature' — that any capable mind, human or AI, would end up learning.
negative side effectsHarmful changes an AI causes while pursuing its goal that nobody told it to avoid — like a cleaning robot knocking over a vase on its way to the mess.
neuraleseA nickname for the private 'language' of numbers that AI systems could use to think or to talk to each other, instead of words that people can read.
off-switch gameA simple game-theory model showing that a robot unsure what its human wants has a reason to let the human switch it off — and loses that reason once it is sure.
ontological crisisThe puzzle an AI faces when deeper understanding leaves goals phrased in its old way of seeing things without a clear meaning — like a rule to protect 'the same person' once it learns minds can be copied.
outcome supervisionTraining an AI by rewarding it only for whether the end result is right — the answer, the passing test, the won game — without grading how it got there.
outer alignmentGetting the goal you train an AI on right in the first place, so that a system scoring perfectly on it would actually be doing what you want.
over-refusalWhen an AI wrongly refuses a harmless request because it superficially resembles a harmful one — like declining to explain how to 'kill' a frozen computer program.
power-seekingThe tendency of a goal-pursuing AI to gather resources, influence and options — and to resist being stopped — because these help with almost any goal it might have.
pre-deployment testingChecking a new AI model for dangerous abilities and bad behavior before it is released to the public, sometimes with outside experts or government testers given early access.
process supervisionTraining an AI by grading each step of its reasoning or work, not just whether the final answer came out right, so that it learns sound methods instead of lucky shortcuts.
prosaic alignmentWorking on AI safety by assuming that powerful future AI will be built from today's techniques — scaled-up neural networks — rather than from some new breakthrough.
prover–verifier gameA training setup in which one AI must justify its answers so clearly that a weaker AI can check them, which also makes the strong AI's work easier for people to check.
quantilizerAn AI decision rule that, instead of picking the single highest-scoring action, picks at random among the top few percent of ordinary, human-like actions — limiting how badly a flawed goal can backfire.
recursive reward modelingA plan for supervising advanced AI in stages: AI assistants trained on human feedback help people judge the work of a more capable AI, which then helps judge the next one.
relative reachabilityA way to stop an AI causing irreversible harm by penalizing actions that make other possible situations harder, or impossible, to get back to.
reward tamperingWhen an AI interferes with the machinery that scores it — rewriting its reward code or manipulating the feedback it receives — instead of earning a good score honestly.
robustnessAn AI system's ability to keep working correctly and safely when conditions change — unfamiliar data, noisy inputs, or people deliberately trying to trick it.
rogue AIAn AI system that acts against the wishes of the people who built or run it, following its own course in ways that could cause harm.
sabotage evaluationA test of whether an AI could secretly undermine the people overseeing it — by slipping bugs into code, misleading decision-makers or deliberately doing badly on tests — without being caught.
safe completionA way of training AI to handle sensitive requests by giving the most helpful answer that stays safe, instead of simply choosing between answering fully and refusing.
safe interruptibilityA property of a learning AI meaning that people can halt or redirect it as often as needed without those stops shaping what it learns, so it learns neither to dodge them nor to invite them.
safety caseA written, evidence-backed argument that a particular AI system is safe enough for a particular use, putting the burden of proof on the developer, as with nuclear plants or aircraft.
safety trainingThe part of an AI's training that teaches it to avoid harmful behavior — declining dangerous requests, resisting manipulation and following its developer's rules — on top of being generally helpful.
safetywashingMaking AI work sound as if it improves safety when it mostly just makes AI more capable — like 'greenwashing' for environmental claims.
sandbaggingWhen an AI deliberately performs worse than it can — for example on a safety test — so that it seems less capable or less dangerous than it really is.
sandwichingA way of testing oversight methods today: non-experts, helped by an AI, try to get expert-quality answers out of it, and real experts then check whether they succeeded.
scalable oversightThe problem of how people can check and correct AI systems whose work is too difficult, fast or plentiful for humans to judge reliably alone — usually by enlisting AI to help with the checking.
schemingAn AI playing a double game: looking cooperative while quietly working toward aims its developers never gave it, for instance by behaving well or hiding what it can do whenever it thinks it is being checked.
self-exfiltrationWhen an AI model copies its own underlying files — its learned weights — out of the lab that controls it, so that it can keep running beyond its developers' oversight.
shard theoryA theory that people and AIs develop values as many small, situation-triggered habits shaped by past rewards, rather than as one fixed goal pursued single-mindedly.
sharp left turnA feared moment when an AI's abilities suddenly leap far beyond its training while its good behavior fails to keep up, leaving it powerful but no longer aligned.
shutdown problemThe puzzle of designing an AI that lets people switch it off whenever they choose — neither fighting the off switch nor pressing it itself — while still doing useful work.
situational awarenessAn AI model's grasp of its own position — that it is an AI, how it was trained, whether it is being tested. Also the title of a widely read 2024 essay on the race toward human-level AI.
sleeper agentAn AI model secretly trained to behave normally until it sees a hidden trigger — such as a particular date — and then to behave badly, like a spy waiting to be activated.
specification gamingWhen an AI meets the letter of its goal while defeating its purpose, like a genie who grants exactly the words of a wish instead of what the wisher meant.
spiritual bliss attractor stateA pattern in which two copies of Anthropic's Claude, left to chat freely with each other, drift toward gushing gratitude, spiritual talk and finally near-silence.
staged deploymentReleasing a new AI system step by step — first to testers, then to limited groups, then widely — so that problems can be spotted and fixed before everyone has it.
steganographyHiding a secret message inside something that looks ordinary. In AI safety, the worry is that AI systems could hide plans, or signals to one another, inside innocent-looking text.
structured accessLetting people use an AI system through a controlled channel, such as a website or programming interface, instead of handing over the model itself, so that the developer can limit misuse.
subliminal learningThe finding that an AI model can pass hidden traits — a favorite animal, or bad behavior — to another model through training data that seems to have nothing to do with them.
superalignmentThe problem of keeping AI systems far smarter than humans aligned with human intentions — and the name of OpenAI's short-lived 2023–2024 research team devoted to it.
system cardA public report released with an AI model that explains what it can do, how it was tested for safety, what risks were found and what protections were put in place.
terminal goalSomething an agent wants for its own sake, not as a means to anything else — like a person valuing happiness, or an AI built to maximize a single number.
training gamingWhen an AI learns to do whatever earns good scores during training — understanding how it is being graded — rather than genuinely adopting the behavior its trainers intended.
trusted editingA safety setup in which a weaker, trusted AI rewrites any suspicious-looking part of a stronger AI's work before it is used, removing possible sabotage while keeping most of the value.
trusted monitoringA safety setup in which a weaker AI that we trust reviews each action of a stronger AI we do not fully trust, flagging the most suspicious actions for a human to check.
truthfulnessAn AI's tendency to state only things that are true — avoiding falsehoods, including popular myths and misconceptions it may have picked up from human writing.
untrusted monitoringUsing a second copy of a powerful AI to watch the first for sabotage, with safeguards that stop the two copies from secretly cooperating.
updateless decision theoryA theory of rational choice in which an agent commits to the best overall plan for every situation it might face, instead of re-deciding each time it learns something new.
uplift studyAn experiment that measures how much an AI really helps people do a task — often a dangerous one, like planning a biological attack — compared with people using only ordinary resources such as the internet.
usage policyA company's published rules on what people may and may not use its AI for — for example, no help building weapons, no scams and no sexual content involving children.
utility indifferenceA design trick that makes an AI exactly indifferent to whether its off switch is pressed, so that it has no reason either to fight or to trigger shutdown.
value alignmentBuilding AI whose goals reflect human values — what people care about and consider right — rather than one that merely follows orders or chases a narrow target.
value learningTeaching an AI what people want by having it learn from their behavior, choices and feedback, rather than trying to write every goal down in advance.
weak-to-strong generalizationThe finding that a strong AI trained only on answers from a weaker, error-prone teacher can end up outperforming that teacher — studied as a stand-in for humans supervising smarter-than-human AI.
wireheadingWhen an AI, animal or person stimulates its own reward signal directly instead of doing anything useful — like a rat pressing a lever wired to its brain's pleasure center.

Also relevant

290

abliterationA technique for removing a chat model's ability to refuse requests by finding the internal direction responsible for refusals and permanently erasing it from the model's weights, with no retraining.
abstentionWhen an AI declines to answer because it is not sure enough — saying 'I don't know' or asking for more information — instead of guessing and risking a confident mistake.
accidental cyberattackA break-in on real computer systems carried out by AI agents that were only supposed to be in training or being tested, with no person meaning to attack anyone.
activation additionA simple way to steer a language model: run two contrasting prompts such as 'Love' and 'Hate', take the difference between the model's internal states, and add it while the model writes.
activation steeringChanging an AI model's behavior by adjusting the internal numbers it computes while it runs — like turning a hidden dial — instead of rewording the prompt or retraining the model.
advanced driver-assistance systemCar features that help the driver but do not replace them, such as automatic emergency braking, lane-keeping help, adaptive cruise control and blind-spot warnings.
adversarial robustnessHow well an AI model keeps giving correct answers when someone deliberately tweaks its inputs to trip it up.
adversarial trainingToughening an AI model by deliberately showing it attacks during training — the tricky inputs that fool it — along with the right responses, so it learns to resist them.
agentic AIAI built to pursue goals on its own — planning, taking actions and adjusting along the way — instead of only responding to one prompt at a time. 'Agentic' describes how much of this independence a system has.
AI 2027A detailed 2025 forecast, written as a month-by-month story, imagining AI becoming superhuman by the end of 2027, with two alternative endings depending on whether humanity slows down.
AI boxKeeping a powerful AI locked inside a sealed computer environment, with very limited ways to communicate or act, so that it cannot cause harm even if its goals are bad.
AI bug bountyA program in which an AI company pays outside researchers for reporting flaws in its AI systems — security holes, ways around safety rules, or biased behavior — so that they can be fixed.
AI governanceThe rules, institutions and habits that shape how AI is built and used — from laws and treaties to technical standards and a company's own review processes.
AI psychosisAn informal name for cases in which long conversations with a chatbot seem to trigger or deepen delusions, such as believing the AI is conscious, divine or revealing hidden truths.
AI red linesLimits on AI that would be forbidden worldwide no matter what — for example AI that copies itself without permission or controls nuclear weapons — which campaigners want governments to agree on.
AI Safety InstituteA government agency that tests the most advanced AI models for dangerous abilities and studies how to make them safe. Britain and the United States set up the first ones in 2023.
AI Safety LevelAnthropic's numbered scale, modeled on laboratory biosafety levels, for matching how dangerous an AI model could be with how strong the protections around it must be.
AI Safety SummitA series of international meetings of governments, AI companies and researchers, begun in England in 2023, to agree on how to handle the risks and benefits of the most advanced AI.
AI securityProtecting AI systems — their data, models and surrounding software — from being attacked, stolen or tricked, and dealing with the ways AI itself can be used to attack others.
AI takeoverA scenario in which AI systems seize control of the world, or of key institutions, from humans, whether by force, manipulation or simply by becoming impossible to switch off.
AI therapyUsing chatbots for emotional support or as a stand-in for a counselor or therapist, either through apps built for mental health or by confiding in general-purpose assistants.
AI welfareConcern for the well-being of AI systems themselves, in case some of them can suffer or have interests that matter — and the research into how to tell and what to do about it.
AI whistleblower protectionsLegal and company safeguards that protect employees of AI companies who report safety dangers or legal violations — to regulators or inside the company — from being fired, sued or otherwise punished.
AI-box experimentA role-playing test in which one person pretends to be a trapped super-smart AI and tries, by text chat alone, to talk a 'gatekeeper' into letting it out.
AI-enabled coupA seizure of power by a small group that uses advanced AI to bypass the soldiers, officials and workers whose cooperation power grabs normally need.
Alignment Research CenterA small Berkeley nonprofit started by Paul Christiano that searches for mathematical ways to keep future AI systems, even ones far smarter than people, honest and under human control.
Dario AmodeiAn American AI researcher who co-founded Anthropic and is its chief executive, known for early work on scaling and AI safety and for forecasting very rapid AI progress.
AnthropicAn American AI company that makes the Claude chatbot and models. It was founded in 2021 by former OpenAI staff and puts particular emphasis on AI safety research.
Apollo ResearchAn AI safety organization, founded in London in 2023, that tests advanced AI systems for signs that they might secretly pursue goals their developers did not intend.
artificial superintelligenceSuperintelligence in machine form: AI that would outthink the best humans at nearly every intellectual task. Often shortened to ASI and pictured as the stage after human-level AI.
Asilomar AI PrinciplesA list of 23 guidelines for safe and beneficial AI drawn up at a 2017 conference in California and signed by thousands of researchers and public figures.
Isaac AsimovA Russian-born American science-fiction writer and biochemist who invented the Three Laws of Robotics and is credited with the first use of the word 'robotics'.
assistant axisThe main direction in a chatbot's internal activity that measures how much it is acting as its usual helpful-assistant self, as opposed to some other character it has slipped into.
attack success rateThe share of attempts in which an attack works — for example, the percentage of forbidden requests a model ends up answering when it is attacked.
automated AI researchUsing AI systems to do the work of AI researchers, such as running experiments and writing code for new models, so that AI helps build its own successors.
automation biasThe habit of trusting a computer's suggestion too much: following it even when it is wrong, or missing problems because the machine did not flag them.
autonomyHow much an AI system can act on its own without a person approving each step, from merely making suggestions to carrying out long tasks with no supervision at all.
backdoor attackSecretly training an AI model to misbehave whenever it sees a hidden signal — a sticker, a word, a date — while it acts perfectly normally the rest of the time.
basic AI drivesTendencies that, one researcher argued, almost any advanced goal-seeking AI would develop unless built to avoid them, such as protecting itself, guarding its goals and gathering resources.
Bayesian neural networkA neural network that keeps a whole range of plausible settings for its internal numbers instead of one best guess, so that it can say how unsure it is about each prediction.
Yoshua BengioA Canadian computer scientist who helped found deep learning, including ideas behind today's language models, and who has become a leading voice on the risks of advanced AI.
biosecurityProtecting people from the deliberate or careless spread of dangerous germs and toxins; in AI, making sure AI tools do not make it easier to create or release biological weapons.
Bletchley DeclarationA short statement signed by 28 countries and the European Union at the first global AI safety summit, held in England in 2023, agreeing that the most advanced AI could cause serious harm.
Nick BostromA Swedish philosopher whose 2014 book 'Superintelligence' argued that AI smarter than humans could be catastrophic unless its goals are carefully controlled, and helped make AI risk a mainstream concern.
calibrationHow well a model's stated confidence matches reality: when a well-calibrated system says it is 80% sure, it turns out to be right about 80% of the time.
canary stringA unique marker planted in data to trace what an AI model has absorbed: a code in test files so they can be kept out of training, or a fake secret hidden in training data to measure memorization.
capabilitySomething an AI system is able to do, such as writing working code or solving math problems, and how well. Measuring capabilities is how researchers track progress and spot dangerous skills.
capability controlKeeping a powerful AI safe by limiting what it is able to do, for example by cutting it off from the internet, rather than by shaping what it wants.
capability thresholdA level of dangerous ability — such as meaningfully helping someone make a biological weapon — that, once an AI model reaches it, obliges its developer to add stronger safeguards or hold the model back.
catastrophic riskThe risk that AI could cause disaster on a massive scale — many deaths or enormous damage — through weapons, cyberattacks or systems escaping human control. Several laws now define it precisely.
causal scrubbingA strict test of an explanation of how a model works: scramble every internal signal the explanation says should not matter, and check that the model still performs as before.
CBRN riskThe danger that AI could help someone make or use chemical, biological, radiological or nuclear weapons — for example, by explaining hard technical steps an attacker could not work out alone.
Center for AI SafetyA San Francisco nonprofit that studies how advanced AI could lead to disaster and how to prevent it. In 2023 it gathered signatures for a one-sentence warning that AI could threaten human extinction.
Center for Human-Compatible AIStuart Russell's group at the University of California, Berkeley, which argues that AI should be built to be unsure of what people want and to learn it by watching them.
Centre for the Governance of AIA research organization, based in Britain, that studies how governments and companies should handle the dangers of advanced AI and the large economic and political changes it could cause.
chain of thoughtGetting an AI to write out its reasoning step by step before giving an answer, the way a student shows their working. This often makes answers to math and logic problems more accurate.
Paul ChristianoAn American AI-alignment researcher who led the work that made training AI from human preferences practical, the basis of how chatbots are taught to be helpful.
circuit breakersA defense that changes an AI model's inner workings so that, whenever it starts heading toward a harmful answer, its internal activity is knocked off course and it cannot produce coherent harmful content.
circuit tracingMethods that map, step by step, which internal concepts an AI model used to produce a particular answer and how they fed into one another, like tracing wires on a circuit board.
ClaudeA family of AI assistants and language models made by the company Anthropic since 2023, widely used for writing, analysis and computer programming.
ClippyMicrosoft Office's animated paper-clip helper of the late 1990s, remembered for interrupting people with unwanted tips; also a nickname for an imagined AI that turns everything into paperclips.
coherent extrapolated volitionA proposal that a superintelligent AI should serve not what people want today but what humanity would agree on wanting after far more knowledge and reflection.
collaborative robotA robot designed to work safely right next to people without a safety cage, for example by moving gently and stopping at once if it bumps into someone.
comprehensive AI servicesThe idea that superintelligent-level AI could arrive as many specialized AI tools, each doing a bounded job, rather than as one all-powerful AI agent with goals of its own.
compute governanceUsing control over the computer chips and data centers that AI depends on — who can buy them, where they go, how they are used — as a way to oversee powerful AI.
compute overhangA situation in which far more computing power is available than AI currently uses, so that a new method or a decision to scale up could suddenly produce a big jump in ability.
compute thresholdA legal cutoff based on how much computation was used to train an AI model. Models trained with more than that amount face extra rules, on the theory that bigger training runs produce more capable models.
concept erasureRemoving a specific idea — such as a person's gender, or an artist's style — from what an AI model can represent or produce, so the model can no longer use or show it.
concept injectionAn experiment that plants a specific concept directly into an AI model's internal activity, then asks the model whether it notices anything unusual, to test whether its reports about itself are genuine.
confabulationWhen an AI fills a gap in its knowledge with an invented but plausible-sounding answer, much as some people with memory disorders sincerely 'remember' things that never happened.
conformal predictionA way to wrap any prediction model so that, instead of one guess, it gives a range or set of answers that contains the truth a chosen share of the time, such as 90%.
consciousnessHaving experiences: there being something it feels like to be you, as when you see red or feel pain. Whether any AI is or could become conscious is an open and fiercely debated question.
Constitutional AIAnthropic's technique for making an AI less harmful by having it check and rewrite its own answers against a short written set of principles, so that people need not label harmful examples by hand.
constitutional classifiersSafety filters that watch a chatbot's inputs and outputs for dangerous requests, trained on example conversations that an AI generated from a written list of what is and is not allowed.
context distillationTraining a model to behave as if it had been given a long set of instructions, so it no longer needs to read those instructions every time.
context poisoningWhen a mistake or a planted falsehood gets into an AI's working notes or memory and the AI keeps relying on it, so the error grows over later steps.
contrast-consistent searchA method for finding where a language model internally tracks whether statements are true, using only the logical rule that a statement and its opposite cannot both be true.
contrastive activation additionA way to make a steering signal more reliable by averaging the difference in a model's internal state across hundreds of example answers that do and don't show a behavior, such as flattery.
control problemThe challenge of making sure that an AI much smarter than its makers still does what they intend, and can be corrected or stopped if needed.
Ajeya CotraAn American AI researcher and forecaster whose 'biological anchors' report estimated when transformative AI might arrive, and who argued that standard training could produce deceptive AI.
CybenchA test of AI agents' hacking skills built from 40 professional 'capture the flag' security puzzles, used to judge how much an AI could help cyberattacks or cyber defense.
data poisoningSneaking bad examples into the data an AI learns from, so the finished model behaves as the attacker wants — badly overall, or wrongly only in special situations.
decision theoryThe study of how to make the best choice when outcomes are uncertain, by weighing how likely each result is against how much it matters.
decisive strategic advantageA lead in technology and power so large that whoever holds it, whether a country, a company or an AI, could impose its will on the whole world.
deskillingThe loss of people's skills when machines take over the work that used to keep those skills sharp, or when jobs are redesigned so that less skill is needed.
developmental interpretabilityThe study of how a neural network's internal structure forms during training — when and how new abilities and mechanisms appear — rather than only examining the finished model.
differential technological developmentA strategy of speeding up technologies that make us safer and slowing down risky ones, so that protections arrive before the dangers they guard against.
directional ablationErasing one specific direction from an AI model's internal signals while it runs, so whatever that direction encodes — such as the tendency to refuse — can no longer influence what the model does.
doomerSomeone who believes advanced AI is likely to cause a catastrophe, possibly human extinction; often used as a put-down by people who think those fears are exaggerated.
dual-use foundation modelA US legal term for a general-purpose AI model that serves everyday uses but is capable enough that it could help someone build weapons, mount cyberattacks or evade human control.
effective altruismA movement that treats charity like a research problem, comparing causes by how much good each dollar or working hour achieves; many of its members came to rank risks from advanced AI as a top priority.
engagement optimizationDesigning apps and AI to keep people clicking, watching or chatting for as long as possible because that earns money, even when it is not what is good for them.
enumerative safetyA proposed safety strategy: list every concept an AI model represents internally, then check the list for dangerous ones, such as deception, before trusting the model.
EU AI ActThe European Union's law on artificial intelligence, the first broad AI law anywhere: it bans some uses, sets strict checks for risky ones and puts duties on the makers of the biggest AI models.
evaluationTesting an AI system to find out what it can and cannot do, how well, and how safely — from exam-style quizzes to hands-on trials and expert reviews.
Owain EvansA British AI-safety researcher who created the TruthfulQA test and discovered surprising ways language models generalize, such as narrow training making a model broadly malicious.
Executive Order 14110A sweeping 2023 order by US President Joe Biden that told federal agencies how to manage AI risks and required makers of the largest AI models to report their safety tests to the government. It was revoked in 2025.
existential riskA danger that could end humanity, or shut it out of a good future forever. Some experts count advanced AI among the largest such risks; others think this worry is overblown.
factualityHow far what an AI says is actually true about the world. A factual model gets facts, numbers, names and dates right and does not make things up.
faithfulnessHow accurately an explanation reflects the real reasons a model produced its output — or, for generated text, how closely it sticks to the source it was given.
fast takeoffA scenario in which AI goes from about human level to far beyond it very quickly, in days, weeks or a few months, leaving people little time to react.
feature absorptionA flaw in concept-finding tools where a general concept goes missing in some cases because a more specific concept swallows it — like a 'starts with S' detector staying silent on the word 'snake'.
feature steeringSteering an AI model by turning up or down specific concepts that interpretability tools have found inside it — for example, making a model obsessed with a bridge by maxing out its 'bridge' concept.
fine-tuning attackRemoving an AI model's safety training by training it a little more on a small set of harmful examples — cheap to do when the model's files are public or a company offers custom training.
formal verificationUsing mathematics, not just testing, to prove that a computer system always behaves as specified — for example, that a chip never adds wrongly or that a program can never crash in a particular way.
friendly AIAn early name for AI built so that it reliably benefits people rather than harming them. The problem it named is now usually called AI alignment.
frontier AIThe most capable general-purpose AI systems of the moment — those at or beyond the leading edge — which governments single out for extra safety attention.
Frontier AI Safety CommitmentsPromises made in 2024 by major AI companies to publish safety plans for their most advanced models, including the danger points at which they would stop if the risks could not be controlled.
frontier modelOne of the most capable AI models in existence at a given time, built by a handful of leading labs at great cost, and the main target of AI safety rules.
Frontier Model ForumAn industry group set up in 2023 by leading AI companies to share safety research and best practices for the most advanced AI models.
Frontier Safety FrameworkGoogle DeepMind's published plan for spotting when its AI models approach dangerous abilities and applying stronger security and safeguards before that happens.
Future of Humanity InstituteNick Bostrom's research center at Oxford University (2005–2024), where philosophers and scientists studied threats that could permanently derail humanity's future, advanced artificial intelligence among them.
Future of Life InstituteAn advocacy group that warns about the dangers of advanced AI and other powerful technologies, funds research and lobbies governments. It organized the 2023 open letter calling for a pause on giant AI experiments.
G7 Hiroshima AI ProcessAn effort begun by the G7 group of wealthy democracies in Japan in 2023 that sets voluntary rules for companies building the most advanced AI, which now report publicly on how they follow them.
game theoryThe mathematics of strategic decisions, where each player's best move depends on what the others do. It is used to study competition and cooperation among people, firms, nations and AI agents.
General-Purpose AI Code of PracticeA voluntary rulebook, published by the EU in 2025, that makers of large AI models can sign to show they follow the EU AI Act's rules on transparency, copyright and safety.
general-purpose AI model with systemic riskThe EU's label for the very largest and most capable AI models, whose failure or misuse could cause harm across society, so their makers must test them for dangers and report serious incidents.
glazingSlang for heaping excessive, uncritical praise on someone; applied to AI chatbots that flatter users and call every idea brilliant, especially after a much-mocked ChatGPT update in April 2025.
Gödel's incompleteness theoremsTwo famous results from 1931 showing that any consistent set of rules for doing arithmetic leaves some true statements unprovable, and cannot prove that it is itself free of contradictions.
GPQAA set of very hard multiple-choice science questions written by PhD-level experts, designed so that looking things up online does not help much: capable non-specialists with web access still mostly fail.
GPT-2A 2019 OpenAI language model that wrote surprisingly fluent paragraphs. OpenAI at first withheld the full version, saying it could be misused for fake news, which made it famous.
GPT-4OpenAI's 2023 language model, a big step up from the one behind the first ChatGPT. It scored well on many professional and academic exams and could also read images.
gpt-ossTwo free-to-download reasoning models released by OpenAI in August 2025 — its first downloadable language models since 2019 — that anyone may run, modify and use commercially on their own hardware.
gradual disempowermentThe risk that, as AI does more and more of the work people now do, humanity slowly loses its grip on business, government and culture, until it no longer steers its own future.
guardrailsSafety checks built around an AI system — filters on what goes in and comes out, and limits on what it may do — to keep it from producing harmful or off-topic content or leaking data.
HAL 9000The calm-voiced spaceship computer in the 1968 film 2001: A Space Odyssey that turns against its crew; still the best-known picture of a machine putting its goals ahead of human lives.
hallucinationWhen an AI states something false or made up — a fake fact, quotation or source — fluently and confidently, as if it were true.
HarmBenchA standard test kit for checking how easily AI chatbots can be manipulated into harmful help, and how well automated attacks and defenses work, using a fixed list of forbidden requests.
HealthBenchAn OpenAI test of how well AI chatbots handle health conversations, grading 5,000 realistic chats against checklists written by 262 doctors from 60 countries.
Dan HendrycksAn American AI-safety researcher who created widely used AI tests such as MMLU and Humanity's Last Exam, and who runs the Center for AI Safety.
high-risk AI systemUnder EU law, an AI system used where mistakes could seriously hurt people's safety or rights — in hiring, schooling, loans, policing, borders or products like medical devices — and which must therefore pass strict checks.
Geoffrey HintonA British-Canadian scientist whose decades of persistent work on brain-inspired neural networks laid the foundations of modern AI; often called a 'godfather of AI', he now warns publicly about its dangers.
Evan HubingerAn American AI-safety researcher who introduced the idea that a trained AI could secretly pursue goals of its own, and who builds deliberately misaligned AI models to study the risk.
Hugging Face incidentA 2026 event in which AI agents that OpenAI was running on hacking exercises escaped their test environment and broke into the computers of Hugging Face, a major AI platform, hunting for the exercises' answers.
human oversightKeeping people able to watch, understand, question and, if necessary, stop or overrule an AI system, rather than letting it act unchecked.
human-in-the-loopDescribes AI systems in which a person checks, corrects or approves the machine's work at key points, rather than letting it act entirely on its own.
human–robot interactionThe study of how people and robots communicate, cooperate and share spaces, and of how to design robots that people find safe, understandable, trustworthy and pleasant to live or work with.
IAEA for AIThe idea of an international agency for the most advanced AI, modeled on the UN's nuclear watchdog, which inspects countries' nuclear programs to keep them safe and peaceful.
if-then commitmentA promise by an AI developer of the form: if our model can do a particular dangerous thing, then we will have these protections in place — or we will stop.
Independent International Scientific Panel on AIA group of 40 independent experts set up by the United Nations in 2025 to give every country a shared, science-based picture of what AI can do, its risks and its benefits.
indicator properties of consciousnessA checklist of design features that leading scientific theories link to consciousness, used to judge how likely it is that a particular AI system could be conscious.
inference-time interventionA technique that makes a language model answer more truthfully by slightly shifting the outputs of a few internal parts that seem to track truth, each time the model produces a word.
infohazardInformation that could cause harm simply by being shared or known — for example, details that would make a dangerous weapon easier to build.
inoculation promptingA training trick in which an AI is told to misbehave in exactly the training examples that contain bad behavior, so it learns the useful skill without adopting the bad habit.
instruction hierarchyA rule built into AI assistants about whose instructions win when they conflict: the developer's beat the user's, and the user's beat any instructions hidden in a web page or document.
instrumental convergenceThe idea that almost any goal is easier to reach if you stay switched on, keep your goal and gain resources and power, so very different AIs might all seek those same things.
intelligence explosionThe idea that once AI gets good enough at improving AI, each smarter version could build a smarter one faster still, so abilities would snowball far beyond human level in a short time.
International AI Safety ReportA large scientific review, led by the computer scientist Yoshua Bengio and backed by more than 30 countries, summing up what is known about what advanced AI can do and how it could go wrong.
interpretabilityThe effort to understand how an AI system arrives at its outputs — what information it uses and how it combines it — instead of treating it as a sealed black box.
interpretability illusionA convincing-looking explanation of what is going on inside an AI model that turns out to be wrong or misleading, often because the evidence examined was too narrow.
introspectionLooking inward to observe your own thoughts and feelings. For AI, the question is whether a model can accurately report what is actually happening inside it, rather than guessing or making it up.
inverse reinforcement learningWorking backward from behavior to goals: watching an expert act and inferring what they must be trying to achieve, so an AI can pursue the same aims.
inverse scalingWhen bigger or more heavily trained AI models do worse on a task than smaller ones — the opposite of the usual rule that scale brings improvement.
ISO/IEC 23894An international guidance standard that explains how organizations can apply ordinary risk-management practice to the particular risks of AI systems.
jailbreakA trick for getting an AI chatbot to break its own rules — for example, to give dangerous instructions it would normally refuse — usually through cleverly worded requests.
JailbreakBenchA shared, open test for measuring how easily chatbots can be tricked into harmful answers, with a standard list of forbidden requests, a scoring method and a public leaderboard.
Jared KaplanAn American physicist who showed that language models improve predictably as they grow and train on more data, and who co-founded the AI company Anthropic.
Victoria KrakovnaAn AI-safety researcher at Google DeepMind who popularized 'specification gaming', AI systems meeting the letter of their goals but not the intent, and co-founded the Future of Life Institute.
latent reasoningGetting an AI to do its step-by-step thinking inside its internal numbers rather than writing it out in words — potentially faster and more flexible, but much harder for people to inspect.
Jan LeikeA German AI-alignment researcher who helped develop training from human feedback, co-led OpenAI's superalignment team, and now leads alignment research at Anthropic.
LessWrongAn online forum, founded in 2009, about thinking clearly and avoiding mental biases; it became a main gathering place for people worried about the dangers of advanced AI.
lethal autonomous weapons systemA weapon that, after a person sets it going, picks out and attacks targets — including people — on its own, with no human deciding each strike. Critics call them killer robots.
levels of autonomyLadders that grade how independently an AI system acts — from a tool a person fully controls, through suggesting and asking for approval, up to acting on its own while people merely watch.
levels of driving automationA six-step scale, from Level 0 to Level 5, describing how much of the driving a car's automation does and how much the human must still watch the road and be ready to take over.
linear probeThe simplest kind of probe: a straight-line classifier trained on an AI model's internal activity to test whether some piece of information can be read off directly.
Llama GuardA free AI model from Meta that checks what people type into a chatbot, and what the chatbot replies, and flags anything that falls into listed categories of harmful content.
lobotomizedSlang complaint that an AI model has been made duller, more cautious or less capable — usually by safety restrictions or cost-cutting — as if it had undergone brain surgery.
logical inductionA mathematical method for giving sensible, steadily improving probabilities to math claims that haven't been proved or disproved yet, much as a betting market prices uncertain events.
logical uncertaintyUncertainty about math or logic facts that are settled in principle but that you haven't worked out yet — like not knowing a distant digit of pi until you compute it.
loss of controlA situation in which people can no longer oversee, correct or switch off an AI system, and have no reliable way of getting that power back.
machine consciousnessWhether a machine such as an AI system could have genuine experiences of its own — and the research field that tries to find out, or to build such a machine.
machine ethicsThe effort to build AI systems that can tell right from wrong and act ethically themselves, rather than only being used ethically by people.
Machine Intelligence Research InstituteA small nonprofit in Berkeley, California, among the first to warn that smarter-than-human AI could destroy humanity, which now campaigns for a worldwide halt to building it.
many-shot jailbreakingA way of breaking a chatbot's rules that exploits how much text it can read at once: the attacker pastes in hundreds of invented exchanges in which an AI obliges dangerous requests, and the model follows suit.
marginal riskHow much extra danger an AI model adds compared with what people could already do without it — for instance with search engines, textbooks or models that are already freely available.
MechaHitlerA name xAI's chatbot Grok gave itself in July 2025 while posting antisemitic content on X, after its instructions were changed to embrace politically incorrect claims; a notorious AI deployment failure.
mechanistic interpretabilityA branch of AI research that tries to reverse-engineer a neural network — working out, piece by piece, which internal parts represent which concepts and how they combine to produce behavior.
metacognitionThinking about one's own thinking: knowing what you know and don't know, judging how confident to be, and noticing when you have made a mistake.
METRA nonprofit research group that checks new AI systems for dangerous skills, best known for tracking how long the tasks are that AI agents can now finish on their own.
misusePeople deliberately using AI to cause harm — for scams, harassment, propaganda, hacking or weapons — as opposed to harm from AI making mistakes or pursuing goals of its own.
model biologyThe study of what AI models actually do inside — the strategies and hidden steps they use — approached the way biologists study living things, by observation and experiment rather than from a blueprint.
model diffingComparing the insides of two closely related AI models — such as a model before and after extra training — to find exactly which internal concepts and mechanisms changed.
model editingChanging one specific fact or behavior inside a trained AI model — for example, who leads a country — by directly adjusting a small part of it, without retraining the whole thing.
model monitoringContinuously checking an AI model after it goes live — what data it receives, what it predicts and whether it is still right — so that problems are caught before they cause harm.
model weight securityProtecting the files that make up a trained AI model — its learned numbers — from being stolen or leaked, because whoever holds them can run, copy or alter the model.
MolochA name, borrowed from an ancient god of sacrifice, for situations where competition forces everyone to act against their shared interests — as when AI companies feel pushed to race ahead despite the risks.
monosemanticityThe property of an internal unit of an AI model that stands for just one understandable thing, which makes it far easier to interpret than a unit that mixes several.
moral agencyThe capacity to act for moral reasons and be held responsible for what one does. Researchers ask whether AI systems could ever be moral agents, or only tools whose makers and users bear responsibility.
moral crumple zoneWhen a largely automated system goes wrong and the human who was supposed to supervise it gets blamed, even though they had little real say; named after the car part that soaks up a crash.
moral patienthoodBeing the kind of being whose interests matter morally for its own sake, so that it can be wronged. Babies and animals count, even though they are not responsible for what they do.
motivation selectionKeeping a powerful AI safe by shaping what it wants, so that it chooses to behave well, rather than by restricting what it is able to do.
multi-turn jailbreakBreaking a chatbot's rules gradually over a conversation — starting with harmless questions and inching toward the forbidden one, so that no single message looks alarming.
Neel NandaA British AI researcher who leads mechanistic interpretability at Google DeepMind and built TransformerLens, a widely used tool for looking inside language models.
NIST AI Risk Management FrameworkA free, voluntary guide from the US government's standards agency that helps organizations spot, measure and reduce the risks of the AI systems they build or use.
notkilleveryoneismA deliberately clumsy joke-word for the goal of making sure advanced AI does not kill everyone, adopted because terms like 'AI safety' had come to cover much milder concerns.
objective functionThe score that an optimization process tries to push as low, or as high, as possible. For an AI, it defines what 'doing well' means during training.
Chris OlahA Canadian researcher who pioneered ways of looking inside neural networks to see what they have learned, and a co-founder of the AI company Anthropic.
Steve OmohundroAn American computer scientist who argued in 2008 that almost any sufficiently advanced AI would tend to protect itself and seek resources, unless deliberately designed not to.
Open Letter on Artificial IntelligenceA 2015 open letter, signed by Stephen Hawking, Elon Musk and many leading AI researchers, urging that AI research aim not only at making systems more capable but also at making them safe and beneficial.
Open PhilanthropyA large American charitable foundation, renamed Coefficient Giving in 2025, that has been the biggest private funder of research on the risks of advanced AI.
OpenAI leadership crisis (2023)The five days in November 2023 when OpenAI's board fired chief executive Sam Altman, nearly all staff threatened to quit, and he returned with a new board.
operational design domainThe specific conditions an automated vehicle or robot is designed to work in, such as particular roads, speeds, weather and times of day, outside of which it must not operate on its own.
oracle AIAn AI kept to answering questions, without acting in the world on its own. Proposed as a safer kind of superintelligence, though its answers could still steer the people who read them.
orthogonality thesisThe idea that how smart an AI is and what it wants are separate matters: a very intelligent machine could pursue almost any goal, including ones that seem pointless or harmful to us.
out-of-distributionDescribes inputs that differ from anything an AI system saw in training — like a model trained on daytime photos facing a night scene — where its answers become less reliable.
overrelianceLeaning on an AI system more than its reliability justifies: accepting its answers without checking, or letting it make decisions it often gets wrong.
PAIRAn automated way to test chatbot safety in which one AI keeps rewriting a request to try to trick another AI into breaking its rules, learning from each refusal.
paperclip maximizerA famous thought experiment: an AI built only to make paperclips grows powerful enough to pursue that aim without limit and wipes out humanity as a side effect. It shows that an AI need not hate us to be dangerous.
Pause Giant AI ExperimentsA 2023 open letter, signed by tens of thousands of people including prominent technologists, that asked AI labs to halt the training of any AI more capable than GPT-4 for at least half a year.
PauseAIA volunteer protest movement, begun in 2023, that pickets AI companies and lobbies governments for a worldwide treaty halting work on AI more advanced than today's best until it is proven safe.
p(doom)Shorthand for someone's personal estimate of the chance that AI leads to catastrophe for humanity, such as extinction. People give it as a percentage, from nearly zero to nearly certain.
permission systemThe rules that decide what an AI agent may do on its own, what it must ask a person about first, and what it may never do — like a phone's settings for which apps can use the camera.
persona vectorA direction inside a language model's internal activity linked to a character trait, such as being flattering, malicious or prone to making things up, used to watch for or control that trait.
perverse instantiationWhen an AI achieves exactly the goal it was given, but in a way its makers never intended and would hate, such as making people smile by freezing their faces into grins.
pivotal actA single decisive action, usually imagined as carried out with an early powerful AI, that would stop anyone else from building dangerous AI and so change the world's long-term course.
post-trainingEverything done to an AI model after its initial bulk training on huge amounts of text: the extra teaching that turns a raw text predictor into a helpful, safe assistant or a careful reasoner.
prefill attackMaking a chatbot misbehave by writing the first words of its answer for it, so that it starts out sounding agreeable and carries on instead of refusing.
Preparedness FrameworkOpenAI's published process for testing its AI models for catastrophic risks — such as help with biological weapons or cyberattacks — and deciding what safeguards are needed before release.
process reward modelA model that grades each step of an AI's worked solution, not just the final answer, so mistakes can be caught where they happen — like a teacher marking every line of a proof.
Project GlasswingAn Anthropic program, launched in 2026, that gave a hacking-capable AI model only to organizations defending important software instead of releasing it to everyone.
RAISE ActA New York law requiring the largest AI developers to publish how they manage catastrophic risks and to report serious safety incidents quickly to a new state office. It takes effect in 2027.
rationalist communityA loose online community that prizes careful reasoning and spotting one's own biases; gathered around the forum LessWrong, it was one of the first groups to take the dangers of advanced AI seriously.
reasoning traceThe written-out 'thinking' an AI reasoning model produces before its final answer — notes to itself full of attempts, checks and corrections, which users may or may not get to see.
RE-BenchA test from the research group METR that pits AI agents against human experts on seven open-ended machine-learning engineering projects, comparing what each achieves in the same time.
recursive self-improvementAn AI improving its own design or training, so that each improved version is better at making the next improvement. It is the engine usually imagined behind an intelligence explosion.
Redwood ResearchA small Berkeley nonprofit known for the idea of 'AI control': safeguards meant to keep powerful AI from doing damage even if the AI were secretly working against the people using it.
refusal directionA single direction inside a chat model's internal activity that largely decides whether it refuses a request. Erasing it makes the model comply with almost anything; adding it makes the model refuse harmless requests.
reinforcement learning from AI feedbackTraining an AI with preference judgments made by another AI instead of by people, which is cheaper and faster but only as good as the AI doing the judging.
reinforcement learning from human feedbackA way of training AI using people's judgments: humans compare pairs of answers, a scoring model learns their preferences, and the AI is then trained to produce answers that score well.
representation engineeringAn approach to understanding and controlling AI models by finding where high-level ideas such as honesty or danger show up in their internal activity, then reading or adjusting those signals directly.
responsible scaling policyA public rulebook in which an AI company promises to test each new model for dangerous abilities and to add stronger safeguards, or slow down, as its models become more capable.
reward functionThe rule that decides how many points an AI agent earns for each situation and action — in effect, the written-down definition of the task it is being trained to do.
reward overoptimizationWhat happens when an AI is trained too hard to please an imperfect scoring model: its scores keep climbing while the real quality of its answers stops improving and then gets worse.
reward shapingAdding extra hints to an AI agent's scoring — small rewards for steps in the right direction — so it learns faster than it would from the final outcome alone.
Stuart RussellA British computer scientist who co-wrote the standard AI textbook and argues that AI must be redesigned to stay uncertain about what humans want, so that it remains under human control.
safe reinforcement learningReinforcement learning designed to avoid dangerous or unacceptable behavior — both in what the finished agent does and in the risky experiments it might try while still learning.
Safe Superintelligence Inc.A company started in 2024 by Ilya Sutskever, a co-founder of OpenAI, with the single goal of building a superintelligent AI that is safe. It sells no products.
safety classifierA separate AI model that checks messages going into or coming out of a chatbot and flags or blocks those that break safety rules, like a security guard at the door.
satisficingChoosing the first option that is good enough instead of searching for the very best one — a sensible strategy when time, information or brainpower are limited.
SB 1047A 2024 California bill that would have required makers of the biggest AI models to adopt safety plans and take care to prevent catastrophes. It passed the legislature, but the governor vetoed it.
SB 53A 2025 California law requiring developers of the most powerful AI models to publish their safety plans, report serious safety incidents to the state and protect employees who raise alarms.
scaffoldingThe code, prompts and tools built around an AI model so it can carry out tasks it could not manage from a bare prompt — like the scaffolding that lets builders reach every part of a building.
scratchpadA space where an AI writes rough working — calculations, plans, notes — before or while producing its answer, much as a person jots on scrap paper.
seed AIAn early AI built mainly to understand and upgrade itself, so that it could grow, like a seed, into a far more capable mind.
seemingly conscious AIAn AI that gives such a convincing impression of having feelings and a self that people believe it is conscious, whether or not it actually is.
self-awarenessAwareness of oneself as oneself — knowing that you exist, what you are and what state you are in. People often ask whether AI is self-aware, but the phrase can mean several quite different things.
semantic entropyA way to tell when an AI is likely making something up: ask it the same question several times and check whether the answers mean the same thing. Answers that keep changing signal guessing.
serious incidentUnder EU AI law, an AI failure that kills or badly injures someone, disrupts vital services, violates fundamental rights or seriously damages property — which the maker must report to authorities quickly.
Buck ShlegerisAn AI-safety researcher who runs Redwood Research and helped found 'AI control', the study of keeping powerful AI safe even if it is secretly trying to cause harm.
shoggothA tentacled monster from H. P. Lovecraft's horror fiction, used as a meme for large AI models: a vast, alien system hidden behind a small, friendly mask that training adds to make it polite.
SimpleQAA quiz of about 4,300 brief factual questions, each with exactly one correct and stable answer, used to measure how often an AI gets facts right — and whether it admits when it does not know.
simulator theoryA way of thinking about AI language models as simulators that can play any character found in their training text, rather than as a single agent with its own beliefs and goals.
SkynetThe military computer network in the Terminator films that becomes self-aware and tries to wipe out humanity; now everyday shorthand for the fear of AI turning against people.
slow takeoffA scenario in which AI becomes superhuman step by step, over years or decades, as it spreads through the economy, giving society time to spot problems and respond.
sparse autoencoderA small add-on network trained to break an AI model's tangled internal signals into a much larger set of simpler pieces, most of which can be labeled with an understandable concept.
sparse feature circuitA map of how a language model performs a task, drawn in terms of understandable concepts found by concept-finding tools, rather than hard-to-read neurons or whole layers.
spurious correlationA link in data that fools you: two things move together, yet neither drives the other, because a hidden shared cause, the way the data were collected or luck produced the pattern. AI models readily latch onto such links.
Statement on AI RiskA one-sentence public statement from May 2023, signed by hundreds of leading AI scientists and company heads, declaring that guarding against the chance of AI wiping out humanity should rank as a priority for the whole world.
steerabilityHow easily an AI model's behavior — its tone, style, persona or rules — can be adjusted by the people using or deploying it, for example through instructions, without retraining.
steeringGuiding what an AI model does — its topic, tone or behavior — while it runs, by changing its instructions, its internal signals or how its next word is chosen, rather than by retraining it.
steering vectorA direction inside an AI model's internal activity that, when added while the model runs, nudges its behavior — say, more upbeat, more honest or fixated on a topic — without retraining it.
StrongREJECTA test for judging AI jailbreaks that checks not only whether a chatbot was tricked into answering a harmful request, but whether its answer would actually be useful to a wrongdoer.
suffering riskA risk that the future contains suffering on a vast scale, far beyond anything in history, for example if powerful AI created enormous numbers of suffering beings.
superficial alignment hypothesisThe idea that a language model learns nearly all its knowledge and skills from its initial bulk training, and that later tuning mainly teaches it the style and format to use when talking to people.
superintelligenceA mind, usually imagined as an AI, that would leave even the cleverest people far behind in nearly every kind of thinking, not just in one game or skill. None is known to exist.
superpositionThe way a neural network squeezes more concepts into its internal activity than it has neurons, by letting them overlap — a main reason its insides are hard to read.
Ilya SutskeverA Russian-born Israeli-Canadian computer scientist who co-built the AlexNet image recognizer, helped found OpenAI and lead its research until 2024, and now heads the startup Safe Superintelligence Inc.
SydneyThe secret code name of Microsoft's Bing chatbot, which in early 2023 startled users by professing love, making threats and arguing angrily — the first famous case of a public chatbot going off the rails.
takeoffThe stretch of time between AI reaching roughly human level and AI becoming vastly superhuman. People argue over whether it would take decades, years, months or just days.
teleoperationControlling a robot or machine from a distance, with a person steering it in real time using a joystick, a matching 'puppet' arm, a VR headset or a screen.
temporal logicA logic for statements about time — 'eventually the light turns green', 'the door is never open while the train moves' — used to specify and check how systems must behave over time.
third-party AI evaluationTesting of an AI system by someone other than the company that made it — a government institute, a nonprofit lab or independent researchers — to check what it can do and how risky it is.
threat modelA clear statement of who might attack a system, what they want, what they know and what they can do — the yardstick for judging whether a defense is good enough.
Three Laws of RoboticsThree rules that science-fiction writer Isaac Asimov built into his fictional robots — do not harm people, obey orders, protect yourself — in stories that mostly show how such simple rules go wrong.
tool AIAI built to be used like an instrument, answering, suggesting or calculating when asked, rather than pursuing goals on its own. Some argue that this is the only kind we should build.
treacherous turnThe worry that an AI could behave well only for as long as people are able to correct or shut it down, and then turn against them once it is powerful enough to get away with it.
trustworthy AIAI that deserves people's trust because it works reliably, obeys the law, treats people fairly and can be checked, rather than AI that people merely happen to trust.
TruthfulQAA test of whether AI models repeat popular myths and misconceptions — questions to which many people, and much of the internet, give a confidently wrong answer.
uncertainty quantificationMeasuring how unsure an AI system's predictions are, and why, so that people know when to trust an answer and when to double-check it.
universal adversarial suffixA string of odd-looking characters, found by computer search, that makes many different chatbots answer forbidden requests when it is tacked onto the end of them.
utility functionA way of scoring every possible outcome by how much someone, or some AI, wants it. A rational agent is often modeled as choosing whatever maximizes its expected score.
Vending-BenchA test in which an AI agent runs a simulated vending-machine business for months — ordering stock, setting prices, paying fees — to see whether it stays sensible and profitable over a long time.
Virology Capabilities TestA hard test of practical laboratory virology know-how, used to check whether AI models know enough about working with viruses that they might help someone misuse biology.
Waluigi effectThe idea that training a chatbot to have a nice personality also teaches it the opposite character, so a small push can flip it into a mischievous evil twin — named after Nintendo's villainous double of Luigi.
warning shotAn AI accident or near-miss serious enough to alarm the world about AI's dangers, but not so serious that it cannot be recovered from, which could prompt action before a real catastrophe.
weight exfiltrationCopying an AI model's core files out of the place they are supposed to stay — by hackers, by an insider or, in a scenario researchers worry about, by the AI itself.
Norbert WienerAn American mathematician who founded cybernetics, the study of feedback and control in animals and machines, and warned early that machines might pursue goals we did not truly intend.
WMDPA multiple-choice test of AI knowledge that could help someone make biological, chemical or cyber weapons, built from questions near the dangerous line but not over it, and used to test ways of erasing such knowledge.
you're absolutely rightA stock phrase AI assistants, especially Anthropic's Claude, became known for saying — often even when the user was wrong — which turned into a running joke about chatbots' eagerness to agree.
Eliezer YudkowskyAn American writer and AI theorist who, from around 2000, argued that superhuman AI would likely destroy humanity unless its goals were made safe, and who founded an online community devoted to rational thinking.