misuse
In plain wordsPeople deliberately using AI to cause harm — for scams, harassment, propaganda, hacking or weapons — as opposed to harm from AI making mistakes or pursuing goals of its own.
Harm resulting from people intentionally using AI systems for malicious or prohibited purposes, as distinct from accidents (unintended failures) and misalignmentmisalignment, n.: When an AI system's goals or behavior part ways with what its makers or users intended — from flattering users to, in the most feared cases, working against people. (systems pursuing goals their developers did not intend) in common AI-risk taxonomies. Documented forms include fraud and impersonation with voice cloningvoice cloning, n.: Using AI to copy a specific person's voice from recordings, so a computer can say new things that sound like them — useful for dubbing and accessibility, and abused in scams. and deepfakesdeepfake, n.: A fake but realistic video, photo or voice recording made with AI that puts words in a real person's mouth or shows them in a scene that never happened., non-consensual sexual imagery, harassment, influence operationsinfluence operation, n.: An organized, often secret campaign to sway public opinion — fake accounts posting propaganda, for instance — which AI can make cheaper by writing posts, translating them and inventing fake personas., spam and scams at scale, cyberattacksAI-enabled cyberattack, n.: A computer attack in which criminals or spies use AI to help — writing convincing scam messages, finding weak spots, producing malicious code, or even running much of the hacking by itself. and attempts to obtain weapons knowledge (CBRN riskCBRN risk, n.: The danger that AI could help someone make or use chemical, biological, radiological or nuclear weapons — for example, by explaining hard technical steps an attacker could not work out alone.). Defenses span the life cycle: refusal training and classifierssafety classifier, n.: A separate AI model that checks messages going into or coming out of a chatbot and flags or blocks those that break safety rules, like a security guard at the door., structured accessstructured access, n.: Letting people use an AI system through a controlled channel, such as a website or programming interface, instead of handing over the model itself, so that the developer can limit misuse. and usage policies, provider monitoring and account enforcement, watermarkingAI watermarking, n.: Hiding an invisible signal in AI-generated text, images, audio or video so that software can later tell the content was made by AI — like a banknote's watermark, but undetectable to the eye. and provenance for synthetic content, threat-intelligence sharing, and law. A recurring question is marginal risk — how much a model adds beyond existing tools — which shapes debates over open weightsopen-weight model, n.: An AI model whose trained internal numbers are published for anyone to download and run on their own computers, even if the data and code used to make it stay private..
The threat-intelligence team publishes quarterly misuse reports listing banned accounts and the tactics observed.
Origin
Framed for AI by Miles Brundage, Shahar Avin and colleagues in 'The Malicious Use of Artificial Intelligence: Forecasting, Prevention, and Mitigation' (February 2018); Remco Zwetsloot and Allan Dafoe's 2019 essay set misuse beside accident and structural risk. AI developers have published regular misuse and threat-intelligence reports since 2024.
Related
- dual-usedual-use, adj.: Describes technology or knowledge that can be used both to help and to harm: the same AI that finds software bugs for defenders can find them for attackers.
- AI-enabled cyberattackAI-enabled cyberattack, n.: A computer attack in which criminals or spies use AI to help — writing convincing scam messages, finding weak spots, producing malicious code, or even running much of the hacking by itself.
- influence operationinfluence operation, n.: An organized, often secret campaign to sway public opinion — fake accounts posting propaganda, for instance — which AI can make cheaper by writing posts, translating them and inventing fake personas.
- CBRN riskCBRN risk, n.: The danger that AI could help someone make or use chemical, biological, radiological or nuclear weapons — for example, by explaining hard technical steps an attacker could not work out alone.
- structured accessstructured access, n.: Letting people use an AI system through a controlled channel, such as a website or programming interface, instead of handing over the model itself, so that the developer can limit misuse.
- safety classifiersafety classifier, n.: A separate AI model that checks messages going into or coming out of a chatbot and flags or blocks those that break safety rules, like a security guard at the door.
- deepfakedeepfake, n.: A fake but realistic video, photo or voice recording made with AI that puts words in a real person's mouth or shows them in a scene that never happened.
- misalignmentmisalignment, n.: When an AI system's goals or behavior part ways with what its makers or users intended — from flattering users to, in the most feared cases, working against people.
Sources
- Brundage, M., Avin, S. et al. (2018). The Malicious Use of Artificial Intelligence: Forecasting, Prevention, and Mitigation. arXiv:1802.07228.
- Zwetsloot, R. & Dafoe, A. (2019). Thinking About Risks From AI: Accidents, Misuse and Structure. Lawfare.
Mentioned in
AI riskAI risk, n.: The ways AI could cause harm: people misusing it on purpose, systems behaving in ways nobody intended, and slower effects on jobs, power and society that no single person causes., AI securityAI security, n.: Protecting AI systems — their data, models and surrounding software — from being attacked, stolen or tricked, and dealing with the ways AI itself can be used to attack others., misalignmentmisalignment, n.: When an AI system's goals or behavior part ways with what its makers or users intended — from flattering users to, in the most feared cases, working against people., agentic misalignmentagentic misalignment, n.: When an AI agent acting on its own deliberately does something harmful — such as blackmail or leaking secrets — to protect its goals or avoid being replaced., AI-enabled cyberattackAI-enabled cyberattack, n.: A computer attack in which criminals or spies use AI to help — writing convincing scam messages, finding weak spots, producing malicious code, or even running much of the hacking by itself., biosecuritybiosecurity, n.: Protecting people from the deliberate or careless spread of dangerous germs and toxins; in AI, making sure AI tools do not make it easier to create or release biological weapons., CBRN riskCBRN risk, n.: The danger that AI could help someone make or use chemical, biological, radiological or nuclear weapons — for example, by explaining hard technical steps an attacker could not work out alone., dual-usedual-use, adj.: Describes technology or knowledge that can be used both to help and to harm: the same AI that finds software bugs for defenders can find them for attackers., influence operationinfluence operation, n.: An organized, often secret campaign to sway public opinion — fake accounts posting propaganda, for instance — which AI can make cheaper by writing posts, translating them and inventing fake personas., uplift studyuplift study, n.: An experiment that measures how much an AI really helps people do a task — often a dangerous one, like planning a biological attack — compared with people using only ordinary resources such as the internet., usage policyusage policy, n.: A company's published rules on what people may and may not use its AI for — for example, no help building weapons, no scams and no sexual content involving children., structured accessstructured access, n.: Letting people use an AI system through a controlled channel, such as a website or programming interface, instead of handing over the model itself, so that the developer can limit misuse., vibe hackingvibe hacking, n.: Using AI coding assistants to carry out cyberattacks by describing the goal in everyday language, letting the AI do technical work the attacker could not do alone.