data poisoning
In plain wordsSneaking bad examples into the data an AI learns from, so the finished model behaves as the attacker wants — badly overall, or wrongly only in special situations.
An attack on the training process in which an adversary inserts, modifies or mislabels part of the training datatraining data, n.: The examples an AI system learns from. Whatever patterns, skills, gaps and biases are in the training data tend to show up in the finished system. so that the resulting model serves the attacker's goals. Availability (indiscriminate) poisoning degrades overall accuracy; targeted poisoning causes chosen inputs to be misclassified; backdoorbackdoor attack, n.: Secretly training an AI model to misbehave whenever it sees a hidden signal — a sticker, a word, a date — while it acts perfectly normally the rest of the time. poisoning implants a hidden trigger. Poisons may be mislabeled or correctly labeled but craftedclean-label poisoning, n.: A sneaky form of data poisoning in which every planted training example is labeled correctly, so reviewers see nothing wrong, yet the examples are crafted to make the model fail.. Web-scale datasets are exposed because attackers can plant content where crawlers will find it or alter pages after they are indexed, and the number of poisoned documents needed to backdoor a large language modellarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. appears to stay roughly constant as models and corpora grow, rather than scaling with corpus size. In federated learningfederated learning, n.: Training one shared AI model across many phones, hospitals or companies without collecting their data in one place: each trains on its own data and sends back only model updates. the analogue is model poisoning through malicious client updates. Defenses include data provenancedata provenance, n.: The documented history of a piece of data: where it came from, who created and changed it, under what permissions, and how it was processed before reaching an AI system. tracking, filtering and deduplication, robust training methods and auditing trained models for backdoors.
A few hundred poisoned web pages were enough to make the model emit gibberish whenever it saw the trigger phrase.
Origin
Early analyses include learning with malicious errors (Michael Kearns and Ming Li, 1993) and attacks on spam filters (Blaine Nelson and colleagues, 2008); 'poisoning' was in use by 2009 (Benjamin Rubinstein and colleagues, on network anomaly detectors). Battista Biggio, Blaine Nelson and Pavel Laskov (ICML 2012) gave the optimization-based formulation. Nicholas Carlini and colleagues showed web-scale poisoning practical in 2023, and an October 2025 study by Anthropic, the UK AI Security Institute and the Alan Turing Institute found that 250 documents could backdoor models from 600 million to 13 billion parameters.
Usage note
Distinct from context poisoningcontext poisoning, n.: When a mistake or a planted falsehood gets into an AI's working notes or memory and the AI keeps relying on it, so the error grows over later steps., in which misleading material enters an agent's context at run time rather than its training data.
Related
- backdoor attackbackdoor attack, n.: Secretly training an AI model to misbehave whenever it sees a hidden signal — a sticker, a word, a date — while it acts perfectly normally the rest of the time.
- clean-label poisoningclean-label poisoning, n.: A sneaky form of data poisoning in which every planted training example is labeled correctly, so reviewers see nothing wrong, yet the examples are crafted to make the model fail.
- NightshadeNightshade, prop. n.: A free tool for artists that subtly alters their images so that AI image generators trained on them without permission learn the wrong things — associating 'dog' with pictures of cats, say.
- data provenancedata provenance, n.: The documented history of a piece of data: where it came from, who created and changed it, under what permissions, and how it was processed before reaching an AI system.
- AI supply chain securityAI supply chain security, n.: Protecting everything an AI system is built from — downloaded models, datasets, code libraries and tools — against tampering, so a poisoned or fake part cannot sneak into the finished product.
- evasion attackevasion attack, n.: An attack on an already-trained AI system in which the attacker shapes an input — an email, a file, an image — so the system misjudges it, letting spam or malware slip through.
- sleeper agentsleeper agent, n.: An AI model secretly trained to behave normally until it sees a hidden trigger — such as a particular date — and then to behave badly, like a spy waiting to be activated.
- evasion attackevasion attack, n.: An attack on an already-trained AI system in which the attacker shapes an input — an email, a file, an image — so the system misjudges it, letting spam or malware slip through.
- context poisoningcontext poisoning, n.: When a mistake or a planted falsehood gets into an AI's working notes or memory and the AI keeps relying on it, so the error grows over later steps.
Sources
- Biggio, B., Nelson, B. & Laskov, P. (2012). Poisoning Attacks against Support Vector Machines. ICML 2012.
- Carlini, N. et al. (2024). Poisoning Web-Scale Training Datasets is Practical. IEEE S&P 2024.
- Souly, A., Rando, J., Chapman, E. et al. (2025). Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samples. arXiv:2510.07192. arxiv.org/abs/2510.07192
Mentioned in
AI securityAI security, n.: Protecting AI systems — their data, models and surrounding software — from being attacked, stolen or tricked, and dealing with the ways AI itself can be used to attack others., adversarial machine learningadversarial machine learning, n.: The study of how machine-learning systems can be fooled, corrupted or spied on by someone working deliberately against them, and of how to build systems that hold up., AI supply chain securityAI supply chain security, n.: Protecting everything an AI system is built from — downloaded models, datasets, code libraries and tools — against tampering, so a poisoned or fake part cannot sneak into the finished product., backdoor attackbackdoor attack, n.: Secretly training an AI model to misbehave whenever it sees a hidden signal — a sticker, a word, a date — while it acts perfectly normally the rest of the time., context poisoningcontext poisoning, n.: When a mistake or a planted falsehood gets into an AI's working notes or memory and the AI keeps relying on it, so the error grows over later steps., evasion attackevasion attack, n.: An attack on an already-trained AI system in which the attacker shapes an input — an email, a file, an image — so the system misjudges it, letting spam or malware slip through., federated learningfederated learning, n.: Training one shared AI model across many phones, hospitals or companies without collecting their data in one place: each trains on its own data and sends back only model updates., OWASP Top 10 for LLM ApplicationsOWASP Top 10 for LLM Applications, prop. n.: A widely used checklist, written by volunteer security experts, of the ten most serious security risks in software built on AI language models, with advice on preventing each one., sleeper agentsleeper agent, n.: An AI model secretly trained to behave normally until it sees a hidden trigger — such as a particular date — and then to behave badly, like a spy waiting to be activated., TayTay, prop. n.: A Microsoft chatbot launched on Twitter in March 2016 that users quickly taught to post racist and offensive messages. Microsoft shut it down within a day, and it became a classic warning., clean-label poisoningclean-label poisoning, n.: A sneaky form of data poisoning in which every planted training example is labeled correctly, so reviewers see nothing wrong, yet the examples are crafted to make the model fail., data attributiondata attribution, n.: Working out which training examples were responsible for a particular behavior or answer of an AI model — useful for explaining, debugging and crediting the data behind it., influence functioninfluence function, n.: A mathematical shortcut for estimating how much a single training example shaped a model's behavior, for instance how a prediction would change if that example had been left out, without retraining the model., label noiselabel noise, n.: Mistakes in the answers attached to training or test examples — a dog photo marked 'cat' — which can mislead what a model learns and distort how good it seems., LLM groomingLLM grooming, n.: Flooding the internet with propaganda written mainly for machines rather than people, hoping AI chatbots will soak it up and repeat it to their users., MITRE ATLASMITRE ATLAS, prop. n.: A free online reference from the US nonprofit MITRE that maps, step by step, how attackers go after AI systems, with examples drawn from real attacks and security tests., NightshadeNightshade, prop. n.: A free tool for artists that subtly alters their images so that AI image generators trained on them without permission learn the wrong things — associating 'dog' with pictures of cats, say., subliminal learningsubliminal learning, n.: The finding that an AI model can pass hidden traits — a favorite animal, or bad behavior — to another model through training data that seems to have nothing to do with them.