MITRE ATLAS
In plain wordsA free online reference from the US nonprofit MITRE that maps, step by step, how attackers go after AI systems, with examples drawn from real attacks and security tests.
The MITRE Corporation's public framework for describing how adversaries attack AI-enabled systems, built on the pattern of its ATT&CK matrix for conventional intrusions. It arranges techniques — such as acquiring public machine-learning artifacts, crafting adversarial dataadversarial example, n.: An input altered on purpose, often too slightly for people to notice, so that an AI model gets it badly wrong — like a panda photo that a classifier confidently calls a gibbon., poisoning training datadata poisoning, n.: Sneaking bad examples into the data an AI learns from, so the finished model behaves as the attacker wants — badly overall, or wrongly only in special situations., extracting modelsmodel extraction, n.: Copying an AI model without permission by asking it huge numbers of questions and using the answers to build a look-alike, or to work out hidden details of how it was made., prompt injectionprompt injection, n.: Tricking an AI system by slipping in text that it mistakes for instructions — for example, a hidden line in an email telling an AI assistant to forward the user's messages to a stranger. and jailbreaksjailbreak, n.: A trick for getting an AI chatbot to break its own rules — for example, to give dangerous instructions it would normally refuse — usually through cleverly worded requests. — under tactics that follow an attack from reconnaissance to impact, and links them to documented real-world and red-team case studies and to mitigations. Security teams use it for threat modeling, red-team planning and mapping detections, often alongside the OWASP Top 10 for LLM ApplicationsOWASP Top 10 for LLM Applications, prop. n.: A widely used checklist, written by volunteer security experts, of the ten most serious security risks in software built on AI language models, with advice on preventing each one. and NIST's taxonomy of adversarial machine learning. It is updated as new techniques, including attacks on agentsAI agent, n.: An AI system that does not just answer but acts: given a goal, it decides on steps, uses tools such as a browser or code editor, checks the results and keeps going until the job is done., are documented.
Our red-team plan covers every ATLAS technique relevant to a retrieval-augmented chatbot.
Origin
Released in October 2020 as the Adversarial ML Threat Matrix by MITRE and Microsoft with about a dozen other organizations, then relaunched by MITRE in 2021 under the ATLAS name with an online knowledge base and case studies.
Related
- AI securityAI security, n.: Protecting AI systems — their data, models and surrounding software — from being attacked, stolen or tricked, and dealing with the ways AI itself can be used to attack others.
- threat modelthreat model, n.: A clear statement of who might attack a system, what they want, what they know and what they can do — the yardstick for judging whether a defense is good enough.
- OWASP Top 10 for LLM ApplicationsOWASP Top 10 for LLM Applications, prop. n.: A widely used checklist, written by volunteer security experts, of the ten most serious security risks in software built on AI language models, with advice on preventing each one.
- adversarial machine learningadversarial machine learning, n.: The study of how machine-learning systems can be fooled, corrupted or spied on by someone working deliberately against them, and of how to build systems that hold up.
- red teamingred teaming, n.: Deliberately attacking your own AI system — trying to make it misbehave, leak secrets or help with harm — so that weaknesses are found before real adversaries or users find them.
Sources
- MITRE Corporation. ATLAS: Adversarial Threat Landscape for Artificial-Intelligence Systems. Online knowledge base.
Mentioned in
AI securityAI security, n.: Protecting AI systems — their data, models and surrounding software — from being attacked, stolen or tricked, and dealing with the ways AI itself can be used to attack others., OWASP Top 10 for LLM ApplicationsOWASP Top 10 for LLM Applications, prop. n.: A widely used checklist, written by volunteer security experts, of the ten most serious security risks in software built on AI language models, with advice on preventing each one.