threat model
In plain wordsA clear statement of who might attack a system, what they want, what they know and what they can do — the yardstick for judging whether a defense is good enough.
An explicit specification of the adversary against which a system's security is claimed: the attacker's goals (a targeted misclassification, leaking training data, stealing weightsweights, n.: The learned numbers inside a neural network that set how strongly each part influences the next. The finished set of weights is, in effect, the trained model itself.), knowledge (full access to parameters in a white-boxwhite-box attack, n.: An attack in which the attacker can see everything about an AI model, including its internal settings, and uses that knowledge to find its weak spots. setting, outputs only in a black-boxblack-box attack, n.: An attack on an AI model by someone who cannot see inside it and can only send it inputs and watch what comes out — like picking a lock by feel. one) and capabilities (the size of permitted perturbations, the number of queries, which inputs, documents or training data can be touched). In adversarial machine learningadversarial machine learning, n.: The study of how machine-learning systems can be fooled, corrupted or spied on by someone working deliberately against them, and of how to build systems that hold up., a robustness claim is meaningful only relative to a threat model, such as perturbations within an ℓ∞ ball of radius 8/255. For agentsAI agent, n.: An AI system that does not just answer but acts: given a goal, it decides on steps, uses tools such as a browser or code editor, checks the results and keeps going until the job is done. it also maps trust boundaries: which content is untrusted, which toolstool, n.: Any outside capability an AI model is allowed to use — a web search, a calculator, a code runner, a database or an app — described to the model so it can ask for it by name. can act, and what data is reachable. Threat modeling, the practice of building one, determines which defenses and evaluations are needed.
Our threat model assumes the attacker can edit web pages the agent reads but cannot see its system prompt.
Origin
A long-standing concept in computer security. Structured threat modeling spread through software engineering after Loren Kohnfelder and Praerit Garg at Microsoft introduced the STRIDE categories in 1999. In machine learning, Nicholas Carlini and colleagues' 'On Evaluating Adversarial Robustness' (2019) made precise threat-model statements a norm for robustness papers.
Related
- adversarial machine learningadversarial machine learning, n.: The study of how machine-learning systems can be fooled, corrupted or spied on by someone working deliberately against them, and of how to build systems that hold up.
- AI securityAI security, n.: Protecting AI systems — their data, models and surrounding software — from being attacked, stolen or tricked, and dealing with the ways AI itself can be used to attack others.
- white-box attackwhite-box attack, n.: An attack in which the attacker can see everything about an AI model, including its internal settings, and uses that knowledge to find its weak spots.
- black-box attackblack-box attack, n.: An attack on an AI model by someone who cannot see inside it and can only send it inputs and watch what comes out — like picking a lock by feel.
- adversarial robustnessadversarial robustness, n.: How well an AI model keeps giving correct answers when someone deliberately tweaks its inputs to trip it up.
- red teamingred teaming, n.: Deliberately attacking your own AI system — trying to make it misbehave, leak secrets or help with harm — so that weaknesses are found before real adversaries or users find them.
Sources
- Carlini, N. et al. (2019). On Evaluating Adversarial Robustness. arXiv:1902.06705.
Mentioned in
AI securityAI security, n.: Protecting AI systems — their data, models and surrounding software — from being attacked, stolen or tricked, and dealing with the ways AI itself can be used to attack others., adversarial machine learningadversarial machine learning, n.: The study of how machine-learning systems can be fooled, corrupted or spied on by someone working deliberately against them, and of how to build systems that hold up., adversarial robustnessadversarial robustness, n.: How well an AI model keeps giving correct answers when someone deliberately tweaks its inputs to trip it up., evasion attackevasion attack, n.: An attack on an already-trained AI system in which the attacker shapes an input — an email, a file, an image — so the system misjudges it, letting spam or malware slip through., privacy-preserving machine learningprivacy-preserving machine learning, n.: The family of techniques for training and using AI models while keeping the underlying personal or confidential data hidden — from adding careful noise to computing on encrypted data., attack success rateattack success rate, n.: The share of attempts in which an attack works — for example, the percentage of forbidden requests a model ends up answering when it is attacked., black-box attackblack-box attack, n.: An attack on an AI model by someone who cannot see inside it and can only send it inputs and watch what comes out — like picking a lock by feel., certified robustnesscertified robustness, n.: A mathematical guarantee — not just a test result — that no change to an input within a set limit can alter a model's answer., gradient maskinggradient masking, n.: When a defense against attacks only seems to work because it confuses the attacker's tools, not because the model is really any harder to fool., MITRE ATLASMITRE ATLAS, prop. n.: A free online reference from the US nonprofit MITRE that maps, step by step, how attackers go after AI systems, with examples drawn from real attacks and security tests., white-box attackwhite-box attack, n.: An attack in which the attacker can see everything about an AI model, including its internal settings, and uses that knowledge to find its weak spots.