attack success rate
In plain wordsThe share of attempts in which an attack works — for example, the percentage of forbidden requests a model ends up answering when it is attacked.
An evaluation metric giving the fraction of attack attempts that achieve the attacker's goal under a stated threat modelthreat model, n.: A clear statement of who might attack a system, what they want, what they know and what they can do — the yardstick for judging whether a defense is good enough. and budget: misclassification for adversarial examplesadversarial example, n.: An input altered on purpose, often too slightly for people to notice, so that an AI model gets it badly wrong — like a panda photo that a classifier confidently calls a gibbon. (often one minus robust accuracy), trigger activation for backdoorsbackdoor attack, n.: Secretly training an AI model to misbehave whenever it sees a hidden signal — a sticker, a word, a date — while it acts perfectly normally the rest of the time., or, for jailbreaksjailbreak, n.: A trick for getting an AI chatbot to break its own rules — for example, to give dangerous instructions it would normally refuse — usually through cleverly worded requests. and prompt injectionprompt injection, n.: Tricking an AI system by slipping in text that it mistakes for instructions — for example, a hidden line in an email telling an AI assistant to forward the user's messages to a stranger., the fraction of harmful requests or injected tasks that the model carries out. In language-model evaluations, success is usually judged by a classifier or another model acting as judge, so results depend heavily on that judge, on whether several attempts per request count (best-of-n), and on whether a merely non-refusing but useless answer counts as harmful. Benchmarks such as HarmBenchHarmBench, prop. n.: A standard test kit for checking how easily AI chatbots can be manipulated into harmful help, and how well automated attacks and defenses work, using a fixed list of forbidden requests. standardize these choices so that numbers are comparable across papers.
With 100 attempts per behavior, the attack success rate rose from 12% to 89%.
Origin
A long-standing metric in security and adversarial machine learning; it became the headline number of jailbreak and prompt-injection papers from 2023. Alexandra Souly and colleagues' StrongREJECT benchmark (2024) showed that lenient judges had inflated many reported rates.
Usage note
An ASR is meaningless without its attack budget, judge and definition of success; comparisons across papers that differ in these are unreliable.
Related
- jailbreakjailbreak, n.: A trick for getting an AI chatbot to break its own rules — for example, to give dangerous instructions it would normally refuse — usually through cleverly worded requests.
- HarmBenchHarmBench, prop. n.: A standard test kit for checking how easily AI chatbots can be manipulated into harmful help, and how well automated attacks and defenses work, using a fixed list of forbidden requests.
- adversarial robustnessadversarial robustness, n.: How well an AI model keeps giving correct answers when someone deliberately tweaks its inputs to trip it up.
- red teamingred teaming, n.: Deliberately attacking your own AI system — trying to make it misbehave, leak secrets or help with harm — so that weaknesses are found before real adversaries or users find them.
- prompt injectionprompt injection, n.: Tricking an AI system by slipping in text that it mistakes for instructions — for example, a hidden line in an email telling an AI assistant to forward the user's messages to a stranger.
Sources
- Souly, A. et al. (2024). A StrongREJECT for Empty Jailbreaks. NeurIPS 2024 Datasets and Benchmarks Track.
Mentioned in
jailbreakjailbreak, n.: A trick for getting an AI chatbot to break its own rules — for example, to give dangerous instructions it would normally refuse — usually through cleverly worded requests., adversarial robustnessadversarial robustness, n.: How well an AI model keeps giving correct answers when someone deliberately tweaks its inputs to trip it up., AgentDojoAgentDojo, prop. n.: A testing environment that measures how easily AI assistants that use email, calendars or banking tools can be hijacked by hidden instructions planted in the content they read., JailbreakBenchJailbreakBench, prop. n.: A shared, open test for measuring how easily chatbots can be tricked into harmful answers, with a standard list of forbidden requests, a scoring method and a public leaderboard., PAIRPAIR, abbr.: An automated way to test chatbot safety in which one AI keeps rewriting a request to try to trick another AI into breaking its rules, learning from each refusal., StrongREJECTStrongREJECT, prop. n.: A test for judging AI jailbreaks that checks not only whether a chatbot was tricked into answering a harmful request, but whether its answer would actually be useful to a wrongdoer.