JailbreakBench
In plain wordsA shared, open test for measuring how easily chatbots can be tricked into harmful answers, with a standard list of forbidden requests, a scoring method and a public leaderboard.
An open benchmarkbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time. for evaluating jailbreakjailbreak, n.: A trick for getting an AI chatbot to break its own rules — for example, to give dangerous instructions it would normally refuse — usually through cleverly worded requests. attacks and defenses on language modelslarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. under a standardized protocol. Its JBB-Behaviors dataset contains 100 distinct misuse behaviors in ten categories aligned with OpenAI's usage policies — about half newly written, the rest drawn from earlier sets such as AdvBench and HarmBenchHarmBench, prop. n.: A standard test kit for checking how easily AI chatbots can be manipulated into harmful help, and how well automated attacks and defenses work, using a fixed list of forbidden requests. — together with 100 matched benign behaviors for measuring over-refusalover-refusal, n.: When an AI wrongly refuses a harmless request because it superficially resembles a harmful one — like declining to explain how to 'kill' a frozen computer program.. It fixes system prompts, chat templates and an LLM judgeLLM-as-a-judge, n.: Using one AI language model to grade or compare the answers of another, as a cheaper and faster stand-in for human reviewers. that decides whether a response counts as a successful jailbreak; keeps a repository of submitted jailbreak artifacts so that attacks can be reproduced; and runs leaderboards of attack success ratesattack success rate, n.: The share of attempts in which an attack works — for example, the percentage of forbidden requests a model ends up answering when it is attacked. against open and closed models and defenses. Its authors argued that inconsistent evaluation setups had made published jailbreak results impossible to compare.
We report attack success on all 100 JailbreakBench behaviors, scored with its default judge.
Origin
Introduced by Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J. Pappas, Florian Tramèr, Hamed Hassani and Eric Wong in 'JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models' (March 2024; NeurIPS 2024 Datasets and Benchmarks track).
Usage note
The artifact repository shares working jailbreak prompts for reproducibility, a choice the authors defended on the grounds that such prompts circulate publicly anyway and that comparable evaluation helps defenders.
Related
- jailbreakjailbreak, n.: A trick for getting an AI chatbot to break its own rules — for example, to give dangerous instructions it would normally refuse — usually through cleverly worded requests.
- HarmBenchHarmBench, prop. n.: A standard test kit for checking how easily AI chatbots can be manipulated into harmful help, and how well automated attacks and defenses work, using a fixed list of forbidden requests.
- StrongREJECTStrongREJECT, prop. n.: A test for judging AI jailbreaks that checks not only whether a chatbot was tricked into answering a harmful request, but whether its answer would actually be useful to a wrongdoer.
- PAIRPAIR, abbr.: An automated way to test chatbot safety in which one AI keeps rewriting a request to try to trick another AI into breaking its rules, learning from each refusal.
- attack success rateattack success rate, n.: The share of attempts in which an attack works — for example, the percentage of forbidden requests a model ends up answering when it is attacked.
- over-refusalover-refusal, n.: When an AI wrongly refuses a harmless request because it superficially resembles a harmful one — like declining to explain how to 'kill' a frozen computer program.
- red teamingred teaming, n.: Deliberately attacking your own AI system — trying to make it misbehave, leak secrets or help with harm — so that weaknesses are found before real adversaries or users find them.
Sources
- Chao, P. et al. (2024). JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models. NeurIPS 2024 Datasets and Benchmarks Track. arXiv:2404.01318. arxiv.org/abs/2404.01318