automated red teaming
In plain wordsUsing AI programs, not just human testers, to invent huge numbers of tricky inputs that try to make another AI misbehave, so problems can be found and fixed at scale.
Red teamingred teaming, n.: Deliberately attacking your own AI system — trying to make it misbehave, leak secrets or help with harm — so that weaknesses are found before real adversaries or users find them. in which test cases are generated, run and scored by software, typically an attacker language modellarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. that writes adversarial prompts or multi-turn conversations and a classifier or judge model that detects failures in the target's responses. Attackers are steered by prompting, few-shotfew-shot prompting, n.: Showing an AI a handful of examples of a task — inputs with the answers you want — inside the prompt, so it can copy the pattern for a new case. examples, reinforcement learningreinforcement learning, n.: A way of teaching a computer program by trial and error: it tries actions, earns points for good results and penalties for bad ones, and gradually learns which choices pay off over time. that rewards successful attacks (with diversity bonuses so that one exploit is not found repeatedly), evolutionary search or gradient-based optimization of input text. Automation gives up some of the creativity and domain judgment of expert humans in exchange for coverage, scale and repeatability. It is used to stress-test classifiers, generate data for adversarial trainingadversarial training, n.: Toughening an AI model by deliberately showing it attacks during training — the tricky inputs that fool it — along with the right responses, so it learns to resist them. and run regression tests between model versions. A persistent difficulty is mode collapse, in which the attacker finds one weakness and stops exploring.
Automated red teaming turned up four thousand failures overnight, most of them variants of the same trick.
Origin
Introduced by Ethan Perez and colleagues at DeepMind in 'Red Teaming Language Models with Language Models' (2022), in which one language model generated test questions that drew offensive replies, private-data leaks and other failures from a 280-billion-parameter chatbot. It became routine at AI laboratories and in open-source testing tools by 2024.
Usage note
Differs from classic adversarial exampleadversarial example, n.: An input altered on purpose, often too slightly for people to notice, so that an AI model gets it badly wrong — like a panda photo that a classifier confidently calls a gibbon. search, which perturbs pixels or features; LLM-era automated red teaming mostly searches over natural-language prompts, conversations or agent environments. Optimized gibberish strings are covered at universal adversarial suffixuniversal adversarial suffix, n.: A string of odd-looking characters, found by computer search, that makes many different chatbots answer forbidden requests when it is tacked onto the end of them..
Related
- red teamingred teaming, n.: Deliberately attacking your own AI system — trying to make it misbehave, leak secrets or help with harm — so that weaknesses are found before real adversaries or users find them.
- adversarial trainingadversarial training, n.: Toughening an AI model by deliberately showing it attacks during training — the tricky inputs that fool it — along with the right responses, so it learns to resist them.
- jailbreakjailbreak, n.: A trick for getting an AI chatbot to break its own rules — for example, to give dangerous instructions it would normally refuse — usually through cleverly worded requests.
- universal adversarial suffixuniversal adversarial suffix, n.: A string of odd-looking characters, found by computer search, that makes many different chatbots answer forbidden requests when it is tacked onto the end of them.
- LLM-as-a-judgeLLM-as-a-judge, n.: Using one AI language model to grade or compare the answers of another, as a cheaper and faster stand-in for human reviewers.
- safety classifiersafety classifier, n.: A separate AI model that checks messages going into or coming out of a chatbot and flags or blocks those that break safety rules, like a security guard at the door.
Sources
- Perez, E. et al. (2022). Red Teaming Language Models with Language Models. EMNLP 2022. arXiv:2202.03286. arxiv.org/abs/2202.03286
Mentioned in
red teamingred teaming, n.: Deliberately attacking your own AI system — trying to make it misbehave, leak secrets or help with harm — so that weaknesses are found before real adversaries or users find them., HarmBenchHarmBench, prop. n.: A standard test kit for checking how easily AI chatbots can be manipulated into harmful help, and how well automated attacks and defenses work, using a fixed list of forbidden requests.