AI bug bounty
In plain wordsA program in which an AI company pays outside researchers for reporting flaws in its AI systems — security holes, ways around safety rules, or biased behavior — so that they can be fixed.
A program that pays external researchers for responsibly reporting flaws in AI systems, adapting the software industry's bug bounties. Scope varies. Security bounties cover vulnerabilities in AI products and infrastructure, such as prompt injectionprompt injection, n.: Tricking an AI system by slipping in text that it mistakes for instructions — for example, a hidden line in an email telling an AI assistant to forward the user's messages to a stranger. that leaks user data or makes an agent take unauthorized actions; safety bounties reward universal jailbreaksjailbreak, n.: A trick for getting an AI chatbot to break its own rules — for example, to give dangerous instructions it would normally refuse — usually through cleverly worded requests. that defeat a model's safeguards against high-severity misuse, such as questions about biological weapons; and bias bounties reward evidence of discriminatory behavior. Many programs exclude individual jailbreaks or offensive outputs, treating them as content issues rather than vulnerabilities. By 2025 top rewards ran to tens of thousands of dollars, and to $100,000 for critical security flaws at OpenAI; Google's dedicated AI vulnerability reward program, launched that year, pays up to $30,000. Bounties complement internal red teamingred teaming, n.: Deliberately attacking your own AI system — trying to make it misbehave, leak secrets or help with harm — so that weaknesses are found before real adversaries or users find them.; researchers argue that they cover only part of the risk surface and that third-party flaw-disclosure processes and legal safe harbors for good-faith testing are also needed.
Her universal jailbreak earned the top AI bug bounty, and the safety classifier was retrained within a week.
Origin
Twitter ran the first widely reported algorithmic bias bounty, on its image-cropping algorithm, at the DEF CON hacking conference in 2021. OpenAI opened a bug bounty in April 2023 that excluded jailbreaks; Google and Microsoft extended reward programs to generative AI later that year, Anthropic launched bounties for universal jailbreaks in 2024, and OpenAI offered biosecurity jailbreak bounties in 2025.
Usage note
Distinct from public jailbreak competitions, which are time-limited challenges with prizes, though the line is blurred. Bounty terms that forbid publication or require confidentiality have been criticized for limiting independent scrutiny.
Related
- red teamingred teaming, n.: Deliberately attacking your own AI system — trying to make it misbehave, leak secrets or help with harm — so that weaknesses are found before real adversaries or users find them.
- jailbreakjailbreak, n.: A trick for getting an AI chatbot to break its own rules — for example, to give dangerous instructions it would normally refuse — usually through cleverly worded requests.
- prompt injectionprompt injection, n.: Tricking an AI system by slipping in text that it mistakes for instructions — for example, a hidden line in an email telling an AI assistant to forward the user's messages to a stranger.
- safety classifiersafety classifier, n.: A separate AI model that checks messages going into or coming out of a chatbot and flags or blocks those that break safety rules, like a security guard at the door.
- constitutional classifiersconstitutional classifiers, n.: Safety filters that watch a chatbot's inputs and outputs for dangerous requests, trained on example conversations that an AI generated from a written list of what is and is not allowed.
- algorithmic biasalgorithmic bias, n.: Unfairness in software that judges or serves people: it consistently produces worse results for some groups, such as women, older people or ethnic minorities, often because it learned from skewed or incomplete records.
- AI incidentAI incident, n.: A real-world case in which an AI system, through a flaw, a breakdown or the way someone used it, ends up hurting people, property or rights, as in an injury, a wrongful arrest or a privacy breach.
Sources
- Longpre, S. et al. (2024). A Safe Harbor for AI Evaluation and Red Teaming. arXiv.
- Longpre, S. et al. (2025). In-House Evaluation Is Not Enough: Towards Robust Third-Party Flaw Disclosure for General-Purpose AI. arXiv.