safety classifier
In plain wordsA separate AI model that checks messages going into or coming out of a chatbot and flags or blocks those that break safety rules, like a security guard at the door.
A classifierclassifier, n.: A program or model that sorts inputs into categories, for example flagging an email as spam or a chatbot message as harmful. Many AI safety filters are classifiers. that labels prompts, responses or whole conversations as acceptable or violating according to a content or usage policy, so that they can be blocked, rewritten, escalated or logged. Input classifiers screen requests before they reach the main model; output classifiers screen generated text, often while it streams, so that a response can be cut off mid-generation; conversation-level classifiers catch harm spread across turns. Implementations range from small fine-tuned transformers and guard models trained on a hazard taxonomy (such as Llama Guard and ShieldGemma) to reasoning models that apply a policy supplied at run time, and cheap probeslinear probe, n.: The simplest kind of probe: a straight-line classifier trained on an AI model's internal activity to test whether some piece of information can be read off directly. on the main model's own activationsactivation, n.: The numbers a neural network computes inside itself as it processes an input — a reading of which of its artificial neurons are firing, and how strongly, at that moment.. The central trade-offs are recall on rare but severe harms against false positives that cause over-refusalover-refusal, n.: When an AI wrongly refuses a harmless request because it superficially resembles a harmful one — like declining to explain how to 'kill' a frozen computer program., plus latency, compute cost and robustness to deliberate evasion.
The output safety classifier halted the stream as soon as the answer drifted into restricted territory.
Origin
Automated toxicity classifiers predate chatbots, notably Jigsaw and Google's Perspective API (2017). OpenAI described its moderation classifier in 2022, Meta released Llama Guard in December 2023, and by 2025 input and output classifiers were a standard layer in frontier-model deployments, including Anthropic's constitutional classifiersconstitutional classifiers, n.: Safety filters that watch a chatbot's inputs and outputs for dangerous requests, trained on example conversations that an AI generated from a written list of what is and is not allowed..
Related
- guardrailsguardrails, n.: Safety checks built around an AI system — filters on what goes in and comes out, and limits on what it may do — to keep it from producing harmful or off-topic content or leaking data.
- constitutional classifiersconstitutional classifiers, n.: Safety filters that watch a chatbot's inputs and outputs for dangerous requests, trained on example conversations that an AI generated from a written list of what is and is not allowed.
- content moderationcontent moderation, n.: Checking what people post, or what an AI produces, and removing, limiting or labeling whatever breaks the rules or the law — done by people, by AI, or by both together.
- over-refusalover-refusal, n.: When an AI wrongly refuses a harmless request because it superficially resembles a harmful one — like declining to explain how to 'kill' a frozen computer program.
- linear probelinear probe, n.: The simplest kind of probe: a straight-line classifier trained on an AI model's internal activity to test whether some piece of information can be read off directly.
- jailbreakjailbreak, n.: A trick for getting an AI chatbot to break its own rules — for example, to give dangerous instructions it would normally refuse — usually through cleverly worded requests.
Sources
- Markov, T. et al. (2023). A Holistic Approach to Undesired Content Detection in the Real World. AAAI 2023.
- Inan, H. et al. (2023). Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations. arXiv preprint.
Mentioned in
AI safetyAI safety, n.: The effort to keep AI from doing serious harm, whether because it pursues aims its makers never intended, is turned to bad uses or simply malfunctions — a concern that grows as systems become more capable., classificationclassification, n.: Sorting things into categories — spam or not spam, which animal is in a photo, which language a text is written in — by learning from examples that are already labeled., classifierclassifier, n.: A program or model that sorts inputs into categories, for example flagging an email as spam or a chatbot message as harmful. Many AI safety filters are classifiers., guardrailsguardrails, n.: Safety checks built around an AI system — filters on what goes in and comes out, and limits on what it may do — to keep it from producing harmful or off-topic content or leaking data., jailbreakjailbreak, n.: A trick for getting an AI chatbot to break its own rules — for example, to give dangerous instructions it would normally refuse — usually through cleverly worded requests., misusemisuse, n.: People deliberately using AI to cause harm — for scams, harassment, propaganda, hacking or weapons — as opposed to harm from AI making mistakes or pursuing goals of its own., non-consensual intimate imagerynon-consensual intimate imagery, n.: Sexual or nude images of a real person that are shared, or made, without their consent — including fakes created with AI that show someone undressed or in sexual scenes., refusalrefusal, n.: When an AI declines to do what it was asked — 'I can't help with that' — usually because the request seems harmful or breaks the rules it was trained to follow., automated red teamingautomated red teaming, n.: Using AI programs, not just human testers, to invent huge numbers of tricky inputs that try to make another AI misbehave, so problems can be found and fixed at scale., content moderationcontent moderation, n.: Checking what people post, or what an AI produces, and removing, limiting or labeling whatever breaks the rules or the law — done by people, by AI, or by both together., defense in depthdefense in depth, n.: Protecting against AI risks with several independent layers of safeguards, so that if one fails — say, the model's own training — another, such as a filter or human review, still catches the problem., nudification appnudification app, n.: An app or AI tool that takes an ordinary photo of a real person and fakes a nude or sexual image of them, almost always without their consent., usage policyusage policy, n.: A company's published rules on what people may and may not use its AI for — for example, no help building weapons, no scams and no sexual content involving children., AI bug bountyAI bug bounty, n.: A program in which an AI company pays outside researchers for reporting flaws in its AI systems — security holes, ways around safety rules, or biased behavior — so that they can be fixed., circuit breakerscircuit breakers, n.: A defense that changes an AI model's inner workings so that, whenever it starts heading toward a harmful answer, its internal activity is knocked off course and it cannot produce coherent harmful content., constitutional classifiersconstitutional classifiers, n.: Safety filters that watch a chatbot's inputs and outputs for dangerous requests, trained on example conversations that an AI generated from a written list of what is and is not allowed., Llama GuardLlama Guard, prop. n.: A free AI model from Meta that checks what people type into a chatbot, and what the chatbot replies, and flags anything that falls into listed categories of harmful content., multi-turn jailbreakmulti-turn jailbreak, n.: Breaking a chatbot's rules gradually over a conversation — starting with harmless questions and inching toward the forbidden one, so that no single message looks alarming., PAIRPAIR, abbr.: An automated way to test chatbot safety in which one AI keeps rewriting a request to try to trick another AI into breaking its rules, learning from each refusal., structured accessstructured access, n.: Letting people use an AI system through a controlled channel, such as a website or programming interface, instead of handing over the model itself, so that the developer can limit misuse., trusted monitoringtrusted monitoring, n.: A safety setup in which a weaker AI that we trust reviews each action of a stronger AI we do not fully trust, flagging the most suspicious actions for a human to check.