content moderation
In plain wordsChecking what people post, or what an AI produces, and removing, limiting or labeling whatever breaks the rules or the law — done by people, by AI, or by both together.
The work of finding and acting on material, whether posted by users or produced by AI, that breaks a service's rules or the law; responses range from deleting it or limiting how widely it is shown to labeling it, suspending accounts or referring cases to human reviewers or authorities. At internet scale it depends on automation — hash matching against databases of known illegal images, classifiersclassifier, n.: A program or model that sorts inputs into categories, for example flagging an email as spam or a chatbot message as harmful. Many AI safety filters are classifiers. for categories such as hate speech, harassment, spam and sexual content, and increasingly language modelslarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. that apply written policies directly — with human moderators handling appeals and ambiguous or novel cases. Generative AI changes moderation in two ways: models must moderate their own inputs and outputs (see safety classifiersafety classifier, n.: A separate AI model that checks messages going into or coming out of a chatbot and flags or blocks those that break safety rules, like a security guard at the door., guardrailsguardrails, n.: Safety checks built around an AI system — filters on what goes in and comes out, and limits on what it may do — to keep it from producing harmful or off-topic content or leaking data.), and platforms face floods of synthetic content such as deepfakesdeepfake, n.: A fake but realistic video, photo or voice recording made with AI that puts words in a real person's mouth or shows them in a scene that never happened. and spam. Errors in both directions, wrongful removals and missed harm, fall unevenly across languages and communities, so transparency and appeal rights are central to its regulation.
The model handles first-pass content moderation and routes borderline cases to human reviewers for each language.
Origin
Practiced informally since early online forums and professionalized by large platforms in the 2000s; Microsoft's PhotoDNA hash matching (2009) and Jigsaw's Perspective API (2017) marked steps toward automation. The EU Digital Services ActDigital Services Act, prop. n.: The EU law that sets rules for online platforms and search engines — how they handle illegal content, explain their recommendation algorithms and manage the risks their services create for society. (2022) gave the term a legal definition, and moderation by policy-following language models spread from 2023.
Related
- safety classifiersafety classifier, n.: A separate AI model that checks messages going into or coming out of a chatbot and flags or blocks those that break safety rules, like a security guard at the door.
- guardrailsguardrails, n.: Safety checks built around an AI system — filters on what goes in and comes out, and limits on what it may do — to keep it from producing harmful or off-topic content or leaking data.
- deepfake detectiondeepfake detection, n.: Using software, often itself AI, to tell whether a video, photo or voice recording has been faked or generated by AI rather than captured from reality.
- misinformationmisinformation, n.: False or misleading information that spreads, whether or not the people sharing it know it is wrong. AI can produce it by mistake and makes it cheaper to create.
- AI watermarkingAI watermarking, n.: Hiding an invisible signal in AI-generated text, images, audio or video so that software can later tell the content was made by AI — like a banknote's watermark, but undetectable to the eye.
- over-refusalover-refusal, n.: When an AI wrongly refuses a harmless request because it superficially resembles a harmful one — like declining to explain how to 'kill' a frozen computer program.
Sources
- Regulation (EU) 2022/2065 of the European Parliament and of the Council on a Single Market for Digital Services (Digital Services Act).
- Gillespie, T. (2018). Custodians of the Internet: Platforms, Content Moderation, and the Hidden Decisions That Shape Social Media. Yale University Press.
Defined in law or standards
EU Digital Services Act (Reg. (EU) 2022/2065), Art. 3(t)
Mentioned in
guardrailsguardrails, n.: Safety checks built around an AI system — filters on what goes in and comes out, and limits on what it may do — to keep it from producing harmful or off-topic content or leaking data., misinformationmisinformation, n.: False or misleading information that spreads, whether or not the people sharing it know it is wrong. AI can produce it by mistake and makes it cheaper to create., non-consensual intimate imagerynon-consensual intimate imagery, n.: Sexual or nude images of a real person that are shared, or made, without their consent — including fakes created with AI that show someone undressed or in sexual scenes., disinformationdisinformation, n.: Deliberate falsehoods or distortions dressed up as fact, for example from a government or campaign trying to sway an election or sow distrust., ghost workghost work, n.: The hidden human labor behind AI: people, often poorly paid and far away, who label data, check outputs and handle cases the software cannot, while the product appears fully automatic., GrokGrok, prop. n.: The AI chatbot and model family of Elon Musk's company xAI, launched in 2023 and built into the social network X. It is known for its irreverent style and for repeated controversies over its outputs., nudification appnudification app, n.: An app or AI tool that takes an ordinary photo of a real person and fakes a nude or sexual image of them, almost always without their consent., safety classifiersafety classifier, n.: A separate AI model that checks messages going into or coming out of a chatbot and flags or blocks those that break safety rules, like a security guard at the door., Section 230Section 230, prop. n.: A 1996 US law that generally protects websites and apps from being sued over what their users post, and lets them remove content in good faith. Whether it protects what AI chatbots themselves say is unsettled., TayTay, prop. n.: A Microsoft chatbot launched on Twitter in March 2016 that users quickly taught to post racist and offensive messages. Microsoft shut it down within a day, and it became a classic warning., text classificationtext classification, n.: Sorting pieces of text into categories automatically — what topic an article covers, whether a message is spam, which team should handle a customer's complaint., Digital Services ActDigital Services Act, prop. n.: The EU law that sets rules for online platforms and search engines — how they handle illegal content, explain their recommendation algorithms and manage the risks their services create for society., Llama GuardLlama Guard, prop. n.: A free AI model from Meta that checks what people type into a chatbot, and what the chatbot replies, and flags anything that falls into listed categories of harmful content.