influence operation
In plain wordsAn organized, often secret campaign to sway public opinion — fake accounts posting propaganda, for instance — which AI can make cheaper by writing posts, translating them and inventing fake personas.
An organized campaign, usually run by a state, political group or hired contractor, to steer what a chosen population believes or does, frequently in secret through fake accounts, front organizations and fabricated content. Generative AI can cut costs at each stage: writing and translating posts and comments, generating profile images and personas, producing synthetic mediasynthetic media, n.: Images, video, audio or text created or substantially altered by AI rather than recorded or written directly by people, whether for art, entertainment, business or deception. and deepfakesdeepfake, n.: A fake but realistic video, photo or voice recording made with AI that puts words in a real person's mouth or shows them in a scene that never happened., summarizing audience reactions and automating account management. AI providers' threat reports since 2024 have described state-linked operations using chatbots this way while finding that most drew little authentic engagement; the larger concern is future scaling and personalization of persuasion. Countermeasures include provider detection and account bans, platform enforcement against coordinated inauthentic behavior, provenance and labeling of AI content, and public attribution.
The influence operation had a chatbot write hundreds of comments a day in four languages, but few real users engaged.
Origin
A long-established military and intelligence term. Josh Goldstein and colleagues analyzed how language models could change influence operations in a January 2023 report from Georgetown's CSET, OpenAI and the Stanford Internet Observatory; in May 2024 OpenAI reported disrupting five covert operations, linked to Russia, China, Iran and an Israeli firm, that had used its models.
Related
- disinformationdisinformation, n.: Deliberate falsehoods or distortions dressed up as fact, for example from a government or campaign trying to sway an election or sow distrust.
- misinformationmisinformation, n.: False or misleading information that spreads, whether or not the people sharing it know it is wrong. AI can produce it by mistake and makes it cheaper to create.
- misusemisuse, n.: People deliberately using AI to cause harm — for scams, harassment, propaganda, hacking or weapons — as opposed to harm from AI making mistakes or pursuing goals of its own.
- synthetic mediasynthetic media, n.: Images, video, audio or text created or substantially altered by AI rather than recorded or written directly by people, whether for art, entertainment, business or deception.
- deepfake detectiondeepfake detection, n.: Using software, often itself AI, to tell whether a video, photo or voice recording has been faked or generated by AI rather than captured from reality.
- AI watermarkingAI watermarking, n.: Hiding an invisible signal in AI-generated text, images, audio or video so that software can later tell the content was made by AI — like a banknote's watermark, but undetectable to the eye.
Sources
- Goldstein, J. A., Sastry, G., Musser, M., DiResta, R., Gentzel, M. & Sedova, K. (2023). Generative Language Models and Automated Influence Operations: Emerging Threats and Potential Mitigations. arXiv:2301.04246.
Mentioned in
misusemisuse, n.: People deliberately using AI to cause harm — for scams, harassment, propaganda, hacking or weapons — as opposed to harm from AI making mistakes or pursuing goals of its own., disinformationdisinformation, n.: Deliberate falsehoods or distortions dressed up as fact, for example from a government or campaign trying to sway an election or sow distrust.