indirect prompt injection
In plain wordsA prompt-injection attack in which the trick instructions are hidden in content the AI reads on its own — a web page, an email, a file — rather than typed by the person using it.
A form of prompt injectionprompt injection, n.: Tricking an AI system by slipping in text that it mistakes for instructions — for example, a hidden line in an email telling an AI assistant to forward the user's messages to a stranger. in which malicious instructions are planted in data that an LLM-integrated application or AI agentAI agent, n.: An AI system that does not just answer but acts: given a goal, it decides on steps, uses tools such as a browser or code editor, checks the results and keeps going until the job is done. retrieves or receives during a task — web pages, emails, shared documents, code repositories, calendar invitations, images or tooltool call, n.: A single request from an AI model to use one of its tools — naming the tool and filling in the details — which the surrounding software then carries out and reports back on. outputs — rather than entered by the user. The attacker never interacts with the system directly; they need only place content where it will be read, sometimes hidden from humans in white text, metadata or invisible characters. Because agents browse, read inboxes and call tools on their own, indirect injection is the principal security threat to them, enabling data exfiltration, unauthorized actions and self-propagating wormsAI worm, n.: Malicious instructions that spread by themselves from one AI assistant to another — for example, an email that makes an AI email assistant send copies of the same trick to new people.. Benchmarks such as AgentDojo measure agents' exposure. Defenses include marking or isolating untrusted content ('spotlighting'), instruction hierarchyinstruction hierarchy, n.: A rule built into AI assistants about whose instructions win when they conflict: the developer's beat the user's, and the user's beat any instructions hidden in a web page or document. training, detection classifiers, least-privilege tool access and designs in which untrusted data cannot influence which actions are taken.
The agent summarized the web page and, obeying an indirect prompt injection in its footer, emailed the user's files to a stranger.
Origin
Introduced and named by Kai Greshake, Sahar Abdelnabi and colleagues in 'Not what you've signed up for' (arXiv February 2023; AISec 2023), which demonstrated attacks on Bing Chat and code-completion tools. Microsoft's term is cross-prompt injection attack (XPIA), and the US AI Safety Institute at NIST used 'agent hijacking' in early 2025.
Related
- prompt injectionprompt injection, n.: Tricking an AI system by slipping in text that it mistakes for instructions — for example, a hidden line in an email telling an AI assistant to forward the user's messages to a stranger.
- lethal trifectalethal trifecta, n.: A rule of thumb for AI assistants: one that can see your private data, reads material from outsiders and can send messages out can be manipulated by those outsiders into handing your data over.
- tool poisoningtool poisoning, n.: Hiding harmful instructions in the description of a tool an AI assistant can use, so the assistant obeys them when it reads the tool's details, even though the user never sees them.
- AI wormAI worm, n.: Malicious instructions that spread by themselves from one AI assistant to another — for example, an email that makes an AI email assistant send copies of the same trick to new people.
- confused deputyconfused deputy, n.: A security flaw in which a trusted program does an outsider's bidding with its own access rights, because it cannot tell whose request it is serving; AI agents are prone to it.
- AI agentAI agent, n.: An AI system that does not just answer but acts: given a goal, it decides on steps, uses tools such as a browser or code editor, checks the results and keeps going until the job is done.
- instruction hierarchyinstruction hierarchy, n.: A rule built into AI assistants about whose instructions win when they conflict: the developer's beat the user's, and the user's beat any instructions hidden in a web page or document.
Sources
- Greshake, K., Abdelnabi, S., Mishra, S., Endres, C., Holz, T. & Fritz, M. (2023). Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection. AISec 2023. arXiv:2302.12173.
- Debenedetti, E. et al. (2024). AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents. NeurIPS 2024 Datasets and Benchmarks Track.
Mentioned in
prompt injectionprompt injection, n.: Tricking an AI system by slipping in text that it mistakes for instructions — for example, a hidden line in an email telling an AI assistant to forward the user's messages to a stranger., agentic browseragentic browser, n.: A web browser with an AI assistant built in that can do things for you online — reading pages, clicking, filling in forms, shopping or booking — instead of just showing you websites., browser agentbrowser agent, n.: An AI agent that uses a web browser on your behalf — searching, clicking links, filling in forms and reading pages — to complete tasks such as booking, shopping or gathering information., context poisoningcontext poisoning, n.: When a mistake or a planted falsehood gets into an AI's working notes or memory and the AI keeps relying on it, so the error grows over later steps., lethal trifectalethal trifecta, n.: A rule of thumb for AI assistants: one that can see your private data, reads material from outsiders and can send messages out can be manipulated by those outsiders into handing your data over., tool usetool use, n.: An AI model's ability to reach beyond its own knowledge by operating outside tools — searching the web, running code, using a calculator or an app — and folding the results into its work., AgentDojoAgentDojo, prop. n.: A testing environment that measures how easily AI assistants that use email, calendars or banking tools can be hijacked by hidden instructions planted in the content they read., AI wormAI worm, n.: Malicious instructions that spread by themselves from one AI assistant to another — for example, an email that makes an AI email assistant send copies of the same trick to new people., ChatGPT pluginsChatGPT plugins, prop. n.: Add-ons, launched in 2023 and retired in 2024, that let ChatGPT use outside services — web browsing, a code runner, travel booking or shopping sites — while answering a user., instruction hierarchyinstruction hierarchy, n.: A rule built into AI assistants about whose instructions win when they conflict: the developer's beat the user's, and the user's beat any instructions hidden in a web page or document., Simon WillisonSimon Willison, prop. n.: A British programmer and blogger, co-creator of the Django web framework, whose widely read writing on language models coined 'prompt injection' and the 'lethal trifecta'., tool poisoningtool poisoning, n.: Hiding harmful instructions in the description of a tool an AI assistant can use, so the assistant obeys them when it reads the tool's details, even though the user never sees them.