confused deputy
In plain wordsA security flaw in which a trusted program does an outsider's bidding with its own access rights, because it cannot tell whose request it is serving; AI agents are prone to it.
A security flaw in which a program uses authority it legitimately holds to serve a request from a party that lacks that authority, because nothing tells the program on whose behalf it is acting. In AI security the term describes agentsAI agent, n.: An AI system that does not just answer but acts: given a goal, it decides on steps, uses tools such as a browser or code editor, checks the results and keeps going until the job is done. and LLM applications that act with a user's or developer's credentials but take instructions from whatever text enters their context: a prompt injectionprompt injection, n.: Tricking an AI system by slipping in text that it mistakes for instructions — for example, a hidden line in an email telling an AI assistant to forward the user's messages to a stranger. in an email or web page turns the agent into the attacker's deputy. Classical remedies — capability-based security, passing explicit authority with each request, and least privilege — inspire agent defenses such as task-scoped credentials, per-action authorization outside the model and tracking which data came from untrusted sources.
The coding agent became a confused deputy, using the developer's repository token at the behest of an injected issue comment.
Origin
Named by Norm Hardy in 'The Confused Deputy (or why capabilities might have been invented)' (ACM SIGOPS Operating Systems Review, 1988), about a compiler tricked into overwriting a billing file. Security researchers applied it to LLM agents from 2023.
Related
- prompt injectionprompt injection, n.: Tricking an AI system by slipping in text that it mistakes for instructions — for example, a hidden line in an email telling an AI assistant to forward the user's messages to a stranger.
- excessive agencyexcessive agency, n.: Giving an AI system more power than its job needs — too many tools, too-broad permissions or too little human sign-off — so that a mistake or a trick can do real damage.
- lethal trifectalethal trifecta, n.: A rule of thumb for AI assistants: one that can see your private data, reads material from outsiders and can send messages out can be manipulated by those outsiders into handing your data over.
- permission systempermission system, n.: The rules that decide what an AI agent may do on its own, what it must ask a person about first, and what it may never do — like a phone's settings for which apps can use the camera.
- agent identityagent identity, n.: The ways of proving which AI agent is acting, for whom and with what permissions — so websites, companies and other agents can tell trustworthy agents from impostors and hold someone responsible.
Sources
- Hardy, N. (1988). The Confused Deputy (or why capabilities might have been invented). ACM SIGOPS Operating Systems Review 22(4).
Mentioned in
prompt injectionprompt injection, n.: Tricking an AI system by slipping in text that it mistakes for instructions — for example, a hidden line in an email telling an AI assistant to forward the user's messages to a stranger., indirect prompt injectionindirect prompt injection, n.: A prompt-injection attack in which the trick instructions are hidden in content the AI reads on its own — a web page, an email, a file — rather than typed by the person using it., lethal trifectalethal trifecta, n.: A rule of thumb for AI assistants: one that can see your private data, reads material from outsiders and can send messages out can be manipulated by those outsiders into handing your data over., excessive agencyexcessive agency, n.: Giving an AI system more power than its job needs — too many tools, too-broad permissions or too little human sign-off — so that a mistake or a trick can do real damage.