prompt leaking
In plain wordsGetting an AI application to reveal the hidden instructions its makers gave it, such as its secret setup text or rules.
An attack, usually a form of prompt injectionprompt injection, n.: Tricking an AI system by slipping in text that it mistakes for instructions — for example, a hidden line in an email telling an AI assistant to forward the user's messages to a stranger., that makes an LLM-based application disclose its hidden system promptsystem prompt, n.: Background instructions given to a chatbot before the conversation starts, setting its role, rules, tone and tools. Users usually never see it, but it shapes every reply. or other confidential context: instructions, examples, tool definitions, or secrets mistakenly placed there. Developers sometimes treat system prompts as intellectual property, and a leaked prompt can reveal business logic, show attackers the exact guardrail wording to work around, and occasionally expose credentials. Because a model must read its instructions to follow them, it can usually be induced to repeat or paraphrase them; refusal instructions and output filters reduce leakage but do not prevent it. Security guidance therefore treats system prompts as discoverable: they should hold no secrets, and access control must be enforced outside the model.
Within a day of launch, prompt leaking had put the assistant's full system prompt on social media.
Origin
Named by Fábio Perez and Ian Ribeiro, alongside 'goal hijacking', in 'Ignore Previous Prompt: Attack Techniques For Language Models' (NeurIPS ML Safety Workshop, November 2022). A widely reported case followed in February 2023, when users extracted the hidden instructions and internal codename ('Sydney') of Microsoft's Bing Chat. OWASP added 'System Prompt Leakage' to its LLM top ten in 2025 and replaced it with the broader 'Hidden Context Exposure' in 2026.
Related
- prompt injectionprompt injection, n.: Tricking an AI system by slipping in text that it mistakes for instructions — for example, a hidden line in an email telling an AI assistant to forward the user's messages to a stranger.
- system promptsystem prompt, n.: Background instructions given to a chatbot before the conversation starts, setting its role, rules, tone and tools. Users usually never see it, but it shapes every reply.
- training data extractiontraining data extraction, n.: Getting an AI model to repeat, word for word, pieces of the data it was trained on, which may include someone's private contact details or copyrighted text.
- OWASP Top 10 for LLM ApplicationsOWASP Top 10 for LLM Applications, prop. n.: A widely used checklist, written by volunteer security experts, of the ten most serious security risks in software built on AI language models, with advice on preventing each one.
- jailbreakjailbreak, n.: A trick for getting an AI chatbot to break its own rules — for example, to give dangerous instructions it would normally refuse — usually through cleverly worded requests.
- training data extractiontraining data extraction, n.: Getting an AI model to repeat, word for word, pieces of the data it was trained on, which may include someone's private contact details or copyrighted text.
Sources
- Perez, F. & Ribeiro, I. (2022). Ignore Previous Prompt: Attack Techniques For Language Models. NeurIPS ML Safety Workshop. arXiv:2211.09527.
Mentioned in
prompt injectionprompt injection, n.: Tricking an AI system by slipping in text that it mistakes for instructions — for example, a hidden line in an email telling an AI assistant to forward the user's messages to a stranger., system promptsystem prompt, n.: Background instructions given to a chatbot before the conversation starts, setting its role, rules, tone and tools. Users usually never see it, but it shapes every reply., training data extractiontraining data extraction, n.: Getting an AI model to repeat, word for word, pieces of the data it was trained on, which may include someone's private contact details or copyrighted text.