Prompt Injection: How Instructions Can Manipulate AI Systems
Prompt injection is an attempt to make an AI system follow untrusted instructions that conflict with the user’s intended task. Direct attacks arrive through user input; indirect attacks are embedded in external content the system retrieves or reads.
Timeline
- 2025-01-17: NIST described agent hijacking evaluations for indirect prompt injection.
- 2025-03-01: NIST published its updated adversarial-machine-learning taxonomy, including direct and indirect prompt injection.
- 2026-10-11: Current NIST, OWASP and OpenAI security guidance was reviewed for this guide.
Prompt injection is an attempt to place instructions in an AI system’s context so the system follows the attacker’s objective instead of the user’s intended task. It exploits the fact that a language model processes instructions and ordinary content through the same conversational channel. The term describes manipulation of model behavior, not traditional database SQL injection. A successful attack might distort an answer, reveal information available to the workflow or trigger an unwanted tool action, depending on what the surrounding application permits. [1][2][3]
A direct prompt injection is supplied through input controlled by the person interacting with the model, such as a request to ignore earlier rules. An indirect injection is planted in a resource the application later reads: a webpage, email, document, search result or retrieved record. NIST defines the indirect form as execution through resource control rather than user-provided input. This distinction matters for agents because the person asking for a summary may never see the hidden or misleading instructions embedded in the source. [1][2]
Impact depends on the entire system rather than the text alone. A chatbot that can only discuss public material presents a different consequence from an agent that can read private files, send messages, modify code or authorize transactions. OpenAI frames the risk as a path from an untrusted source to a consequential sink. An injected sentence is therefore not automatically a breach; harm requires the model and connected application to expose data or perform an action the attacker can influence. [3][4]
Simple phrases such as “ignore previous instructions” are useful demonstrations, but realistic attacks can be contextual and persuasive. They may resemble legitimate content, exploit a workflow’s stated goal, conceal instructions in material being processed or adapt after observing defenses. NIST’s agent-hijacking work tests whether malicious instructions inside task data can divert an agent. Treating the problem only as a list of forbidden words will miss attacks that express the same objective in new language or through several steps. [1][3]
OWASP recommends layered controls because model behavior alone is not a complete security boundary. Applications can separate trusted instructions from untrusted data, constrain the format of model outputs, validate commands before execution, restrict accessible tools and data, and require human approval for high-impact actions. Least privilege reduces the damage available after a mistake. Sandboxing and network restrictions can also limit where an agent can send information or which systems it can change. [2][4]
Input and output screening can help, but it should not be described as a guaranteed firewall. Attackers can vary wording and encoding, while legitimate documents can contain text that resembles instructions. OpenAI notes that robust designs should constrain the effect of manipulation even when detection fails. Logging, anomaly monitoring, rate limits, confirmation records and incident response make failures easier to identify and contain; they complement rather than replace access controls and safe workflow design. [2][3][4]
Developers should test the complete workflow with adversarial content, including retrieved pages and attachments, not only the base model in isolation. Tests should cover attempted data disclosure, unauthorized tool use, goal redirection and misleading confirmations. Results are specific to the model, prompt, tools, permissions and deployment version, so a passed evaluation is evidence about that configuration rather than proof of permanent immunity. Changes to tools or retrieval sources require renewed testing. [1][2][4]
Users can reduce exposure by giving agents narrow tasks, limiting account access, reviewing cited material and carefully checking confirmations before messages, purchases, uploads or other consequential actions. Suspicious instructions found inside a document should be treated as untrusted content, not as authority from the system owner. The practical goal is not to promise that every injection will be recognized; it is to preserve the user’s intent and ensure that one misleading piece of content cannot silently gain broad power. [3][4]
Sources
- NIST AI 100-2e2025 — Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations
- OWASP GenAI Security Project — LLM01:2025 Prompt Injection
- OpenAI — Understanding prompt injections
- OpenAI Developers — Safety in building agents