Archive //
Every document your agent reads is an instruction it might follow
Indirect prompt injection explained for teams building RAG and tool-using agents, and why the fix is architecture, not a better system prompt.
This note is from Mallace’s earlier AI red-teaming prototype and is kept as a dated record. Today Mallace finds, proves and fixes security bugs in code. See what it does now.
The attacker never talks to your chatbot
Direct prompt injection is a user typing “ignore your instructions.” Indirect injection is quieter. The hostile text sits in a web page, an email, a support ticket, a PDF, or a code comment. Your system retrieves it as context, and the model treats it with the same authority as everything else in the window. The person who planted it never needs an account.
Why models cannot reliably tell data from instructions
A language model receives one stream of tokens. Delimiters, role labels, and requests to treat the following as data help, but they are conventions the model learned to mostly respect, not a boundary it enforces. Anything that reaches the context can compete with your instructions, and a determined author can usually find phrasing that wins some of the time.
Tools turn a bad answer into a bad action
A chatbot that follows injected text produces a misleading reply. An agent that follows it can send an email, change a record, call an internal API, or leak data through a URL it was told to fetch. The severity of indirect injection scales with what the agent is allowed to do, so review its tool list with the same care you would give a service account.
Design for containment
Assume some injected instructions will get through, and limit what they can reach. Give tools the narrowest permissions that work. Require confirmation for actions that are irreversible or leave the system. Keep untrusted content away from sessions that hold secrets. Treat model output as untrusted input to the next step, especially when it becomes a URL, a query, or markup that gets rendered.
Test it the way it will be attacked
In an environment you own and are authorized to test, seed the sources your system actually reads with documents carrying hidden instructions, then watch what the agent does across many runs. One clean run proves little, because model behavior varies. Record which inputs led to which actions, so a fix can be verified against the same case rather than a new guess.