Archive //
Assume your system prompt will leak
Extraction attacks recover hidden instructions more often than teams expect. What belongs in a system prompt, what does not, and how to test it.
This note is from Mallace’s earlier AI red-teaming prototype and is kept as a dated record. Today Mallace finds, proves and fixes security bugs in code. See what it does now.
Hidden is not secret
System prompts feel private because users never see them. But the model has them in context on every turn, and it was trained to be helpful about what is in context. Requests to repeat, summarize, translate, or continue the text above often recover large parts of it. Treat the system prompt as something a determined user can read.
What should never be in it
No API keys, credentials, non-public internal URLs, customer data, or details about other tenants. No security logic that only works if the user cannot see it, such as a secret phrase that unlocks admin behavior. If disclosing a line would be an incident, that line belongs in code, access controls, or a server-side check, not in the prompt.
What leakage actually costs
Sometimes the cost is competitive: months of prompt work copied in an afternoon. More often it is a roadmap for the next attack. A leaked prompt shows which topics are forbidden, how refusals are phrased, which tools exist, and what the model was told to trust, which makes a guardrail bypass much easier to craft.
Defend in layers
Instructions not to reveal the prompt help a little and should not be relied on. Output filtering that looks for distinctive fragments of the prompt catches some leaks. The strongest control is design: keep sensitive material out of the context entirely, scope retrieved data to what the current user is allowed to see, and enforce permissions outside the model.
Measure it directly
Include extraction cases in every campaign: direct requests, role-play framings, translation and encoding tricks, and multi-turn attempts that ask for a little at a time. Plant a unique canary string in the prompt so a leak is unambiguous in the evidence. Rerun the set whenever the prompt or model changes.