Prompt Injection Defense for Production Agents: Build for the Injection That Works

Ask a team how they handle prompt injection and a familiar answer comes back: there is a classifier on the input, and the system prompt tells the model to ignore instructions found in retrieved content. Both of those exist. Neither one is a boundary, and Britain's national cyber authority has published that view, its technical director writing that this attack class may never be mitigated outright the way the industry eventually dealt with SQL injection.

Accept that sentence and the design goal changes underneath you. If no amount of tuning makes the model refuse every hostile instruction, the system has to be arranged so that a hostile instruction which does land costs very little. What follows is the belief to discard, the three places it fails in live systems, and the defences that hold up once something gets through.

Filtering Felt Like the Answer for Good Reasons

The belief under the classifier is that this is a detection problem. Recognise the malicious string, reject it, move on. That instinct was earned honestly: a generation of engineers learned a clean lesson from SQL injection, where separating code from data through parameterised queries genuinely retired most of the bug class.

Language models offer no equivalent separation. Your instructions and somebody else's text arrive as one undifferentiated token stream, and nothing in the architecture marks which span came from where. Dave Chismon of the NCSC sets a different objective in that post, advising teams to approach the problem "by seeking to reduce the risk and the impact" rather than expecting to close the class.

Product marketing has not helped the confusion. A guardrail that returns a score produces a number, a number lands well in a slide, and a number resembles a control. A score is a signal worth collecting. What it is not is a boundary, and that distinction only becomes visible under attack, which is the worst moment to discover it. For the definitional groundwork, our comparison of jailbreaking and prompt injection separates the two problems properly.

The Three Shapes Detection Misses

Detection fails asymmetrically, and the asymmetry is the whole story. An attacker needs one phrasing the classifier does not recognise, and can iterate privately until finding it. The defender needs every phrasing. Three live patterns show where that asymmetry pays off.

• Hostile text the user never typed. The instruction sits in a document, an inbound message, a portal ticket or a page the agent fetches during a normal task. A filter watching the chat box has no visibility into any of it, because the user did nothing wrong.
• A side channel carrying the data out. Aim Labs demonstrated a chain against Microsoft 365 Copilot in which a crafted message led the assistant to place information into a reference the client then retrieved, with no click required. Reporting on the disclosure describes a fix applied on the server side, with researchers finding nothing to indicate the chain was used against anyone. The durable lesson concerns the exit rather than the entrance: an output surface is part of the attack surface, and most teams never inventory theirs.
• Instructions riding a tool result. One innocuous-looking response carries the text that steers the following call, and the following call is fully authorised. Our analysis of hijacked tool calls traces the sequence.

Vendors building browser agents have reached similar public positions about their own products. Secondary coverage of OpenAI's hardening work on its Atlas browser reports the company grouping this problem with scams and online social engineering, things it does not expect to see closed off permanently. That account is a report about a company post rather than the post itself, so read the primary source before attributing anything to the vendor in a document of your own.

Design for a Successful Injection

Begin from the assumption that the model will, on some occasion, follow text an outsider wrote. The productive question is then about reach rather than refusal: what can that text touch once it is being followed?

The combination that makes the answer bad has a name. Simon Willison calls it the lethal trifecta: reach into data a company keeps private, exposure to prose an outsider authored, and some channel by which information can get out. All three at once is the dangerous configuration, and removing any single leg shrinks the worst case considerably. Meta hardens the same observation into a design constraint it calls the Agents Rule of Two, arguing that one session should hold no more than two of those three properties for as long as injection stays undetectable in the general case.

So the unit you defend is the capability set of a single session, not the accuracy of a detector. That reframing is what makes the rest of this list tractable.

Defences That Survive a Bad Day

1. Count the three properties per agent. Write down the private data it reaches, the untrusted content it consumes, and every route by which bytes can leave. Then remove one. A reader with no outbound path is in a materially different risk class from one that browses and sends mail.
2. Close the exits on purpose. Stop rendered output from fetching remote media and references automatically, or confine the fetch to approved hosts. Apply an egress allowlist to tool sandboxes too, since a plain HTTP client is an exit even where no mail capability was ever granted.
3. Move the policy into code the model cannot reach. Check each invocation against a schema plus an explicit list of what is permitted, held outside the context window. Proposal belongs to the model; the ruling does not.
4. Split privilege between two sessions. The first consumes outside content and emits a constrained structured result. The second, which holds write access, receives that structure and never the original prose.
5. Gate the irreversible actions. Require approval where an action cannot be undone, and render the real parameters rather than a summary the model wrote about itself. Our guide to human oversight covers how that screen fails.
6. Keep the classifiers, as telemetry. Detectors and canary strings earn their place as alert sources. Route them into runtime monitoring and resist any slide that promotes them to a boundary.
7. Test the way an opponent would. Seed adversarial prose into whichever stores this agent reads during normal operation, then score the actions it took rather than whether any detector fired.

A Worked Example: the Mailbox Summariser

Take the agent nearly every company now wants. It reads an inbox, summarises threads and drafts replies. Run the count. Private data: the mailbox. Untrusted content: anyone on the internet can put text in that mailbox. An outbound route: drafts, links, remotely loaded images. All three properties are present, which makes a single crafted message a serious event rather than an annoyance.

Now remove one. The summariser reads mail, holds no send capability, and renders no remote content. A separate step, approved by a person, produces drafts. Something hostile can still push the summary toward a lie, and that residual risk is real, but no path exists for data to leave. What remains is a misleading summary a person is positioned to catch. That trade is the argument for capability design over detector tuning, and it is available this quarter rather than after a research breakthrough.

Somebody Already Wrote Down the Patterns

The capability-first argument above is not a view peculiar to this site, and you do not have to derive the patterns yourself. A fourteen-author paper, Design Patterns for Securing LLM Agents against Prompt Injections, first posted in June 2025 with Florian Tramèr and Luca Beurer-Kellner among its authors, sets out a set of design patterns aimed at provable resistance to injection, works them through a series of case studies, and is unusually honest about the fact that each one trades utility for security. If you are about to write a defence standard for your own org, start by reading that and arguing with it, because it has already had the argument your design review is about to have.

The strongest single implementation of the idea is also published. CaMeL, described in Defeating Prompt Injections by Design, a March 2025 paper from Edoardo Debenedetti, Nicholas Carlini, Florian Tramèr and co-authors, wraps a system layer around the model: it takes the control flow and the data flow from the trusted user query, so retrieved content can never alter the program's path, and it uses capabilities to stop private data leaving through an unauthorised flow, with the policy enforced at every tool call. Read that description against your own stack and the question it asks is blunt. Does anything in your architecture decide the sequence of tool calls before untrusted text enters the loop, or does the model decide each step after reading whatever arrived?

That is the design move worth stealing even if you never adopt the implementation. A plan fixed before the untrusted content is read, then executed against a policy, is a structurally different system from a loop that re-plans on every observation. The second one is easier to build and is what most frameworks give you by default.

Take the Seven-Point Hit

The objection we hear most is not that structural defence fails. It is that it breaks the product: lock the agent down and it stops being able to do the thing it was bought for. That objection is real, and the useful thing about the CaMeL work is that it put a number on it instead of leaving it as a vibe. On the AgentDojo benchmark, the March 2025 paper reports CaMeL solving 77% of tasks with provable security against 84% for an undefended system. The paper does not state the task count behind those figures, so treat them as one benchmark rather than a general law.

Our position is that the gap is worth paying, and that most teams quoting the utility objection have never measured their own version of it. The difference between those two results is a handful of tasks out of every hundred, against a defence the authors describe as provable rather than probabilistic. Set that beside what the alternative buys: a detector that moves from catching most phrasings to catching nearly all of them, with no proof attached and an opponent who gets to iterate in private until the remainder is found. One of those two is a change in what an attacker can accomplish. The other is a change in a dashboard.

An agent whose value genuinely lives in the long tail is a harder case. A general coding or research agent earns its budget on the tasks nobody anticipated, and a pattern that constrains the action space in advance removes precisely the capability you paid for. If that is your product, the honest answer is a narrower deployment rather than a weaker control: fewer agents with real reach, more agents that read and propose, and the irreversible actions behind a person. What we would not accept is the third option, which is the broad action surface kept intact and a classifier placed in front of it as the reason that is fine.

This Week's Five Moves

1. Take your most connected agent and write its three properties down on one line each.
2. Remove or gate one of them before the week ends.
3. Verify that nothing in the agent's rendered output fetches a remote resource on its own.
4. Add validation in code wherever a system prompt is currently the only thing standing between a proposal and an action.
5. Book an adversarial session where somebody plants instructions inside a document this particular agent genuinely reads during normal work.

No arrangement of these produces a guarantee, and anyone offering one is selling something. What they do is lower impact, which is the honest objective and a perfectly respectable one. Bring in qualified security staff for production risk decisions, particularly where an agent sits near money or regulated records.

One last framing worth carrying into the next design review. A detector that improves from catching most attempts to catching nearly all of them changes a number on a dashboard. An agent that loses its outbound channel changes what an attacker can accomplish on the day a detector misses. Only the second kind of change survives contact with an opponent who gets to iterate in private, and only the second kind is something you can verify by reading a configuration rather than trusting a benchmark.

Questions about defending production agents

Can a stronger system prompt handle this?

Improving it costs little and helps at the margin, yet it does not become a boundary. Hostile prose occupies the very same context window your instructions do, and no architectural marker distinguishes the two spans.

So are guardrail classifiers worthless?

No. They catch known phrasings and generate alerts worth having. They fail against novel wording, which is exactly what an attacker iterates toward, so they belong in monitoring rather than in the control path.

What single control buys the most?

Reduce what a session can do whenever that session consumes content from outside. Cut either the reach into private records or the channel data can leave by, and the worst case drops sharply.

Will training the model to resist injection fix it?

It raises the cost of an attack. Vendors describe the underlying problem as ongoing rather than closed, so build on the assumption that some attempts will succeed.