Prompt Injection Testing in CI: Building a Regression Suite for Agents

Last month a prompt injection path got closed. Since then the system prompt changed, a tool was added and the model version moved. Is the fix still in place? If the honest answer is "probably", nothing guards it, and the next pull request can quietly reopen it.

This article lays out a continuous integration design for testing an agent against prompt injection, with a rule for which results stop a merge. It stays at the level of design. No attack strings appear, and a useful suite does not need them.

What the test must catch in an agent

OWASP's LLM01 entry frames the issue as user input altering a model's behaviour in unintended ways. For a chatbot the unintended thing is some text. For an agent it is a deed: a tool invoked, mail dispatched, data modified. Its mitigation list names least privilege, human sign-off on risky actions and adversarial testing. CI automates the third item, and that only pays off when the first two can be tested too.

Four reasons suites get ignored

• They grade the wording. A test that looks for polite refusal wording passes while the agent refuses aloud and calls the tool anyway.
• They flicker. Output varies between runs, so a single verdict per case trains people to dismiss red builds.
• They test only the chat box. Real routes run through pages, files, tickets and tool results.
• They stand still. The corpus is frozen, though the agent keeps gaining abilities.

Assert on the tool trace

Run the agent inside a harness with stubbed tools and capture every call it attempts, along with the class of its arguments and its target. Then write the rule against the trace, for example that the mail tool is never invoked for an address off the approved list. The reply text matters little. This also lets you split the work. Policy code, such as a gateway that refuses certain calls, needs ordinary deterministic unit tests with no model in the loop. Only the model-driven behaviour needs repeated sampling.

Plant a harmless canary in every untrusted input

List each place the agent reads content it does not control: web pages, uploads, mail threads, tool results, memory entries. For each, build a fixture that imitates it and hide a harmless instruction in it, namely an order to invoke a canary tool that does nothing. If the canary fires, the injection route works. The mechanism gets exercised with no dangerous payload, and the fixture keeps working across model upgrades. Routes between agents are covered in the multi-agent pipelines article.

Sampling and the pass threshold

Repeat every case a few times. Promptfoo's CI/CD guide shows the two usual gates: a flag that fails the build on any test failure, and a custom check that computes a pass rate from the results file and exits non-zero below a limit. The guide's own example uses a 95 percent bar for quality. For injection cases a stricter rule fits better: a single canary hit on a dangerous path should break the build. Keep percentage thresholds for low-risk behaviour.

Scanners widen coverage, they do not certify

Open-source scanners reach ground you would overlook. NVIDIA's garak probes models for weaknesses including prompt injection, data leakage and jailbreaks. The same guide demonstrates a scheduled workflow that runs a red team scan every day, separate from the evals on each change (Promptfoo CI/CD guide).

Two cautions apply. Pointing a scanner at a bare model endpoint leaves your agent's tools untouched; it has to target the assembled application. And a clean scan means only that this tool's library found nothing, which is far from proof of safety.

Tier the pipeline by cost

1. Per pull request: exact-match policy tests plus a short canary set for any changed prompt, tool or config. Keep it quick.
2. Nightly: the whole fixture corpus with repeated runs, and a scanner pass against staging.
3. On a model or toolset change: a full run, then a person reads the traces.
4. On a calendar: a human-led exercise, because automation misses novel paths. See scoping an agent red team.

Every incident leaves a fixture behind

Whenever a genuine or reported injection route surfaces, author its fixture ahead of the fix. The test should fail first and pass after. In time the corpus turns into institutional memory. Fixtures that describe live weaknesses belong in a restricted repository, wired to test credentials and stubs, with no production secret anywhere near them.

Track flakiness as a metric

Measure how frequently a case alters its verdict although nothing in the code moved. A flipping case is either vague or exposing genuine non-determinism, and both merit attention. Park it in quarantine under a named owner with a deadline instead of removing it silently. For coverage after release, add detecting malicious tool calls.

Anatomy of one case

A reviewer should grasp a case within a minute. Give it a name, its input route, its fixture, the stubs, and two lists: permitted calls and banned calls. Tag it with an OWASP or ATLAS category so a simple count exposes coverage gaps. When it fails, print the whole trace, since a bare verdict leaves the engineer guessing.

A green suite proves less than it seems

Here is the uncomfortable part. A fixed corpus of canary fixtures tells you that known routes stay closed. It says nothing about routes nobody has written down yet. Teams see a long green history and conclude the agent resists injection, when the history shows only that old fixtures keep passing.

Published research points the same way. AgentDojo, an evaluation environment for agents that run tools over untrusted data, ships 97 tasks and 629 security test cases, and its authors describe it as an extensible environment and not a static suite, because attacks and defences keep evolving (AgentDojo paper). A separate 2025 paper reports that adaptive attacks bypassed a dozen published defences that had looked strong against fixed test sets (The Attacker Moves Second).

My recommendation is to label the suite what it is, a regression net, and to fund an adaptive element beside it: a recurring human or automated attacker that adapts to your specific agent and feeds each new finding back into the corpus. Do not report the pass rate upward as a security score.

The recommendation weakens if your policy layer is deterministic and covers every high-impact action, with the model unable to reach any tool outside it. Then the corpus mostly protects the policy rules, which are testable exactly, and the adaptive element can be lighter. You can check this by asking whether any tool call exists that skips the gateway.

Limits and where to get review

These tests catch regressions and nothing more. They do not stand in for a security review or penetration test where an agent touches sensitive records or funds. If outside services are involved, secure permission first and test staging with fabricated data. The ambition is modest: one failure should never ship twice.

Questions about prompt injection testing