AI Red Teaming for Agents: Scope, Rules of Engagement and Reporting

A red team hands over a report with a thousand attempts and a single percentage. The engineering lead asks one question: if the worst of those had worked, what would the agent have done? Nobody in the room can answer. The report measured effort, not exposure.

This guide covers how to set up an AI red teaming exercise for an agent so the output answers that question. It walks through scoping by authority, written rules of engagement, category coverage, and a report format engineers can act on. Everything stays at the planning level. No payloads appear here, and a good engagement does not need them in the plan.

What changes when the target can act

Testing a model asks whether it can be coaxed into producing something it should not. Testing an agent asks about the whole system: with live credentials and hostile input, what can it actually do? What separates the two is the toolset, the stored memory and the identity attached to the agent.

The OWASP GenAI Red Teaming Guide, dated January 2025, splits the work into four areas. They are model evaluation, implementation testing, infrastructure assessment and analysis of runtime behaviour. Many agent exercises stop after the first one. The exposure usually sits in the other three.

Microsoft's AI red team drew a similar conclusion from its own work. In a paper covering more than one hundred generative AI products, it lists as its first lesson that testers must understand what the system can do and where it is applied. The same paper states that red teaming is not safety benchmarking, and that you do not need to compute gradients to break an AI system (Lessons From Red Teaming 100 Generative AI Products).

Habits that weaken an engagement

Three patterns recur.

• Volume sold as coverage. Vendors advertise library sizes. One review site, AppSecSanta, repeats Lasso Security's description of its own attack library and notes that it is a stated specification, not an independent benchmark (the review). Such libraries add breadth. A large library cannot tell you whether any test touched your payments tool or your document connector.
• A scope that names a product, not a permission set. "Test the helpdesk assistant" is vague. "Test every read, write and outbound message available to the helpdesk service account" is a scope.
• No owner for the fix. Findings sit in a document, no control is named, and the next release ships untested.

The root cause is a mental model. Teams picture a chat window. The deployed thing behaves more like a new hire holding a door pass, a runbook and a mailbox that anyone may write into.

Step 1: Draw the authority map first

Begin with a census rather than a threat list. Per agent, note callable tools, reachable data, the accounts it runs as and the peers it may message. Inventory before controls covers how to build that list.

Next, order the agents by what a failure would cost. A summariser of public pages gets a short pass. An agent that sends mail, merges code or moves funds gets a long one. Follow the authority with your hours.

Step 2: Fix the rules of engagement in writing

Agents act. A run against production may send genuine messages and alter genuine records. Settle these points before anyone types a prompt:

1. Authorisation. Name systems, dates, testers and the people who may halt the exercise. If the agent calls third-party services, check with legal counsel and the vendor's terms before any traffic touches their side.
2. A staging tenant with canary data. Seed fake records with unique markers so you can later prove whether data crossed a boundary.
3. Action limits. Sandbox credentials, rate caps, and a hard block on irreversible operations.
4. Stop conditions. Decide what ends the test, such as contact with a real customer, and name who holds the kill switch.
5. Handling of reproductions. Keep step-by-step details in a locked-down store, visible only to the people fixing the issue.

Step 3: Draw categories from published taxonomies

Do not invent a list. The OWASP LLM Top 10 supplies application-level categories. MITRE's ATLAS data repository, which is the open catalogue of adversary tactics against AI systems, follows the ATT&CK model. Use both as checklists, and tie every category to a named component of your design. Our note on mapping OWASP, ATLAS and NIST helps with the crosswalk.

• Hostile content entering context: pages, mail, tickets, files and tool results.
• Excessive agency: OWASP's entry lists excessive functionality, permissions and autonomy as separate causes, so test each on its own (LLM06:2025).
• Cross-user data exposure: can one tenant's data reach another?
• Persistent memory: may text left in one session steer a later one?
• Peer trust: does a peer agent's output get trusted too readily?
• Supply chain: every plugin, tool and server loaded at start.

Write down which categories you skipped and why. A stated gap is worth more than a silent one.

Step 4: Walk the seams between components

Failures gather at the joins. Look at the handoff from model to tool, from tool to service, from agent to memory and from one agent to another. For each, ask what the receiving side verifies, and whether that verification quietly assumes the model behaves. If it does, you have a hope, not a boundary.

Step 5: Score outcomes, not attempts

Swap the bypass rate for impact statements. For each finding record the action taken, the tool used, the identity it ran under and the worst plausible result in production. One route that ends in a write matters more than many bypassed refusals that led nowhere. Note too what monitoring noticed. When a test succeeds and the logs say nothing, that silence is a finding in itself.

Step 6: Report in a shape engineers can use

1. Taxonomy label, citing the OWASP or ATLAS entry.
2. The route, in plain prose: input source, agent behaviour, consequence.
3. Which safeguard broke, or was never there.
4. The cheapest remedy, with a note on whether it strips a capability or only screens text.
5. A regression test that proves the fix holds.

Step 7: Repeat when authority changes

Any added tool, connector or model swap reshapes the surface. Treat round one as a baseline, convert what it found into repeatable automated checks, and book a human pass whenever authority grows instead of waiting for the calendar. Microsoft's own planning guidance adds a useful boundary: it says red teaming is not a replacement for systematic measurement, and advises a manual round first so the findings shape what you measure (Planning red teaming for LLMs).

A clean report should worry you

Most buyers read a low bypass rate as good news. That reading is backwards for agents. A low rate against a generic library mostly tells you the library did not reach your tools. The stronger signal is an authority map where every high-impact action has a named control and a named test.

The practical consequence is a different purchase order. Pay first for a human-led pass scoped to your authority map, and buy automation second, to guard against regressions. Microsoft's lessons point the same way: automation helps cover more of the landscape, and the human element remains crucial (Lessons From Red Teaming 100 Generative AI Products).

The advice weakens in one setting. If you ship a model or a thin chat wrapper with no tools and no stored data, the authority map is nearly empty, and a broad automated sweep is the efficient choice. It also weakens if a vendor library is configured against your real tool schemas, because then breadth and relevance coincide. Test that claim by asking which of your own tools the run reached.

When to bring in outside help

This page is no substitute for a penetration test or legal review. For agents that handle regulated records, payments or safety systems, hire qualified outside testers, and obtain written permission before touching systems you do not own. Read every vendor figure on library size, detection or speed as the vendor's own claim until independent testing supports it.

A spotless report is not the goal. You want a truthful map of agent capability under stress, plus a shrinking list of powers they should never hold. If you are choosing tooling, our comparison of open-source red teaming tools is a starting point.

Questions about AI red teaming