Incident Response for Agentic AI: What to Do When an Agent Has Already Acted

We built an agent that had read and write access to a project management system, a file storage service, and an internal wiki. It was supposed to help users reorganize documentation. Three days after deployment, we found it had moved files, renamed directories, and updated wiki pages across a dozen projects - based on a single ambiguous user request. No tool call had failed. No error had been raised. The agent had, by its own lights, done exactly what it was asked to do. The incident was already complete by the time we knew it had started.

Incident response for agentic AI is a different discipline than incident response for traditional applications. The failure mode is not a crash or an outage. It is a system that worked correctly, in the wrong direction, faster than you could observe it. This guide walks through what to do when you are already past the point of prevention.

Understanding Agentic Incident Response

Not every unexpected agent behavior is an incident. A failed tool call that returns an error and halts the task is recoverable - the agent tried, it stopped, the state is intact. An unauthorized state change is more serious: the agent modified something it should not have, but the scope is bounded and potentially reversible. Exfiltration or lateral movement is the worst case: data has left your systems, or the agent has used one system as a foothold to access others.

The reason agentic incidents feel disorienting is that the agent may have completed multiple steps before anything triggers an alert. Traditional incident response assumes you can stop the bad thing from happening. Here, you are often reconstructing what already happened - what the agent accessed, what it changed, what it sent where, and in what order. The response is not about stopping the agent. It is about understanding the action chain, containing the damage, and making sure it cannot happen the same way again.

One honest acknowledgment: your initial scope assessment will probably be wrong. The full blast radius of an agentic incident takes time to map, and the instinct to declare containment early is one to resist.

Why Agentic Incidents Escape Detection

The detection lag is structural. An agent receives a prompt, makes a sequence of tool calls, and returns a response - often within seconds or minutes. Each individual tool call is probably logged. The problem is that traditional monitoring is watching for errors, timeouts, and resource exhaustion. It is not watching for semantic correctness: whether the sequence of calls the agent made represents something it was supposed to do.

This is the visibility gap that makes agentic incidents hard to catch in real time. An agent that reads a customer record, queries a second system for related data, writes a summary to a shared document, and sends a notification - all within a single task execution - has made four tool calls, each of which succeeded, each of which is individually logged, and none of which triggered an alert. The incident is in the meaning of the chain, not in any single call.

Reconstructing that meaning after the fact is possible, but it requires that you have the right data: tool call logs with arguments, the user input that initiated the task, the agent's system prompt at the time of execution, and timestamps that let you sequence everything correctly. If you are missing any of these, reconstruction becomes guesswork.

Immediate Containment: The First 15 Minutes

The containment hierarchy is: stop the agent from taking further action, preserve evidence of what it already did, then assess scope. In that order. Do not assess scope while the agent is still running.

Stopping the agent means revoking the credentials it is operating under. If it is using a service account, API key, or OAuth token, revoke it now. This is the primary containment lever. Killing the process is not enough - if the credentials are still valid, the same behavior can be triggered again. Revoke first, investigate second.

Before you do anything else to the environment, snapshot everything the agent touched: execution logs, tool call history, modified state, any intermediate outputs it generated. Do not assume your normal log retention will preserve this. Agentic systems often produce high-volume, short-lived logs that rotate quickly. If you are in the first fifteen minutes, you may still have the full trace. An hour later, you may not.

We once lost forty minutes of tool call history because we assumed the logging pipeline would buffer it. It did not. What remained was enough to reconstruct the rough sequence, but not the arguments passed to each call. That gap made the root cause investigation significantly harder than it needed to be.

Reconstructing the Action Chain

Start with the user input that triggered the agent. Then trace every tool call in sequence. For each call, you want: the tool name, the arguments passed, the return value, and the timestamp. What you are building is a transaction log of the agent's actions - not just what it called, but what information it used to make each decision.

The decision point where things went wrong is usually one of three places. The user input was ambiguous or malicious and caused the agent to misinterpret its task. The agent's constraints were underspecified and it filled the gap with behavior you did not intend. Or a tool was available to the agent that should never have been - access control failed at the capability layer.

The system prompt is the first thing to review in reconstruction. Whatever constraints or guidelines the agent was operating under are documented there. If the agent violated them, you have a prompt-level failure: either the constraints were not specific enough, or the model did not follow them under the conditions of this particular input. If the agent followed the system prompt exactly and still caused harm, the constraints themselves were wrong.

Assessing Damage and Scope

The damage assessment framework has four questions: what data or systems did the agent access, what did it actually modify or exfiltrate, who else could have been affected, and what is the current state of each affected system.

Access and exfiltration are not the same thing, and the distinction matters for both your response and your regulatory obligations. An agent that read sensitive data but did not transmit it anywhere still represents a security incident - it violated a boundary it should not have been able to cross - but the containment and remediation path is different than if the data left your environment.

On blast radius: if the agent had credentials that could have accessed multiple systems, assume it could have acted on all of them. Even if your logs show it only called three of the seven services it had access to, the question is not what it did - it is what it could have done with the credentials it held. OWASP's LLM Top 10 treats excessive agency and overprivileged tool access as a primary risk category precisely because the blast radius of a compromised agent scales with the breadth of its access, not with the narrowness of its task.

Remediating Unauthorized Changes

Remediation priority: restore data integrity if the agent modified systems, revoke any credentials the agent may have exposed or created, then audit downstream systems the agent had access to.

The restore-from-backup decision is not always straightforward. Restoration is faster and cleaner, but it erases evidence - if the agent's modifications are overwritten, you may lose information about what exactly it changed. Manual reversion of each change is slower and more error-prone, but it preserves the audit trail. Our default is to document every change first, then restore, so you have a record of what the agent actually did even after the environment is clean.

If the agent had access to credential management systems - secrets managers, key vaults, anything that stores or generates credentials - assume it may have read or copied what was in scope. Do a full audit of what was accessible under its identity and rotate anything it could have touched. This is not optional. Credentials that the agent read but did not use are still compromised.

Investigating Root Cause: Prompt, Model, or Access Control?

The root cause decision tree has three branches. Did the agent receive a malicious or ambiguous user input that caused it to misinterpret its task? Did the agent's system prompt fail to adequately constrain the behavior? Or did the agent have access to capabilities it should not have had, and would a properly constrained agent have been unable to cause this damage even with the same input?

Test for reproducibility. Can you trigger the same behavior with the same input in a controlled environment? If yes, you have a deterministic failure - either the prompt is wrong, the constraints are wrong, or the access controls are wrong, and you can fix it. If the behavior does not reproduce, you may be dealing with model nondeterminism, which is harder to address and harder to prove you have addressed.

If the user input contained unusual characters, encoding, or structure, treat it as a potential prompt injection attempt. Early research on prompt injection focused on direct user manipulation, but the more dangerous variant in agentic systems is indirect injection - content the agent retrieves from external sources (a document, a web page, a database record) that contains instructions intended to redirect the agent's behavior. Review the full content of every external resource the agent fetched during the incident.

Preventing Recurrence: Architectural Changes

Least privilege applied to agents is not a metaphor - it is a specific design requirement. The agent should have access only to the tools and data it needs for its intended task, scoped as narrowly as possible. If the agent's job is to read and summarize documents in a specific directory, it should not have credentials that can write to that directory, access other directories, or call services unrelated to document retrieval. Every additional capability is additional blast radius.

Tool-level constraints are a second layer of defense. Even if the agent has access to a database query tool, you can constrain what that tool will execute: read-only queries only, specific tables only, result size limits, no schema-inspection calls. The tool is not a binary - accessible or not. It is a surface you can shape.

For high-risk actions - deleting records, transferring data across system boundaries, modifying access controls, sending external communications - implement approval workflows. NIST's AI risk management guidance frames this as human oversight at consequential decision points, which is exactly right. The approval workflow does not have to be manual for every action. It can be automated for low-risk actions and require human confirmation only when the action exceeds a defined risk threshold. But the threshold needs to exist, and it needs to be enforced at the tool layer, not just in the agent's instructions.

Communicating the Incident to Stakeholders

The communication hierarchy: internal security and engineering teams get immediate, detailed information. Business stakeholders and leadership get an executive summary within hours. Affected users and external parties get notified based on your legal and regulatory obligations, on whatever timeline those require.

For non-technical stakeholders, translate the technical description into impact language. Not "the agent made unauthorized API calls" but "the AI system accessed customer data it should not have been able to reach, and we are auditing what it read." Not "the agent violated its system prompt constraints" but "the AI system took actions outside the boundaries we set for it, and we are investigating how that happened." The goal is accurate, not simplified - do not obscure the severity, but do not hide it behind jargon either.

If the incident involved personal data, your regulatory notification obligations may be on a tight clock. GDPR Article 33 requires notification to supervisory authorities within 72 hours of becoming aware of a qualifying breach. CCPA and state privacy laws have their own timelines. Know your obligations before an incident occurs, not during one.

When to Seek Support

Escalate immediately if the incident involves exfiltration of sensitive data - PII, credentials, trade secrets - or if the agent modified systems in ways that affect production integrity or user data. Escalate if you cannot establish the full scope of the incident within the first hour, or if you find evidence of lateral movement across systems. Escalate if the incident pattern suggests deliberate exploitation rather than accidental misuse.

Support has different shapes depending on context. Internal escalation means your security team and, depending on scope, your incident response retainer. External escalation - law enforcement, regulatory bodies, affected third parties - depends on what the investigation reveals. If you are uncertain whether you have crossed a reporting threshold, get legal counsel involved early. That conversation is much easier before you have missed a notification deadline.

The cost of delay in agentic incidents is not abstract. Every hour the incident is not fully contained is an hour during which the agent's credentials may still be valid somewhere, the modified data may be propagating downstream, or the evidence you need for root cause analysis is rotating out of your logging pipeline. If you are unsure whether you need help, the answer is that you need help.