A Practical Incident Response Plan for AI Agents

The four candidate sources resolved. NIST SP 800-61 Rev. 2 is withdrawn (superseded by Rev. 3, April 2025), so I’ll cite Rev 3. OWASP’s 2025 LLM Top 10 and the NIST AI RMF pages are live and support the claims I’ll make. Now drafting the article and checking the prose word count.An AI agent with credentials is an operator, not a feature. If yours can query a production database, call a vendor API, and email customers, an incident is no longer only a question of uptime. The questions become what the agent did and how fast you can pull its access.

Most teams running infrastructure already have an incident response plan for servers and accounts. Check whether yours says anything about agents. If it doesn’t, close that gap in advance, because closing it mid-incident is much harder.

Why agent incidents are different

Conventional incidents come in two flavors: things that go down and things that get broken into. Agents add a third, where the system stays up, keeps working, and does the wrong thing with confidence. A hung process pages someone. An agent that quietly emails a refund promise to the wrong customers pages nobody, because nothing crashed.

Three properties drive this. Agent behavior is probabilistic, so the same input can produce different actions on different runs. Agents have a wide action space, meaning every tool and API you wired in. And the inputs that steer them are often untrusted, since agents read whatever lands in a ticket queue or on a web page. The OWASP Top 10 for LLM Applications names these risks directly: prompt injection is first on its 2025 list, with excessive agency and unbounded consumption also on it.

The business translation is blunt. An agent with write access to billing can do things a bug can’t: corrupt records, message real customers, and run up an API bill in minutes. Recovery isn’t restoring a service. It’s auditing every action the agent took, deciding which ones to undo, and explaining the rest to customers.

Borrow the framework, then adapt it

You don’t need to invent a response process. NIST’s current incident response guidance, SP 800-61 Revision 3 (published April 2025), folds incident response into cybersecurity risk management and covers preparing for incidents, detecting them, responding, and recovering. That skeleton holds for agents. What changes is what you prepare, what you detect, and what you contain.

Preparation: shrink the blast radius

Start with an inventory. List every agent you run and every credential it holds. For each, ask what the least privilege version looks like: a read-only replica instead of the primary database, a scoped API key instead of the main account, tokens that expire instead of tokens that live forever. Agents should never hold a shared admin credential, because you can’t revoke what everyone shares.

Set hard limits next. A spend cap per run and per day. A maximum number of tool calls per task. A timeout on long reasoning chains. Unbounded consumption is a known failure mode, and a loop that calls a paid API all night produces a bill, not just an outage.

Version everything that shapes behavior: prompts, tool definitions, model configuration. If a prompt change breaks the agent at 2 a.m., you roll it back the same way you roll back a deploy.

Then log every tool call. The record needs a timestamp, the agent, the prompt version, the tool, the arguments, the result, and whether a human approved it:

{
  "ts": "2026-03-14T02:11:09Z",
  "agent": "support-triage",
  "prompt_version": "v14",
  "tool": "refund_customer",
  "args": {"customer_id": "8841", "amount": 49.0},
  "result": "ok",
  "approved_by": "diaz",
  "run_id": "9f3c2e"
}

That log line is what turns a mystery into an investigation. Storage costs less than the engineer-hours you’d spend reconstructing events by hand.

Last, write the severity ladder before you need it. Decide in advance what counts as severe: an action on production data, a leak of customer information, spend past a threshold, a customer-facing message that went wrong. Severity decided during the incident gets decided badly.

Detection: watch behavior, not just errors

Error rates tell you when an agent crashed. They tell you nothing when it’s confidently doing the wrong thing. So instrument behavior. Track tool-call volume and velocity, spend per hour, permission denials, calls to destinations the agent rarely touches, and how often humans override approval gates. A sudden jump in tool calls per minute is often the first honest signal that something is off, and you can turn it into an alert with the monitoring you already run.

Log enough context to answer why, not just what. That means prompts and retrieved content, not only actions. Here’s the tradeoff: full prompts often contain customer data, so log them with redaction and a retention policy you actually enforce. Depth speeds investigations. Retention bounds the exposure.

Containment: build and test the kill switch

The most valuable control here is an unambiguous way to stop an agent and revoke its credentials in minutes. Layers help: disable the deployment, revoke its tokens and OAuth grants, pause the queue feeding it, and fall back to human handling for whatever it was doing.

The tradeoff is autonomy against control. Every approval gate adds latency and human labor, so gate by blast radius rather than gating everything. Read-only lookups run free. Writes to production, payments, and outbound messages wait for a person. That keeps the fast path fast where mistakes are cheap.

And drill it. A kill switch nobody has tested is a hypothesis, not a control.

Eradication and recovery

Find the trigger before you restore autonomy: which prompt version, which tool change, which poisoned input. Rotate every credential the agent had, because you can’t verify what it read. Then bring it back in stages. Re-run your evaluation suite against the fix. Replay recorded traffic if you kept it. Run in shadow mode, where the agent proposes actions without executing them, then supervised, then autonomous.

Post-incident

Run a blameless postmortem like you would for any outage, with two additions. Turn the incident into a regression test in your eval suite, and re-audit the agent’s permissions. An incident is a cheap source of evidence that an agent was over-privileged.

This is also where governance connects. NIST’s AI Risk Management Framework, released in January 2023, organizes AI risk around four functions: govern, map, measure, manage. NIST followed it with a Generative AI Profile in July 2024 aimed at risks specific to generative AI. The useful idea is the loop. What an incident teaches you should change how you govern the agent, not just the prompt.

The tradeoffs in plain terms

Autonomy versus control is the big one. Guardrails slow agents down and consume human attention, so price each gate against the damage the gated action can do, and reprice when the tool changes.

Logging depth versus privacy is the second. More context makes investigations faster and exposure larger. Redaction and retention are how you buy a measure of both.

Buy versus build is the third. If you’re running agents on a managed platform, check what it already gives you, since audit logging, spend caps, and permission scopes are common platform features. Self-hosting keeps everything in your environment and gives you full control, but you own every one of those controls yourself. Small teams usually do better starting from platform defaults and tightening over time.

What it costs

The plan is mostly engineering time: permissions, caps, logging, alert thresholds, a runbook. Ongoing costs are log storage, monitoring, and the on-call attention that approval gates demand. The expensive version is the one you don’t have. It gets paid out in corrupted data and apology emails, with an API bill nobody budgeted for.

Decisions you can make this week

1. Inventory every credential your agents hold.
2. Cut each one to least privilege.
3. Set spend caps and tool-call limits.
4. Log every tool call, with prompt versions.
5. Build the kill switch and test it end to end.
6. Write a one-page runbook and rehearse a tabletop exercise against it.

Frequently asked questions

What counts as an AI agent incident? Any event where an agent took a wrong action, exposed data, or consumed resources abnormally. That includes prompt injection attempts and runaway loops, not just outages.

Do I need a separate incident response plan? No. Extend the plan you already have. The process is the same; what you add are agent-specific triggers, runbook entries, and containment steps.

How much logging is enough? Every tool call at minimum. Prompts and retrieved context where privacy allows, with redaction and a retention policy. The test is simple: if you can’t reconstruct an incident from your logs, you don’t have enough.

Can’t we just fix prompt injection? Not reliably today. Treat it as a containment problem. An agent that can only do limited damage survives injection when it succeeds. Least privilege is the defense that holds.

Who owns the plan? Name a person. Security and the team running the agents share the work, but a named owner keeps the plan from drifting.

How often should we rehearse? A quarterly tabletop is a reasonable floor, plus a drill whenever you give an agent a new tool.

If you’re wiring agents into your infrastructure and want a second pair of eyes on the controls before something goes sideways, get in touch.