AWS Builder Center

AWS DevOps Agent - Testing directed actions end to end

AWS DevOps Agent investigates incidents on its own, but never changes your infrastructure without a human approving each action. This article walks the full loop on a real serverless failure — alarm, autonomous investigation, root cause, and a human-approved directed action that fixes it — with the safety model behind approvals (layered opt-in, an elevated IAM role as the hard ceiling, per-action approval cards) and the gotchas we hit driving it.

The scenario

The setup only needs to produce a genuine, repeatable incident to point a directed action at. Here it is a checkout API that shares an account-level Lambda concurrency pool with a batch workload. When the batch workload drains the pool, the checkout API is starved of concurrency and starts throttling — customers see errors, even though the checkout function itself is perfectly healthy. That "the victim is not the cause" shape is what makes it a good test: the fix lives on a different resource than the one that alarmed. Any incident works; the rest of the article is about what the agent does with it and how the fix gets approved.

How directed actions work (the safety model)

Worth stating plainly, because it shapes everything:
  • Read-only actions (describe/list/investigate) are on by default.
  • Directed actions (anything that mutates) are off by default and require layered opt-in: enable them on the Agent Space, register an elevated IAM role per account, and approve each action at execution time.
  • The agent builds each approval's session policy only from a curated list of supported actions. An action outside that list can never run, even if the IAM role allows it.
  • Hard guardrails apply regardless of approval: no delete-class operations, no  iam:PassRole .
So there are two independent gates below every approval: the curated action list and the elevated IAM role. A human clicking "approve" cannot exceed either.

Following the investigation

When the alarm forwards the incident to the Agent Space webhook, an investigation starts on its own, no one paged. The Summary and Timeline update live as the agent correlates metrics, logs, and CloudTrail.
The insight to watch for is the cross-resource reasoning: the function that alarmed is healthy but starved, and the real cause is elsewhere. The agent also correlates the onset with the specific change that triggered it, from CloudTrail. This is exactly the reasoning an on-call engineer would spend the 2 a.m. hour on, and it is done before anyone opens the console.
At this point the agent has proposed a fix. Nothing has touched the infrastructure.

Approving a directed action

This is the part most people get wrong the first time, so it is worth being precise.
Directed actions only materialize in the Agent Space Chat, not inside the investigation conversation. A remediation request typed into the investigation is declined — the agent replies that its role there is limited to investigating and proposing mitigations. The same request in the Chat produces a real approval card. Same agent, same configuration, different surface.
In the Chat, ask for the specific mitigation, for example: "Cap the report-batch SQS event source mapping at MaximumConcurrency=50."
The agent responds with an approval card and pauses. The card shows the exact API operation, the target resource, the parameters, a risk assessment, the blast radius, and a rollback note. Nothing runs until the operator decides.
Two controls are worth knowing before approving:
  • Parameters can be narrowed (tighten a proposed value), but never widened.
  • The approval is single-use or carries a reuse window of up to 4 hours. The reuse window is the closest thing to standing autonomy — useful for a recurring, well-understood action, and still bounded and auditable.
On approval, the agent executes under scoped credentials from the elevated role, and the action is attributed to the approving operator in CloudTrail. The incident clears shortly after.

The finding worth remembering

An approved action can still be blocked. During testing, an action the operator approved failed at execution because the elevated role did not include that particular resource. The operator's "yes" is necessary, not sufficient — the IAM ceiling defined up front is the real limit. That is defense in depth at no extra cost, and the strongest reason directed actions are safe to enable: the approval is human, but the boundary is IAM.

Caveats worth stating

  • The right remediation is usually to cap or reserve, not to disable — keep the legitimate workload running while protecting the victim. Prefer the reversible, bounded action.
  • Which actions are allowed at all is an IAM-and-curation decision made once, deliberately. Keep the elevated role tight; it is the last line of defense under every approval.

Takeaway

Detection and root cause are autonomous; the fix is human-approved by design. The agent does the detective work; the operator stays in control of the change, with the exact operation, blast radius, and rollback in front of them, and an IAM ceiling underneath that no approval can exceed.

References

Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article