Designing SecOps Agent: a multi-agent incident response system in AWS (work in progress)
This isn't a 'this is how I finished my project' post. It's a 'this is how I'm designing something calmly, after rushing cost me an entire hackathon' post — and I think that part is worth sharing too.
Where does this come from
My previous project was an agent pipeline for business analytics (natural language → SQL → charts), built in six days for a Kiro + AWS bootcamp. It worked, technically — but I didn’t manage to deliver it on time because access to the Bedrock models on my account took longer to get approved than the competition lasted. AWS solved the issue for me a few hours after the deadline closed. That irony is what made me choose my next project without the same rush.
What I really wanted to build from the start was a SecOps Agent: a system that receives security alerts, classifies them, correlates them with threat intelligence, and — the part I’m most interested in — carefully decides when it can act on its own and when it needs a human to approve the action. This time, without an artificial deadline, I’m designing it in layers, starting with the most boring and most important one: the layer that decides whether to trust an alert.
The map: incident response cycle → agents
I used the NIST cycle (Identify → Contain → Eradicate → Recover → Lessons Learned) as the backbone of the architecture, not as decoration:
Identificar → Router + Triage Agent (en construcción)Contener/Erradicar → Remediation Agent (diseño pendiente)Recuperar/Lecciones → Notifier Agent (diseño pendiente)
Starting with "Identify" was no accident—it is the only layer that can be built as a purely read-only component, making it the most logical place to make cheap mistakes early on.
The component I’ve already fully designed: Router + Triage Agent
This agent receives alerts via webhook, validates and normalizes them, classifies them by severity level (critical / high / medium / low / informational), correlates them with a threat knowledge base using RAG, and decides which agent to route them to—it never executes any actions itself. Three design decisions took more time to think through than to implement, and these are the ones I most want to document here.
- HMAC alone isn't enough I decided that alerts would arrive via a generic webhook—allowing any external source to register and send events, rather than locking myself into a single provider. This means I cannot trust an alert simply because it arrives in the correct format. The initial design validated authenticity using HMAC: each source has a shared secret and signs its payload before sending it. That sounds sufficient, but it isn't—HMAC proves the message comes from the claimed source, but it doesn't prove the message is new. Someone capturing a legitimate signed payload could replay it later, and the signature would remain valid. The fix was to bind the timestamp to the signature itself, rather than just verifying it separately:
signed_string = f"{timestamp}.{payload.decode('utf-8')}"signature = hmac.new(secret, signed_string.encode('utf-8'), hashlib.sha256)
And reject any request with a timestamp that differs from the server clock by more than a few minutes. This is a convention used by various SaaS platforms for their own webhooks; once you understand the reason why (a replay could reactivate a critical alert that has already been resolved, and this system might trigger automated downstream remediation), it stops feeling like paranoia and starts feeling like a reasonable minimum standard.
2. A three-tier remediation model based on "blast radius"
I haven't fully designed the Remediation Agent yet, but I have decided on the framework that will govern it, because it changes how I build everything that came before it:
Level | Example | ¿Automatic? |
|---|---|---|
Triage | Classify severity, correlate with threat intelligence | Yes — it is read-only. |
Reversible remediation, low risk | Temporary rate-limiting rule in WAF, with guaranteed expiration. | Yes, but with a guaranteed rollback. |
High-risk remediation | Isolate an EC2 instance, revoke credentials | Never automatic — requires explicit human approval. |
The underlying reason: with a generic webhook, a false alert (due to a third-party configuration error, or something worse) shouldn't be able to give an external party the ability to isolate your own infrastructure. The cost of a false positive at the first level is low; at the third level, it is high. The architecture should reflect that asymmetry, rather than treating them the same for the sake of coding convenience.
- "Immutable audit" is not just a term; it is a decision involving trade-offs.
The requirement stated that the audit log had to be immutable. It is easy to write that word, but harder to decide what it means in terms of actual IAM. CloudWatch Logs alone does not guarantee this—any role with sufficient permissions can delete a log group.
I ultimately decided—explicitly and with documentation, as a conscious trade-off—to use CloudWatch Logs with an explicit
Deny on logs:DeleteLogGroup and logs:PutRetentionPolicy for all application roles. This ensures system-level immutability: neither the agent itself nor a compromised version of it can erase its tracks. It does not protect against an account administrator with broad IAM permissions; that would require S3 with Object Lock in a separate account, and I consciously decided to leave that out of scope for now. I prefer a documented trade-off over an "immutable" promise I cannot fully keep.A lesson I carried over from my previous project: don't rely on a single LLM provider.
After what happened with Bedrock, I designed this system so that the classification can run on either Bedrock or the direct Anthropic API, depending on an environment variable—without changing a single line of the client-facing code. The correlation with threat intelligence does specifically rely on Bedrock Knowledge Bases (there is no direct equivalent in the other path), so that part degrades gracefully rather than failing completely if Bedrock is unavailable.
I didn't add this because I was facing the same problem again—in fact, I’ve already confirmed that Bedrock is enabled on this account. I added it because the lesson wasn't "Bedrock is unreliable"; it was "any managed service can have a bad spell, and building without an escape hatch is a gamble I’ve already lost once."
How I am working with Kiro during this cycle
One change I made compared to the previous project: this time, I’m using Kiro’s spec workflow (requirements.md in EARS format → design.md → tasks.md) much more deliberately, and I have a second AI review each document before moving on to the next. That extra step has already uncovered real issues—a replay attack vulnerability in the initial requirements.md, a canonicalization ambiguity regarding the HMAC signature in the design.md, and a configuration variable (the LLM model ID) that was left implicit in tasks.md and likely would have slipped past me until the code was already written.
It’s not that Kiro generates poor specs—it produces surprisingly comprehensive ones, incorporating property-based testing right from the design phase. It’s simply that having a second review before writing code is far less costly than discovering the same problem after deployment.
What's next?
- Complete the implementation of the Router + Triage Agent (already broken down into tasks, bottom-up: data models → deterministic utilities → LLM → ingestion handler → infrastructure).
- Designing the Remediation Agent using the already-defined three-level model—this is the part I care most about getting right, because it is where the system stops being "read-only" and begins making decisions with real-world consequences.
- Design the human approval mechanism for the high-risk level—I still don't know if it will be a separate endpoint, a Slack integration, or something simpler for this scope.
If any of this is useful for your own design, or if you spot a gap I might be overlooking, I’d love to hear about it—that is literally the point of this post, not to show off a finished project.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article