Building a Safe Event-Driven DevOps Agent with AWS Lambda and Amazon DynamoDB — V2 Security Update

SlackOps separates diagnosis, approval, and deterministic pull-request execution with fixed AWS read adapters, immutable plans, short-lived credentials, restricted egress, and observed-capability gates. This V2 update explains what changed after the original deployment and what was verified on AWS.

Paul Choi

Technical Architect At Hyundai Autoever

Published Jun 23, 2026

Last modified Jul 17, 2026

Project url : https://devpost.com/software/slackops-one-agent-two-control-planes

V2 update — July 17, 2026. Giving an AI agent access to operational systems creates two opposing requirements: it needs enough context to investigate an incident, but it must not have enough authority to turn a bad instruction into a production change. The current design replaces the generic AWS MCP read surface with fixed adapters and adds immutable approval plans, short-lived credentials, deterministic pull-request execution, egress controls, and observed-capability gates.

This post preserves the original V1 architecture and documents the V2 hardening of SlackOps, a DevOps agent that runs Claude Code Headless on Amazon EC2. The design now separates evidence collection, model analysis, human approval, and deterministic pull-request execution into distinct runtime boundaries.

The goal is not to replace Amazon CloudWatch or an on-call engineer. SlackOps sits between an existing signal and the engineer who owns the response.

The Core Problem: Read Access and Write Access Are Different

A generic “human in the loop” rule creates the wrong trade-off.

If an engineer must approve every log query, automation adds latency to the safest part of incident response. If the agent can write first and ask later, approval is meaningless.

SlackOps divides commands into three levels:

Level	Scope	Examples	Runtime behavior
L0 — Observe	Read and analyze	Logs, diagnosis, governance scan	Runs automatically
L1 — Prepare	Create a reviewable artifact	Terraform review, code diff, pull request	Stops when a diff needs approval
L2 — Execute	Change live infrastructure	Deploy, apply, restart, IAM or database mutation	Disabled

The boundary is enforced by command registration, tool allowlists, AWS Identity and Access Management (IAM), and the job state machine. It does not depend on a prompt asking the model to behave safely.

Architecture: Two Control Planes, Four Producers

SlackOps exposes two operator control planes and four producers that enter the same DynamoDB state machine:

Slack uses Socket Mode for legacy synchronous commands, natural-language requests, notifications, and approval actions.
A Next.js dashboard on Vercel uses Amazon DynamoDB for queued jobs, results, audit history, metrics, chat, and approvals.

Amazon EventBridge and AWS Lambda provide the event-driven producer. A Lambda function converts an Amazon CloudWatch alarm state change into a pending proposal. A worker on EC2 claims queued jobs, collects bounded evidence through command-specific read adapters, and invokes Claude Code Headless with an empty AWS tool allowlist for L0 analysis.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
CloudWatch ALARM → EventBridge → Lambda (proposal only)
                              │
Slack DM / commands → Amazon DynamoDB ← Web dashboard
                              │ conditional claim
                              ▼
                    EC2 SlackOps worker
                  ┌───────────┴───────────┐
                  │                       │
       fixed read adapters       approved write path
       bounded evidence          plan re-verification
       <untrusted_data>           short-lived GitHub token
       L0 tools = 0               deterministic open_pr
                  │                       │
                  ▼                       ▼
          read-only AWS data          GitHub pull request                                          ┌────────────────────────┐
Amazon CloudWatch alarm ──► EventBridge ──► AWS Lambda             │
                                          │ detect() → propose     │
                                          └───────────┬────────────┘
                                                      │ PENDING
                                                      ▼
┌─────────────────────┐                 ┌────────────────────────────┐
│ Vercel dashboard    │◄───────────────►│ Amazon DynamoDB           │
│ jobs / approvals    │                 │ Job / Audit / Metric      │
└─────────────────────┘                 └────────────┬───────────────┘
                                                     │ conditional claim
                                                     ▼
┌─────────────────────┐                 ┌────────────────────────────┐
│ Slack Socket Mode   │◄───────────────►│ Amazon EC2                │
│ commands / notices  │                 │ worker + Claude Headless │
└─────────────────────┘                 └────────────┬───────────────┘
                                                     │ read-only tools
                                                     ▼
                                        ┌────────────────────────────┐
                                        │ AWS API MCP               │
                                        │ CloudWatch / Config / SSM │
                                        └────────────────────────────┘

The dashboard and DynamoDB table remain available continuously. EC2 is provisioned only when the Claude worker is needed. Lambda can still accept an alarm and persist a job while the worker is offline.

Path 1: Synchronous Diagnosis from Slack

Slack provides the shortest path for an operator who is already responding to an incident:

1
2
3
4
5
6
/devops diagnose checkout-service
→ validate the service argument
→ collect bounded AWS evidence through a fixed read adapter
→ sanitize and isolate it as <untrusted_data>
→ invoke Claude Code Headless with L0 tools = 0
→ return the diagnosis to Slack

Socket Mode is an outbound connection, so the application does not need a public inbound endpoint. Administrative access to EC2 uses AWS Systems Manager Session Manager rather than SSH.

The EC2 instance profile is bootstrap-only. Root provisions separate one-hour runtime, internal-MCP, and audit credentials and refreshes them every 45 minutes. Agent services cannot query IMDS directly. For L0 diagnosis, the application fetches bounded evidence through fixed adapters and the Claude process receives no AWS tool.

Legacy /devops slash commands still execute synchronously. The primary natural-language Slack DM path uses the DynamoDB conversation bus and proposes jobs into the shared queue. Keeping both paths explicit avoids claiming that every Slack interaction has the same execution model.

Path 2: Event-Driven Triage from a CloudWatch Alarm

The asynchronous path starts with a CloudWatch alarm changing to ALARM:

1
2
3
4
5
6
7
8
CloudWatch ALARM
    → EventBridge rule
    → Lambda alarm producer
    → deterministic signal mapping
    → DynamoDB PENDING job
    → EC2 worker claim
    → read-only diagnosis
    → DONE + Slack notification + dashboard result

The alarm name and state reason are untrusted data. Lambda does not turn either value into a free-form command. A deterministic detect() function searches for known markers such as 5xx errors, timeouts, or restarts, then selects a command from a fixed registry and validates the service argument.

This separation gives Lambda two narrow responsibilities:

Map a known signal to a permitted proposal.
Persist that proposal in DynamoDB.

Lambda does not invoke Claude and does not modify infrastructure.

In the live test, a 5xx alarm produced an L0 diagnose job. The worker ran it automatically because diagnosis is read-only. There was no approval step. Approval applies only when a workflow produces a code diff.

Why keep detection serverless?

The alarm can be recorded while EC2 is offline. DynamoDB preserves the job until the worker is available, which decouples signal ingestion from the most expensive runtime component.

Path 3: A Pull Request Bound to the Approved Diff

The pull-request workflow has two stages.

Prepare

Claude creates a branch, changes code, runs tests, and returns a diff. The prepare-stage tool list explicitly removes git push and gh pr create.

When the worker receives a diff, it stores the artifact and moves the job to AWAITING_APPROVAL.

Execute

An operator reviews the diff in the Vercel dashboard and approves or rejects it. Approval uses a conditional DynamoDB update:

1
2
Condition: status = AWAITING_APPROVAL
Update:    status = APPROVED

Only the first valid transition succeeds. A second approval receives a conditional-check failure instead of triggering duplicate work.

After approval, the worker reclaims the job and re-verifies the canonical ExecutionPlan, including the request and diff hashes, paths, workspace, policy, tool chain, capabilities, and risk ceiling. It then mints a repository-scoped GitHub App installation token and calls app.pr_execution.open_pr, which performs a fixed branch → add approved paths → commit → push → pull-request sequence without an LLM in the write path. The token is revoked on every exit path, and the remote path set must match before the job becomes DONE.

The two state machines are intentionally different:

1
2
3
4
5
6
7
L0 read path:
PENDING → RUNNING → DONE | FAILED

L1 pull-request path:
PENDING → RUNNING → AWAITING_APPROVAL
        → APPROVED → RUNNING → DONE | FAILED
        → REJECTED

DynamoDB: Coordination, Not Just Persistence

The table stores jobs and the records needed to operate them:

Item	Key pattern	Access pattern
Job	PK=JOB#<id>, SK=META	Current state and result
Audit	PK=JOB#<id>, SK=AUDIT#<time>	Per-job lifecycle history
Metric	PK=JOB#<id>, SK=METRIC#<time>	Duration, tokens, and cost
Detection config	PK=CONFIG#detections	Scheduled and on-demand scan settings
Conversation	PK=CHAT#<id>	Dashboard chat bus

Two global secondary indexes support status claims and feed queries. The Python worker and TypeScript dashboard implement the same key and transition contract.

Two conditional writes are central to the design:

Atomic claim: a worker can change a job to RUNNING only when the stored state matches the expected claimable state.
Approval lock: a dashboard action can change a job to APPROVED or REJECTED only from AWAITING_APPROVAL.

Why use conditional writes?

The worker polls, the dashboard updates jobs, and event producers can write concurrently. A read followed by an unconditional update would allow two workers or two operators to act on stale state. The condition moves conflict detection into DynamoDB, so the design does not need a separate lock service or queue coordinator.

The Permission and Identity Model

Each runtime receives a different AWS identity.

EC2 worker

The instance profile is used only to bootstrap the host. The worker runs with a short-lived runtime role limited to the required fixed AWS reads and DynamoDB control-plane access. A separate root-only audit role can append to the deployment-owned security log, while the runtime role is explicitly denied from writing to that sink.

Lambda producer

Lambda needs only the permissions required to read its notification configuration and write a proposal to DynamoDB. It does not receive the worker's diagnostic tool surface.

Vercel dashboard

Vercel runs outside AWS, so it cannot use the EC2 instance profile. The deployment uses a table-scoped IAM credential. Read-only deployments need GetItem and Query; enabling job creation or approval additionally requires narrowly scoped PutItem and UpdateItem. The credential is stored as a Vercel environment variable and can be revoked after evaluation.

This is why “no stored AWS keys” applies specifically to the EC2 agent runtime, not to every component in the system.

Prompt-Injection Boundaries: Reduce the Sinks

Prompt injection enters through more than one path, and each path needs a different control.

Operator input

Service names and command arguments are validated for length and allowed characters. Free text is placed in fixed prompt templates as untrusted data. Unknown commands are denied by default.

Tool selection

Each command receives a fixed tool allowlist. The PR prepare stage removes push and PR-creation tools. L2 actions remain disabled even if an input asks for one.

Fixed adapter results

CloudWatch and other operational evidence is fetched by command-specific adapters, capped, sanitized, and placed inside the same <untrusted_data> boundary as operator input. The L0 model tool allowlist is empty. This does not claim to solve prompt injection; it reduces the action path available when the model is misled.

Generated output

A diff is data to review, not authorization to execute. The worker cannot reach the push stage until DynamoDB contains a valid approval transition.

The important lesson is that a sanitizer, an IAM policy, and an approval gate solve different problems. None of them replaces the others.

Telemetry and Operating Cost

Queued executions append audit events and write duration, token count, cost, success, and error records to DynamoDB. The dashboard aggregates these records by command and displays recent runs.

The worker now uses Claude stream-json output to reconstruct observed tool calls. Each call is resolved through the same argv guard, recorded in the hash-linked audit trajectory, and re-aggregated into observed capabilities. A capability that exceeds the approved set or score fails the job with capability_drift. A live AWS Distro for OpenTelemetry export pipeline remains future work.

The operating model keeps the persistent components inexpensive:

Vercel serves the dashboard.
DynamoDB uses on-demand capacity.
EventBridge and Lambda remain available for signal ingestion.
EC2 runs only during a demonstration or requested live test.

One event-driven diagnosis displayed an API-equivalent Claude estimate of approximately $0.15. The project currently authenticates with a Claude subscription OAuth token, so this is a client-side comparison metric, not the amount personally billed. After live verification, I terminated the EC2 instance; infrastructure cost claims in this post are operating-scenario estimates rather than a copy of one account bill.

Live Validation

I ran the following path on AWS:

1
2
3
4
5
6
7
8
CloudWatch ALARM
  → EventBridge
  → Lambda
  → DynamoDB proposal
  → EC2 worker
  → Claude read-only diagnosis
  → DONE
  → Slack notification

The validation also covered:

Slack /devops ping from the EC2 runtime
CloudWatch diagnosis through fixed read adapters and a short-lived runtime role
An AWS write attempt rejected with denied by security policy
Vercel reading the deployed DynamoDB table
Duplicate approval rejected by a conditional update
563 automated tests, Ruff, strict mypy, and the documentation budget gate passing on July 17, 2026

The checkout-service incident was staged. The alarm transition, EventBridge delivery, Lambda invocation, DynamoDB state changes, Claude execution, and Slack notifications were real.

Known Limitations

EC2 is not always on. Lambda can queue an event while the worker is offline, but diagnosis waits until the worker starts.
The system consumes existing signals; it is not a replacement for CloudWatch, Datadog, PagerDuty, or an SLO platform.
L2 infrastructure execution is intentionally unavailable.
The pull-request implementation is human-gated, but GitHub branch protection and the final merge remain external controls.
ADOT export remains future work. Worker tool calls are now observed through stream-json and checked by the capability-drift gate.

What This Actually Solves

The design provides independent control over:

Triage latency: L0 work does not wait for approval.
Mutation authority: a code diff cannot cross the DynamoDB approval state on its own.
Concurrency: conditional writes reject stale claims and duplicate approvals.
Credential scope: EC2, Lambda, and Vercel receive separate permissions.
Idle cost: event ingestion and the dashboard remain available without an always-on Claude worker.

The result is a bounded response layer: existing monitoring detects the condition, SlackOps performs read-only triage or prepares a reviewable change, and a human retains control of mutation.

Summary

Diff → plan-bound approval → JIT token → deterministic PR

Concern	Implementation
Operator interface	Slack Socket Mode + Next.js on Vercel
Event ingestion	CloudWatch alarm → EventBridge → Lambda
Coordination	DynamoDB single table + conditional writes
Agent runtime	Claude Code Headless on EC2
AWS access	Bootstrap-only profile + short-lived STS + fixed adapters
Read path	L0 runs automatically
Write path	Diff → conditional human approval → pull request
Audit	Hash-linked trajectory + observed capability gate
Cost control	EC2 provisioned only when needed

The core rule is straightforward: observation may run automatically, but mutation requires a human-controlled state transition.

I created this article for the purposes of entering the #H0Hackathon.

# devops # claude # amazon-dynamodb # amazon-eventbridge # amazon-cloudwatch

Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.

Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article

Builder Center

Building a Safe Event-Driven DevOps Agent with AWS Lambda and Amazon DynamoDB — V2 Security Update

Paul Choi

The Core Problem: Read Access and Write Access Are Different

Architecture: Two Control Planes, Four Producers

Path 1: Synchronous Diagnosis from Slack

Path 2: Event-Driven Triage from a CloudWatch Alarm

Path 3: A Pull Request Bound to the Approved Diff

Prepare

Execute

DynamoDB: Coordination, Not Just Persistence

The Permission and Identity Model

EC2 worker

Lambda producer

Vercel dashboard

Prompt-Injection Boundaries: Reduce the Sinks

Operator input

Tool selection

Generated output

Telemetry and Operating Cost

Live Validation

Known Limitations

What This Actually Solves

Summary