
Building an AIOps Incident History Tracker with Amazon DynamoDB — Part04
Demo 4 answers all of that by adding Amazon DynamoDB as a persistent incident store; every Bedrock analysis is saved as a structured record the moment it completes.
Series: AWS-Bedrock (4 articles)
- …
- 4Building an AIOps Incident History Tracker with Amazon DynamoDB — Part04 This article
This is Part 4 of the AIOps on AWS series. Part 1 — Self-Healing Infrastructure | Part 2 — SNS Email Alerts | Part 3 — Slack Incident Bot
Disclaimer: This is a learning demo, not production-ready. Always test thoroughly in a dev or pre-prod environment before applying any automated remediation to production workloads.
Up until now, every incident in this pipeline was detected, analyzed by AI, and fixed automatically. Alerts went to email and Slack. But once the notification was sent, the data was gone, buried in CloudWatch log streams with no easy way to query it.
What if you could ask:
“How many CRITICAL incidents happened this week?**“What is our average MTTR? Which service fails most often?”
Demo 4 answers all of that by adding Amazon DynamoDB as a persistent incident store; every Bedrock analysis is saved as a structured record the moment it completes.
What We Added
The self-healing pipeline from the previous demos remains unchanged. One new step was inserted after Bedrock analysis:
[Existing] Bedrock Analysis → SSM Remediation → Email + Slack ↓ [NEW] DynamoDB Structured incident record with MTTR + TTL auto-cleanupEvery incident — whether auto-remediated, skipped, or in maintenance mode — is now a queryable DynamoDB item.
DynamoDB Table Design
Table Name : aiops-incidentsPartition Key : instance_id (String) — groups incidents per serverSort Key : timestamp (String) — enables time-range queriesTTL Attribute : ttl (Number) — auto-deletes after 90 daysEach item stores everything from the Bedrock analysis plus remediation outcome and MTTR:
Attribute Example Value
incident_id uuid-xxxx-xxxxservice nginxseverity CRITICALissue Nginx stopped after receiving SIGQUITroot_cause Full Bedrock explanationremediation_action nginx_service_stoppedremediation_status successmttr_ms 9535ssm_command_id cmd-xxxxxxttl Unix timestamp — auto-delete dateMTTR Tracking
MTTR (Mean Time To Resolution) is one of the most important SRE metrics. The pipeline now calculates it automatically:
pipeline_start = time.time() # recorded at Lambda start# ... detection, analysis, SSM fix ...pipeline_end = time.time()mttr_ms = int((pipeline_end - pipeline_start) * 1000)From the logs:
Incident saved to DynamoDB | ID: cc68ee06 | MTTR: 9535ms | Severity: HIGH9.5 seconds from detection to fix — fully automated.
TTL — Automatic Data Cleanup
DynamoDB TTL automatically deletes records older than 90 days — no cron jobs, no manual cleanup, no extra cost. The TTL field stores a Unix timestamp:
python
ttl_timestamp = int(time.time()) + (90 * 24 * 60 * 60)When that timestamp passes, DynamoDB removes the item silently. You can adjust the retention period via the
TTL_DAYS environment variable.Silent Failure Design
Consistent with every previous demo, the DynamoDB write is wrapped in a try/except that never raises:
try: table.put_item(Item=item) print(f"Incident saved | MTTR: {mttr_ms}ms")except Exception as e: print(f"⚠️ Failed to save to DynamoDB: {str(e)}") # Pipeline continues regardlessIf DynamoDB is unreachable, the self-healing and alerting pipeline continues unaffected.
What the Full Pipeline Looks Like Now
EC2 Nginx failure ↓CloudWatch → Lambda ↓Pre-check: startup only? → exit earlyPre-check: maintenance mode? → save to DB + alert only ↓Bedrock (Claude Sonnet) — AI analysis ↓SSM Run Command — auto-remediation ↓DynamoDB ← incident record + MTTR [NEW] ↓SNS → Email + Slack → #aiops-alertsQuerying Your Incident History
Once records are in DynamoDB, you can query them directly from the console or CLI:
All incidents for a specific instance:
aws dynamodb query \ --table-name aiops-incidents \ --key-condition-expression "instance_id = :id" \ --expression-attribute-values '{":id":{"S":"i-0b9dd6a96640e905e"}}'All CRITICAL incidents (scan with filter):
aws dynamodb scan \ --table-name aiops-incidents \ --filter-expression "severity = :s" \ --expression-attribute-values '{":s":{"S":"CRITICAL"}}'

{ "instance_id": { "S": "i-0b9dd6a96640e905e" }, "timestamp": { "S": "2026-07-05T19:13:40.518Z" }, "estimated_impact": { "S": "All HTTP/HTTPS traffic served by Nginx on instance i-0b9dd6a96640e905e is completely unavailable from 2026-07-05 19:13:22 UTC onward. All downstream users, services, and load balancer targets relying on this instance are affected until Nginx is restarted." }, "exit_code": { "S": "0" }, "fix": { "S": "aws ssm send-command --instance-ids i-0b9dd6a96640e905e --document-name AWS-RunShellScript --parameters commands='sudo systemctl start nginx && sudo systemctl status nginx'" }, "incident_id": { "S": "c1a79ca5-c3a9-494f-a1ea-3254b5fe81b8" }, "issue": { "S": "Nginx has stopped due to graceful shutdown triggered by SIGQUIT signal" }, "log_group": { "S": "/aiops/ec2/nginx/error-logs" }, "model_used": { "S": "global.anthropic.claude-sonnet-4-6" }, "mttr_ms": { "N": "12903" }, "prevention": { "S": "1. Enable systemd service auto-restart: set 'Restart=always' and 'RestartSec=5s' in /etc/systemd/system/nginx.service.d/override.conf. 2. Configure a health check (e.g., AWS Route 53 health check or ELB health check) to alert on Nginx downtime. 3. Audit and restrict which processes or users can send signals to the Nginx master process. 4. Set up CloudWatch alarm on Nginx process count metric to trigger SNS notification when count drops to 0. 5. Review why PID 1 sent SIGQUIT — check systemd logs with 'journalctl -u nginx --since \"2026-07-05 19:13:00\"' to determine if shutdown was triggered by a deployment, OOM event, or misconfigured dependency." }, "remediation_action": { "S": "nginx_service_stopped" }, "remediation_status": { "S": "success" }, "root_cause": { "S": "Nginx master process (PID 27164) received signal 3 (SIGQUIT) from PID 1 (init/systemd), initiating a graceful shutdown. Both worker processes (27165 and 27166) completed their in-flight requests and exited with code 0. The master process then exited, leaving Nginx fully stopped on EC2 instance i-0b9dd6a96640e905e." }, "service": { "S": "nginx" }, "severity": { "S": "HIGH" }, "ssm_command_id": { "S": "d0670b4d-f637-454a-9a3f-c0a459bd73bb" }, "ttl": { "N": "1791054820" }}Key Lessons
On-demand capacity mode is the right choice for demo workloads. Provisioned capacity requires you to predict traffic, on-demand scales automatically and costs nothing when idle.
Design your keys around your queries. Using
instance_id as the partition key and timestamp as the sort key means every query for a specific server's incident history is a fast, efficient DynamoDB Query — not an expensive Scan.TTL is free and should always be enabled. Without it, your table grows indefinitely. With it, old data is cleaned up automatically at no extra cost. Always set a TTL on event-driven tables.
Full Source Code
The complete Lambda function for this demo is available on GitHub:
Series: AWS-Bedrock (4 articles)
- …
- 4Building an AIOps Incident History Tracker with Amazon DynamoDB — Part04 This article
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article