AWS Builder Center

Automating Patch Failure Investigations with AWS Systems Manager and AWS DevOps Agent

Learn how to connect AWS Systems Manager Patch Manager with AWS DevOps Agent to automatically collect patching logs, detect failures, and trigger autonomous root cause investigations — reducing patch failure triage from 30-60 minutes to under 5 minutes.

Architecture Overview

┌─────────────────────────────────────────────────────────────────────┐
│ Patch Maintenance Window │
│ (SSM Patch Manager - Weekly) │
└───────────────────────────────┬─────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────┐
│ SSM Run Command / State Manager │
│ Executes patching across target instance groups │
└───────────────────────────────┬─────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────┐
│ SSM Command Output → CloudWatch Logs │
│ /aws/ssm/patch-execution/{instance-id} │
│ Captures: stdout, stderr, exit codes, patch lists │
└───────────────────────────────┬─────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────┐
│ EventBridge Rule │
│ Filter: SSM Command Status = "Failed" OR "TimedOut" │
│ OR: Compliance State Changed → "NON_COMPLIANT" │
└───────────────────────────────┬─────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────┐
│ Lambda Function (Enrichment) │
│ - Collects SSM command output logs │
│ - Gathers instance metadata (OS, patch group, tags) │
│ - Retrieves patch baseline details │
│ - Packages context as investigation trigger │
└───────────────────────────────┬─────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────┐
│ AWS DevOps Agent (Webhook / Agent Space) │
│ - Receives enriched patch failure event │
│ - Investigates root cause autonomously │
│ - Correlates with recent deployments, config changes │
│ - Generates mitigation plan │
└───────────────────────────────┬─────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────┐
│ Slack / ServiceNow │
│ - Posts investigation findings │
│ - Root cause + recommended fix │
│ - Option to auto-remediate or create ticket │
└─────────────────────────────────────────────────────────────────────┘



Step 1: Configure SSM Patch Manager with Detailed Logging

First, ensure your patch operations capture enough detail for investigation. Configure your maintenance window task to send command output to CloudWatch Logs:
{
"TaskArn": "AWS-RunPatchBaseline",
"TaskType": "RUN_COMMAND",
"TaskInvocationParameters": {
"RunCommand": {
"Parameters": {
"Operation": ["Install"],
"RebootOption": ["RebootIfNeeded"]
},
"CloudWatchOutputConfig": {
"CloudWatchLogGroupName": "/aws/ssm/patch-execution",
"CloudWatchOutputEnabled": true
},
"TimeoutSeconds": 3600
}
}
}
Why this matters: By default, SSM truncates command output at 2,500 characters in the console. Sending to CloudWatch Logs captures the full output — every package attempted, every dependency conflict, every error message. DevOps Agent needs this detail to investigate effectively.

Additional Log Collection with a Post-Patch Script

We also run a post-patch SSM document that collects OS-level patching artifacts:
# SSM Document: CollectPatchLogs
schemaVersion: '2.2'
description: 'Collect detailed patching logs post-execution'
mainSteps:
- action: aws:runShellScript
name: collectLogs
inputs:
runCommand:
- |
#!/bin/bash
echo "=== PATCH LOG COLLECTION ==="
echo "Instance: $(curl -s http://169.254.169.254/latest/meta-data/instance-id)"
echo "Timestamp: $(date -u +%Y-%m-%dT%H:%M:%SZ)"
echo ""
# OS Detection
if [ -f /etc/os-release ]; then
. /etc/os-release
echo "OS: $NAME $VERSION_ID"
fi
echo ""
# Package manager logs
echo "=== RECENT PACKAGE OPERATIONS ==="
if command -v yum &> /dev/null; then
echo "--- YUM History (last 5 transactions) ---"
yum history list last-5
echo ""
echo "--- YUM Errors ---"
grep -i "error\|failed\|conflict" /var/log/yum.log 2>/dev/null | tail -20
elif command -v apt-get &> /dev/null; then
echo "--- APT History ---"
grep -A2 "Start-Date" /var/log/apt/history.log | tail -30
echo ""
echo "--- DPKG Errors ---"
grep -i "error\|failed\|conflict" /var/log/dpkg.log 2>/dev/null | tail -20
fi
echo ""
# Held/locked packages
echo "=== HELD/LOCKED PACKAGES ==="
if command -v yum &> /dev/null; then
yum versionlock list 2>/dev/null || echo "No version locks"
elif command -v apt-mark &> /dev/null; then
apt-mark showhold 2>/dev/null || echo "No holds"
fi
echo ""
# Disk space (common failure cause)
echo "=== DISK SPACE ==="
df -h / /var /tmp 2>/dev/null
echo ""
# Reboot pending check
echo "=== REBOOT STATUS ==="
if [ -f /var/run/reboot-required ]; then
echo "REBOOT REQUIRED"
cat /var/run/reboot-required.pkgs 2>/dev/null
else
echo "No reboot pending"
fi



Step 2: EventBridge Rule to Detect Patch Failures

Create an EventBridge rule that triggers on SSM patch failures:
{
"source": ["aws.ssm"],
"detail-type": [
"EC2 Command Status-change Notification",
"Configuration Compliance State Change"
],
"detail": {
"$or": [
{
"status": ["Failed", "TimedOut", "Cancelled"]
},
{
"compliance-status": ["non_compliant"]
}
]
}
}
You can also add a rule for maintenance window execution failures:
{
"source": ["aws.ssm"],
"detail-type": ["Maintenance Window Execution State-change Notification"],
"detail": {
"status": ["FAILED", "TIMED_OUT"]
}
}



Step 3: Lambda Enrichment Function

The Lambda function bridges EventBridge and DevOps Agent by enriching the raw event with context:
import boto3
import json
import urllib3

ssm = boto3.client('ssm')
ec2 = boto3.client('ec2')
logs = boto3.client('logs')

DEVOPS_AGENT_WEBHOOK = "https://your-agent-space-webhook-url"

def lambda_handler(event, context):
detail = event.get('detail', {})
# Extract instance and command info
instance_id = detail.get('instance-id') or extract_instance_id(detail)
command_id = detail.get('command-id')
status = detail.get('status', 'Unknown')
# 1. Get instance metadata
instance_info = get_instance_info(instance_id)
# 2. Get SSM command output (the actual patch logs)
patch_output = get_command_output(command_id, instance_id)
# 3. Get patch compliance details
compliance = get_patch_compliance(instance_id)
# 4. Get recent CloudWatch logs for this instance
cw_logs = get_cloudwatch_patch_logs(instance_id)
# 5. Build investigation payload
investigation_payload = {
"event_type": "patch_failure",
"severity": classify_severity(instance_info, compliance),
"summary": f"Patch operation {status} on {instance_id} ({instance_info.get('name', 'unnamed')})",
"context": {
"instance": {
"id": instance_id,
"name": instance_info.get('name'),
"type": instance_info.get('instance_type'),
"os": instance_info.get('platform'),
"patch_group": instance_info.get('patch_group'),
"environment": instance_info.get('environment'),
"az": instance_info.get('availability_zone')
},
"patch_operation": {
"command_id": command_id,
"status": status,
"maintenance_window": detail.get('window-id'),
"baseline": compliance.get('baseline_id'),
"failed_patches": compliance.get('failed_patches', []),
"missing_patches": compliance.get('missing_count', 0)
},
"logs": {
"ssm_output": patch_output[:10000], # Truncate for payload size
"cloudwatch_logs": cw_logs[:5000]
}
},
"investigation_hints": [
"Check if disk space is insufficient for patch installation",
"Check for package dependency conflicts",
"Check if any packages are version-locked",
"Check if a recent deployment changed dependencies",
"Check if the instance can reach package repositories",
"Check if the patch baseline includes incompatible patches for this OS version"
]
}
# 6. Send to DevOps Agent webhook
trigger_investigation(investigation_payload)
return {"statusCode": 200, "body": "Investigation triggered"}

def get_instance_info(instance_id):
"""Get EC2 instance metadata and tags."""
try:
response = ec2.describe_instances(InstanceIds=[instance_id])
instance = response['Reservations'][0]['Instances'][0]
tags = {t['Key']: t['Value'] for t in instance.get('Tags', [])}
return {
"name": tags.get('Name', 'unnamed'),
"instance_type": instance['InstanceType'],
"platform": instance.get('PlatformDetails', 'Linux'),
"availability_zone": instance['Placement']['AvailabilityZone'],
"patch_group": tags.get('Patch Group', tags.get('PatchGroup', 'default')),
"environment": tags.get('Environment', 'unknown'),
"launch_time": str(instance['LaunchTime'])
}
except Exception as e:
return {"error": str(e)}

def get_command_output(command_id, instance_id):
"""Retrieve full SSM command output."""
if not command_id:
return "No command ID available"
try:
response = ssm.get_command_invocation(
CommandId=command_id,
InstanceId=instance_id
)
return (
f"Status: {response['Status']}\n"
f"Exit Code: {response.get('ResponseCode', 'N/A')}\n"
f"--- STDOUT ---\n{response.get('StandardOutputContent', '')}\n"
f"--- STDERR ---\n{response.get('StandardErrorContent', '')}"
)
except Exception as e:
return f"Error retrieving command output: {str(e)}"

def get_patch_compliance(instance_id):
"""Get patch compliance summary for the instance."""
try:
response = ssm.describe_instance_patch_states(
InstanceIds=[instance_id]
)
if response['InstancePatchStates']:
state = response['InstancePatchStates'][0]
return {
"baseline_id": state.get('BaselineId'),
"missing_count": state.get('MissingCount', 0),
"failed_count": state.get('FailedCount', 0),
"installed_count": state.get('InstalledCount', 0),
"operation": state.get('Operation'),
"operation_end_time": str(state.get('OperationEndTime', ''))
}
return {}
except Exception as e:
return {"error": str(e)}

def get_cloudwatch_patch_logs(instance_id):
"""Pull recent patch logs from CloudWatch."""
try:
response = logs.filter_log_events(
logGroupName='/aws/ssm/patch-execution',
logStreamNamePrefix=instance_id,
limit=50,
interleaved=True
)
messages = [e['message'] for e in response.get('events', [])]
return "\n".join(messages[-30:]) # Last 30 log entries
except Exception:
return "CloudWatch logs not available"

def classify_severity(instance_info, compliance):
"""Classify severity based on environment and failure scope."""
env = instance_info.get('environment', '').lower()
failed = compliance.get('failed_count', 0)
if 'prod' in env and failed > 5:
return "critical"
elif 'prod' in env:
return "high"
elif failed > 10:
return "high"
else:
return "medium"

def trigger_investigation(payload):
"""Send investigation trigger to DevOps Agent webhook."""
http = urllib3.PoolManager()
response = http.request(
'POST',
DEVOPS_AGENT_WEBHOOK,
body=json.dumps(payload),
headers={'Content-Type': 'application/json'}
)
return response.status



Step 4: DevOps Agent Investigation Skill

To make DevOps Agent smarter about patching investigations, we created a custom investigation skill that encodes our team's patching knowledge:
Skill: Patch Failure Investigation

When investigating a patch failure event:

1. TRIAGE: Determine if this is a single-instance issue or fleet-wide:
- Check if other instances in the same patch group also failed
- Check the maintenance window execution summary

2. COMMON ROOT CAUSES (check in this order):
a. Disk space exhaustion (/var, /tmp, / at >90%)
b. Package dependency conflicts (version pinning vs. baseline requirements)
c. Repository connectivity (can the instance reach package repos?)
d. Version locks preventing updates (yum versionlock, apt-mark hold)
e. Kernel updates requiring reboot but reboot blocked
f. Recent application deployment that changed shared libraries
g. SSM Agent version too old to support current patch operations
h. Maintenance window timeout (patches took longer than allowed)

3. CORRELATE:
- Check if any deployments occurred within 24 hours before the patch window
- Check if the instance's security group or NACL was recently modified
- Check if NAT Gateway or VPC endpoint has connectivity issues

4. REMEDIATION:
- Provide specific commands to resolve the issue
- Indicate if a re-run of the patch baseline is safe
- Flag if manual intervention is required (e.g., disk cleanup)
- Note if this is a recurring pattern from previous investigations

5. PREVENTION:
- Recommend a proactive fix to prevent recurrence
- Suggest monitoring improvements (e.g., disk space alarm threshold)



Step 5: Slack Notification with Investigation Results

Once DevOps Agent completes its investigation (typically 2-4 minutes), it posts findings to your configured Slack channel:
🔍 Patch Failure Investigation Complete

Instance: i-0abc123def456 (web-server-prod-03)
Patch Group: production-linux
Maintenance Window: mw-0123456789abcdef0
Status: FAILED (exit code 1)

📋 Root Cause:
The instance ran out of disk space on /var during patch installation.
/var was at 97% utilization (18.2GB/18.8GB) due to accumulated
Docker container logs that were not being rotated.

The patch operation for kernel-5.15.0-1062-aws required 340MB of
temporary space in /var/cache/apt which was unavailable.

🛠️ Mitigation Plan:
1. Clean Docker logs: `docker system prune --volumes`
2. Clear apt cache: `apt-get clean`
3. Re-run patch baseline: safe to retry after cleanup
4. Verify: `df -h /var` shows <70% utilization

⚡ Prevention Recommendation:
- Add CloudWatch alarm for disk utilization >80% on /var
- Configure Docker log rotation (max-size: 10m, max-file: 3)
- Consider moving Docker storage to a dedicated EBS volume

📊 Pattern Detected:
This is the 3rd occurrence of disk-related patch failures in the
production-linux patch group in the last 60 days. Recommend fleet-wide
Docker log rotation policy.






Key Learnings

  1. Log richness matters — The more context you send to DevOps Agent (SSM output, OS logs, disk stats, recent deployments), the more accurate the root cause analysis. Don't truncate.
  1. Investigation skills are your multiplier — Encoding your team's tribal knowledge about common patching failure modes into a skill makes DevOps Agent investigate like your most experienced SRE from day one.
  1. Correlation is the killer feature — DevOps Agent connecting a patch failure to a deployment that happened 6 hours earlier (which removed a shared library) is something no single engineer would check first. The agent checks everything in parallel.
  1. Prevention > Detection — The proactive recommendations from DevOps Agent (e.g., "add disk alarm," "implement log rotation fleet-wide") reduced our next cycle's failures by 70%. This compounding effect is where the real value lives.
  1. Start with production, expand to fleet — We started with production patch groups only (to manage alert volume) then expanded to staging/dev once we tuned the severity classification.



Resources

Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article