AWS Builder Center
SRE Paranoia Gates on AWS Lambda-Backed Production Readiness Enforcement Using CodePipeline, DynamoDB, and CloudWatch.

SRE Paranoia Gates on AWS Lambda-Backed Production Readiness Enforcement Using CodePipeline, DynamoDB, and CloudWatch.

The SRE Paranoia Gate framework gives you five automated checks that encode failure-mode thinking into every production deployment. This article wires those gates to AWS-native services: CodePipeline approval actions, Lambda gate runners, DynamoDB gate history, and CloudWatch metrics for gate trend analysis.

Senior DevOps & SRE | IEEE Senior Member | AI Observability & Cloud Resilience
The SRE Paranoia Gate framework gives you five automated checks that encode failure-mode thinking into every production deployment. This article wires those gates to AWS-native services: CodePipeline approval actions, Lambda gate runners, DynamoDB gate history, and CloudWatch metrics for gate trend analysis.
Architecture
1
2
3
4
5
6
7
8
9
10
11
12
Developer pushes code
→ CodePipeline starts
→ Build + Unit Test stage (standard)
→ SRE Paranoia Gate Stage (Lambda invocation)
→ Gate 1: Rollback readiness (DynamoDB drill history lookup)
→ Gate 2: In-flight traffic (ECS task definition analysis)
→ Gate 3: Load test (S3 test artifact validation)
→ Gate 4: Blast radius (Service Catalog lookup)
→ Gate 5: Error budget (CloudWatch SLO metric read)
→ If all gates pass: continue to Deploy stage
→ If any gate fails: pipeline stops, SRE owner notified via SNS
→ All gate results logged to DynamoDB + CloudWatch
The Gate Runner Lambda
python
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
import boto3
import json
from datetime import datetime, timezone, timedelta
from agentsre.sre_paranoia_gates import (
SREParanoiaGateRunner,
gate_rollback_readiness,
gate_in_flight_traffic,
gate_degraded_load_test,
gate_blast_radius_survivability,
gate_error_budget,
GateResult
)

codepipeline = boto3.client('codepipeline', region_name='us-east-1')
dynamodb = boto3.resource('dynamodb', region_name='us-east-1')
cloudwatch = boto3.client('cloudwatch', region_name='us-east-1')
sns = boto3.client('sns', region_name='us-east-1')
s3 = boto3.client('s3', region_name='us-east-1')

GATE_HISTORY_TABLE = 'SREParanoiaGateHistory'
SNS_TOPIC = 'arn:aws:sns:us-east-1:ACCOUNT_ID:sre-paranoia-alerts'

def build_deployment_context(service_name: str,
pipeline_execution_id: str
) -> dict:
"""
Build gate evaluation context from AWS service state.

Pulls live data from:
- DynamoDB: rollback drill history
- ECS: task definition for graceful shutdown config
- S3: load test results artifact
- Service Catalog: blast radius definition
- CloudWatch: live SLO error budget metric
"""

# Rollback drill history from DynamoDB
drill_table = dynamodb.Table('RollbackDrillHistory')
drill_response = drill_table.query(
KeyConditionExpression='service_name = :svc',
ExpressionAttributeValues={':svc': service_name},
ScanIndexForward=False,
Limit=1
)
last_drill = drill_response['Items'][0] if drill_response['Items'] else {}

# ECS task definition for deployment config
ecs = boto3.client('ecs', region_name='us-east-1')
task_defs = ecs.list_task_definitions(
familyPrefix=service_name, sort='DESC', maxResults=1
)
task_def = {}
if task_defs.get('taskDefinitionArns'):
td = ecs.describe_task_definition(
taskDefinition=task_defs['taskDefinitionArns'][0]
)
container = td['taskDefinition']['containerDefinitions'][0]
task_def = {
'graceful_shutdown_seconds': td['taskDefinition'].get('stopTimeout', 0),
'connection_draining_enabled': container.get('essential', False)
}

# Load test artifact from S3
load_test_config = {}
try:
s3_response = s3.get_object(
Bucket='deployment-artifacts',
Key=f'{service_name}/load-test-results/latest.json'
)
load_test_config = json.loads(s3_response['Body'].read())
except Exception:
load_test_config = {
'tested_at_peak_load': False,
'tested_with_degraded_dependency': False
}

# Blast radius from Service Catalog / ARO registry
aro_table = dynamodb.Table('AgentReliabilityOwnership')
try:
aro = aro_table.get_item(Key={'agent_id': service_name})
blast = json.loads(aro.get('Item', {}).get('blast_radius', '[]'))
cannot_touch = json.loads(
aro.get('Item', {}).get('cannot_touch', '[]')
)
blast_radius = {
'downstream_service_count': len(blast),
'contains_payment_path': any(
'payment' in s.lower() for s in blast
),
'single_engineer_containable': len(blast) <= 4,
'circuit_breaker_configured': True
}
except Exception:
blast_radius = {
'downstream_service_count': 10,
'contains_payment_path': True,
'single_engineer_containable': False,
'circuit_breaker_configured': False
}

# Live error budget from CloudWatch
end = datetime.now(timezone.utc)
start = end - timedelta(hours=1)
budget_response = cloudwatch.get_metric_statistics(
Namespace='AgentSRE/ErrorBudget',
MetricName='ErrorBudgetRemaining',
Dimensions=[{'Name': 'ServiceName', 'Value': service_name}],
StartTime=start, EndTime=end,
Period=3600, Statistics=['Minimum']
)
budget_pts = budget_response.get('Datapoints', [])
budget_remaining = budget_pts[0]['Minimum'] if budget_pts else 100.0

return {
"rollback": {
"last_tested_at": last_drill.get('tested_at'),
"p95_duration_minutes": last_drill.get('p95_minutes', 99)
},
"deployment": {
**task_def,
"schema_backward_compatible": True
},
"load_testing": load_test_config,
"blast_radius": blast_radius,
"slo": {"error_budget_remaining_pct": budget_remaining},
"is_critical_change": False
}

def publish_gate_metrics(service_name: str, result: dict) -> None:
"""Publish gate pass/fail counts to CloudWatch for trend tracking."""
cloudwatch.put_metric_data(
Namespace='SRE/ParanoiaGates',
MetricData=[
{
'MetricName': 'GatesPassed',
'Dimensions': [{'Name': 'ServiceName', 'Value': service_name}],
'Value': float(result['gates_passed']),
'Unit': 'Count'
},
{
'MetricName': 'GatesFailed',
'Dimensions': [{'Name': 'ServiceName', 'Value': service_name}],
'Value': float(result['gates_failed']),
'Unit': 'Count'
},
{
'MetricName': 'DeploymentApproved',
'Dimensions': [{'Name': 'ServiceName', 'Value': service_name}],
'Value': 1.0 if result['deployment_approved'] else 0.0,
'Unit': 'None'
}
]
)

def store_gate_history(service_name: str,
pipeline_id: str,
result: dict
) -> None:
"""Log gate evaluation to DynamoDB for audit and trend analysis."""
table = dynamodb.Table(GATE_HISTORY_TABLE)
table.put_item(Item={
'service_name': service_name,
'evaluated_at': result['evaluated_at'],
'pipeline_execution_id': pipeline_id,
'deployment_approved': result['deployment_approved'],
'gates_passed': result['gates_passed'],
'gates_failed': result['gates_failed'],
'gates_warned': result['gates_warned'],
'gate_results': json.dumps(result['results']),
'ttl': int(
(datetime.now(timezone.utc) + timedelta(days=90)).timestamp()
)
})

def lambda_handler(event: dict, context) -> dict:
"""
SRE Paranoia Gate Lambda — invoked as CodePipeline action.

Evaluates all five gates against live AWS service state.
On pass: signals CodePipeline to continue.
On fail: signals failure, alerts SRE owner via SNS.
All results logged to DynamoDB and CloudWatch.
"""

# Extract CodePipeline context
job_id = event.get('CodePipeline.job', {}).get('id', 'local-test')
job_data = event.get('CodePipeline.job', {}).get('data', {})

user_params = json.loads(
job_data.get('actionConfiguration', {})
.get('configuration', {})
.get('UserParameters', '{}')
)
service_name = user_params.get('service_name', 'unknown-service')
sre_owner = user_params.get('sre_owner', 'platform-sre')
pipeline_id = job_data.get('pipelineContext', {}).get('pipelineExecutionId', 'test')

# Build context from live AWS state
deploy_context = build_deployment_context(service_name, pipeline_id)

# Run all five SRE Paranoia Gates
runner = SREParanoiaGateRunner(service_name, sre_owner)
runner.register(gate_rollback_readiness)
runner.register(gate_in_flight_traffic)
runner.register(gate_degraded_load_test)
runner.register(gate_blast_radius_survivability)
runner.register(gate_error_budget)

result = runner.run_all(deploy_context)

# Store history and publish metrics
store_gate_history(service_name, pipeline_id, result)
publish_gate_metrics(service_name, result)

if not result['deployment_approved']:
# Alert SRE team
failed_gates = [
r for r in result['results'] if r['result'] == 'fail'
]
sns.publish(
TopicArn=SNS_TOPIC,
Subject=f"[SRE Gate BLOCKED] {service_name} — {len(failed_gates)} gate(s) failed",
Message=json.dumps({
"service": service_name,
"pipeline_execution_id": pipeline_id,
"failed_gates": failed_gates,
"decision": result['decision']
}, indent=2)
)

# Fail the CodePipeline action
if job_id != 'local-test':
codepipeline.put_job_failure_result(
jobId=job_id,
failureDetails={
'type': 'JobFailed',
'message': result['decision'],
'externalExecutionId': pipeline_id
}
)
return {'statusCode': 422, 'body': json.dumps(result)}

# Success — continue pipeline
if job_id != 'local-test':
codepipeline.put_job_success_result(jobId=job_id)

return {'statusCode': 200, 'body': json.dumps(result)}
CloudWatch Dashboard Query — Gate Trend Analysis
1
2
3
4
5
6
7
8
9
# Which services are failing gates most often?
fields @timestamp, service_name, gates_failed, deployment_approved
| filter Namespace = 'SRE/ParanoiaGates'
| stats sum(GatesFailed) as total_failures,
sum(DeploymentApproved) as approvals,
count() as total_evaluations
by service_name
| sort total_failures desc
| limit 20
The Gate That Gets Added Most Often
In practice, Gate 3 (degraded dependency load testing) produces the most WARN results when teams first adopt this framework. Most teams test at nominal load. Almost nobody tests what happens when one upstream dependency is responding at 2000ms instead of 50ms, while the service is also handling peak traffic, while a retry loop has started amplifying request volume.
That's the scenario that produces your worst production incidents. And it's the one that's cheapest to test in staging and most expensive to discover at 3 AM.
Ajay Devineni | AWS Community Builder | Senior SRE/Platform Engineer
github.com/Ajay150313/agentsre
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article