Closing the AI Pilot-to-Production Gap on AWS: The Three SRE Components Every Bedrock Agent Deployment Is Missing
This week a startup raised $50M to help enterprises move AI agents from stalled pilots to production. The pitch: "AI that survives contact with production."
For AWS teams running Amazon Bedrock Agents, the pilot-to-production gap has a concrete shape: an agent that performs well in a development environment with clean test data, predictable Lambda action group responses, and a small user base — deployed to production and exposed to real traffic, real tool failures, and real edge cases without the reliability infrastructure to detect or contain degradation.
This article covers the three SRE components that close this gap for Bedrock deployments: SLOs defined before go-live, a named ownership model backed by SSM Parameter Store, and a runbook that actually works at 2am.
No proprietary data referenced. All patterns are general AWS architecture.
Why Bedrock Pilots Fail in Production
The specific failure modes I see in Amazon Bedrock Agents transitions from pilot to production follow a consistent pattern:
Lambda action group failures surface differently in production. In development, Lambda functions are called with test payloads that succeed predictably. In production, real user requests expose edge cases: unexpected input formats, upstream API rate limits, Lambda cold starts under concurrent load. The Bedrock orchestration layer retries on ambiguous responses — and without a tool invocation efficiency baseline, a retry loop runs until the billing cycle closes.
Behavioral drift is invisible to standard CloudWatch. The Bedrock CloudWatch integration measures InvocationErrors, ModelLatency, and Throttles. It does not measure whether the agent's decision quality has drifted — whether it's selecting the right action groups, handling edge cases within expected behavioral bounds, or maintaining its escalation rate within defined limits. An agent can be technically healthy by every CloudWatch metric while producing systematically wrong outputs.
No named owner means no incident response. In every pilot I've observed fail in production, the investigation starts with "whose agent is this?" The answer is always "the AI team's" — which means nobody's pager fired, the incident escalated before anyone was contacted, and the first response action was taken by whoever noticed the problem rather than whoever understood the system.
Component 1: SLOs Before Go-Live — CloudWatch Custom Metrics
Define your agent's SLOs before the first production traffic hits. Publish behavioral SLIs to CloudWatch from day one.
Using the agentsre library with CloudWatch integration:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
from agentsre import AgentSLICollector, TaskRecord
from agentsre.cloudwatch import CloudWatchPublisher
# Define these at planning time — not after the first incident
PRE_DEFINED_SLOS = {
"DecisionQualityRate": {"target": 85.0, "unit": "Percent", "breach": "below"},
"ToolInvocationEfficiency": {"target": 1.5, "unit": "Ratio", "breach": "above"},
"HumanEscalationRate": {"target": 5.0, "unit": "Percent", "breach": "above"},
"ApprovalQueueDepthDrift": {"target": 2.0, "unit": "Ratio", "breach": "above"},
}
collector = AgentSLICollector()
publisher = CloudWatchPublisher(
agent_id="your-bedrock-agent",
dry_run=False,
)
# Record from your Bedrock invocation logging Lambda
def process_bedrock_trace(trace: dict):
collector.record(TaskRecord(
task_id=trace.get("sessionId"),
task_class=trace.get("metadata", {}).get("task_class", "general"),
tool_calls=len(trace.get("toolUses", [])),
decision_confidence=trace.get("modelOutput", {}).get("confidence", 1.0),
required_escalation=trace.get("humanEscalationRequired", False),
completed=trace.get("sessionState") == "COMPLETED",
))
# Publish every 5 min via EventBridge schedule
def publish_slis(event, context):
for task_class in collector._store.task_classes():
results = collector.collect(task_class)
publisher.publish(results)
# Alert on breach
for r in results:
if r.breached:
sns_client.publish(
TopicArn=get_sro_sns_arn(task_class),
Subject=f"[SLO BREACH] {r.name} — {r.task_class}",
Message=f"{r.name}: {r.value}{r.unit} breached threshold {r.threshold}{r.unit}\nDrift: {r.drift_ratio}x baseline",
)CloudWatch alarms — create one per SLI per task class before go-live:
1
2
3
4
5
6
7
8
9
10
11
12
aws cloudwatch put-metric-alarm \
--alarm-name "BedrockAgent-DQR-Breach" \
--namespace "AgentReliability" \
--metric-name "DecisionQualityRate" \
--dimensions Name=AgentId,Value=your-bedrock-agent \
--statistic Average \
--period 300 \
--threshold 85.0 \
--comparison-operator LessThanThreshold \
--evaluation-periods 3 \
--alarm-actions arn:aws:sns:REGION:ACCOUNT:AgentSROTopic \
--treat-missing-data notBreachingComponent 2: Named Owner Model — SSM Parameter Store + DynamoDB
Every Bedrock agent deployment must have a named SRO registered before go-live. Store ownership in SSM Parameter Store for alarm routing and DynamoDB for governance audit:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
import boto3
import json
from datetime import datetime, timezone
ssm = boto3.client('ssm')
dynamodb = boto3.resource('dynamodb')
table = dynamodb.Table('bedrock-agent-ownership')
def register_agent_sro(
agent_id: str,
agent_alias_id: str,
sro_name: str,
sro_email: str,
sro_sns_arn: str,
task_classes: list,
go_live_date: str,
):
"""
Register before go-live. Required. No exceptions.
"""
# SSM for alarm routing
ssm.put_parameter(
Name=f"/bedrock-agents/{agent_id}/sro/sns-arn",
Value=sro_sns_arn,
Type="String",
Overwrite=True,
Tags=[{"Key": "AgentId", "Value": agent_id}],
)
ssm.put_parameter(
Name=f"/bedrock-agents/{agent_id}/sro/email",
Value=sro_email,
Type="String",
Overwrite=True,
)
# DynamoDB for governance audit trail
table.put_item(Item={
"agent_id": agent_id,
"alias_id": agent_alias_id,
"sro_name": sro_name,
"sro_email": sro_email,
"task_classes": task_classes,
"go_live_date": go_live_date,
"slo_defined": True,
"runbook_url": f"https://your-wiki/runbooks/bedrock-{agent_id}",
"last_slo_review": datetime.now(timezone.utc).isoformat(),
"registered_at": datetime.now(timezone.utc).isoformat(),
})
def get_sro_sns_arn(task_class: str) -> str:
"""Retrieve SRO SNS ARN for alarm routing."""
# Look up agent responsible for this task class
agent_id = lookup_agent_for_task_class(task_class)
param = ssm.get_parameter(Name=f"/bedrock-agents/{agent_id}/sro/sns-arn")
return param["Parameter"]["Value"]Component 3: The Runbook — SSM Automation Document
Store the runbook as an SSM Automation Document so it's executable, not just readable:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
# bedrock-agent-incident-runbook.yaml
description: "Bedrock Agent Production Incident Runbook"
schemaVersion: "0.3"
parameters:
AgentId:
type: String
BreachedSLI:
type: String
TaskClass:
type: String
mainSteps:
- name: DetectAndAttribute
action: aws:executeScript
inputs:
Runtime: python3.11
Script: |
def script_handler(events, context):
# Check which SLI breached and recent deployment activity
import boto3
cw = boto3.client('cloudwatch')
# Pull last 4hr of DQR, TIE, HER for this agent
# Check CodePipeline for deployments in same window
return {
"primary_signal": events["BreachedSLI"],
"recent_deployment": check_recent_deployment(events["AgentId"]),
"investigation_layer": "semantic" if events["BreachedSLI"] in ["DQR","HER"] else "tool"
}
- name: ContainBlastRadius
action: aws:executeAwsApi
inputs:
Service: lambda
Api: PutFunctionConcurrency
FunctionName: "{{AgentId}}-action-group"
ReservedConcurrentExecutions: 2 # Reduce from normal capacity
- name: NotifySRO
action: aws:executeAwsApi
inputs:
Service: sns
Api: Publish
TopicArn: "{{resolve:ssm:/bedrock-agents/{{AgentId}}/sro/sns-arn}}"
Subject: "[ACTION REQUIRED] Bedrock Agent Incident — {{AgentId}}"
Message: "SLI breached: {{BreachedSLI}} / Task class: {{TaskClass}} / Runbook executing"Run the runbook when any SLO alarm fires — before a human even acknowledges the page. By the time the SRO opens the alert, blast radius is already contained.
The Pre-Go-Live Checklist
Before any Bedrock agent goes to production, confirm all five:
1
2
3
4
5
□ SLO targets defined for DQR, TIE, HER per task class
□ CloudWatch alarms created for each SLO
□ Named SRO registered in SSM Parameter Store and DynamoDB
□ Runbook stored as SSM Automation Document and tested
□ Tabletop exercise completed for top 2 failure scenariosThis checklist is the pilot-to-production gap, closed. Not with $50M of new tooling. With the SRE discipline that already exists.
Full open-source implementation: [GITHUB_URL]
LinkedIn discussion: [LINKEDIN_URL]
DEV Community conceptual article: [DEV_URL]
LinkedIn discussion: [LINKEDIN_URL]
DEV Community conceptual article: [DEV_URL]
What's the one item on this checklist your current Bedrock deployment is missing?
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article