The Honesty Gap in SLO Reporting and How AWS CloudWatch Helps You Close It
The problem with 99.99%. Four nines allows 52.6 minutes of total downtime per year.
The problem with 99.99%
Four nines allows 52.6 minutes of total downtime per year.
That's 4.4 minutes per month. One minute per week.
When I audit how teams calculate that number, I find three consistent patterns that inflate it — all of them quietly accepted as industry standard, none of them visible to the customers who signed the SLA.
This post shows how to measure SLO compliance honestly using AWS CloudWatch, and why the "honesty gap" between your dashboard number and your user-experienced uptime matters for both trust and investment decisions.
The three exclusions that eat your credibility
Excluding third-party dependencies
Your auth provider went down. Your CDN had an incident. Your upstream payment processor had degraded throughput.
"Not our fault" is legally defensible. It is operationally useless to the customer who could not log in.
Averaging across regions
"Global uptime 99.99%" while us-east-1 was completely unavailable for 35 minutes is arithmetic that hides a real outage. The engineers who were paged at 2am know what actually happened.
The planned maintenance carve-out
This one is the most honest-sounding and the most self-defeating.
If you routinely need maintenance windows to deploy safely, your deployment process is the reliability problem. Excluding maintenance windows from your SLO means you never have incentive to fix deployments. You keep scheduling windows.
Measuring it honestly on AWS
Architecture
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
Your services
│
├── CloudWatch custom metrics (real-time incident data)
│ Namespace: PlatformReliability
│ Metrics: DowntimeMinutes, IncidentCount,
│ ExcludedMinutes, BurnRate
│
├── DynamoDB (incident store with exclusion audit trail)
│ Table: slo-incidents
│ GSI: by-started-at (time-series queries)
│
├── Lambda (SLO calculator, runs every 5 minutes)
│ Reads incidents from DynamoDB
│ Publishes compliance metrics to CloudWatch
│ Triggers SNS on budget breach
│
└── CloudWatch Dashboard
Four panels:
1. Achieved SLO % (configured policy)
2. Strict SLO % (no exclusions)
3. Error budget remaining (minutes)
4. Current burn rate (×)CloudWatch metric structure
python
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
import boto3
from datetime import datetime, timezone
cw = boto3.client("cloudwatch")
def publish_slo_metrics(
service_name: str,
achieved_pct: float,
strict_pct: float,
budget_remaining_minutes: float,
burn_rate: float,
policy: str,
) -> None:
"""
Publish SLO compliance metrics to CloudWatch.
The key: we publish BOTH the policy-adjusted number AND the
strict number. This makes the honesty gap visible in your
dashboard rather than hidden in a spreadsheet.
Dimensions include the policy name so you can compare
what each policy produces over time without re-running
historical calculations.
"""
now = datetime.now(timezone.utc)
namespace = "PlatformReliability"
base_dims = [
{"Name": "ServiceName", "Value": service_name},
{"Name": "ExclusionPolicy", "Value": policy},
]
metrics = [
{
"MetricName": "AchievedSLOPercent",
"Dimensions": base_dims,
"Value": achieved_pct,
"Unit": "Percent",
"Timestamp": now,
},
{
"MetricName": "StrictSLOPercent", # no exclusions applied
"Dimensions": base_dims,
"Value": strict_pct,
"Unit": "Percent",
"Timestamp": now,
},
{
"MetricName": "HonestyGapPercent", # the gap itself
"Dimensions": base_dims,
"Value": round(achieved_pct - strict_pct, 4),
"Unit": "Percent",
"Timestamp": now,
},
{
"MetricName": "ErrorBudgetRemainingMinutes",
"Dimensions": [{"Name": "ServiceName", "Value": service_name}],
"Value": max(0.0, budget_remaining_minutes),
"Unit": "Count",
"Timestamp": now,
},
{
"MetricName": "BurnRate",
"Dimensions": [{"Name": "ServiceName", "Value": service_name}],
"Value": burn_rate,
"Unit": "None",
"Timestamp": now,
},
]
# CloudWatch accepts up to 1000 metric data points per call
cw.put_metric_data(Namespace=namespace, MetricData=metrics)CloudWatch alarms — the ones that matter
python
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
def create_slo_alarms(service_name: str, slo_target: float) -> None:
"""
Create three alarms per service:
1. Budget breach: remaining budget hits zero
2. Fast burn: burning 14× faster than sustainable (page-worthy)
3. Honesty gap: exclusions are inflating the number by > 0.1%
(this one is unusual — most teams don't alarm on their own
measurement conventions, which is exactly why they should)
"""
cw = boto3.client("cloudwatch")
alarms = [
{
"AlarmName": f"{service_name}-error-budget-exhausted",
"AlarmDescription": (
f"Error budget for {service_name} is exhausted. "
f"Every further incident is a breach."
),
"MetricName": "ErrorBudgetRemainingMinutes",
"Namespace": "PlatformReliability",
"Dimensions": [{"Name": "ServiceName", "Value": service_name}],
"Period": 300,
"EvaluationPeriods": 1,
"Threshold": 0,
"ComparisonOperator": "LessThanOrEqualToThreshold",
"Statistic": "Minimum",
"TreatMissingData": "breaching",
},
{
"AlarmName": f"{service_name}-fast-burn-14x",
"AlarmDescription": (
f"Burn rate for {service_name} exceeds 14×. "
f"At this rate the monthly budget exhausts in ~50 hours."
),
"MetricName": "BurnRate",
"Namespace": "PlatformReliability",
"Dimensions": [{"Name": "ServiceName", "Value": service_name}],
"Period": 300,
"EvaluationPeriods": 2,
"Threshold": 14.4,
"ComparisonOperator": "GreaterThanOrEqualToThreshold",
"Statistic": "Average",
"TreatMissingData": "notBreaching",
},
{
"AlarmName": f"{service_name}-honesty-gap-high",
"AlarmDescription": (
f"Exclusions are inflating {service_name} SLO by > 0.1%. "
f"Review exclusion audit trail."
),
"MetricName": "HonestyGapPercent",
"Namespace": "PlatformReliability",
"Dimensions": [
{"Name": "ServiceName", "Value": service_name},
{"Name": "ExclusionPolicy", "Value": "standard"},
],
"Period": 3600,
"EvaluationPeriods": 1,
"Threshold": 0.1,
"ComparisonOperator": "GreaterThanThreshold",
"Statistic": "Maximum",
"TreatMissingData": "notBreaching",
},
]
for alarm in alarms:
cw.put_metric_alarm(**alarm)
print(f"Created alarm: {alarm['AlarmName']}")The cultural shift that matters more than the tooling
The metric is a lagging indicator of a cultural decision.
Teams that consistently hit their SLOs in my experience share one habit that teams gaming their SLOs don't have: they publish error budget burns in their incident reviews, not just in their internal dashboards.
When you have to explain to your stakeholders why you burned 60% of your monthly budget in 48 hours, your investment priorities change. The architectural debt that caused it stops being a backlog item and starts being next quarter's roadmap.
The engineering cultures I respect most treat an SLO breach as data, not as a failure. Data leads to root cause. Failure leads to blame. One of those produces better systems.
The open source implementation
Full implementation with DynamoDB incident store, Lambda calculator, exclusion audit trail, and CloudWatch dashboard JSON:
github.com/Ajay150313/slo-burn
The core calculator runs without any AWS dependencies (pure Python, zero imports beyond stdlib) so you can use it locally before wiring up CloudWatch.
bash
1
2
3
pip install slo-burn
slo-burn validate --config slos.yaml
slo-burn status --config slos.yamlOne question for the group
What exclusion policy does your team actually use — and does the rest of the business know what it excludes?
If you've never had that conversation explicitly, that's worth 30 minutes in your next reliability review.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article