AWS Builder Center

Cost Observability: Correlating Cloud Spend with Service-Level Metrics Using Python

A practical guide to closing the gap between "our AWS bill went up" and "here's the exact service and deploy that caused it.

Google cloud certified professional machine learning engineer

Why This Topic

AWS Builder Center content that performs well shares a consistent shape: a real implementation experience, a step by step build, an explained architecture decision, and lessons pulled from an actual pain point, not abstract theory. Cost content on the platform tends to split into two disconnected buckets: FinOps dashboards showing spend by service or tag, and engineering observability dashboards showing latency, error rate, and throughput. The two rarely talk to each other, which means the question every engineering leader eventually asks, "which service or deploy caused this month's spend to jump," usually gets answered by a manual, multi-hour spreadsheet exercise instead of a dashboard.
Cost observability closes that gap by treating cloud spend as another telemetry signal, correlated with the same service-level metrics and trace data DevOps teams already collect. This sits directly in your niche, Observability Engineering and DevOps Automation with Python, applied to a problem nearly every engineering org feels but few have automated a real answer for.



The Problem: Spend and Service Health Live in Separate Systems

Cost and Usage Reports (CUR) tell you what was spent, broken down by service, account, and tag. Application metrics tell you what happened: request volume, latency, error rate, deploy events. When these live in separate systems with no shared dimension, answering a simple question becomes a manual investigation:
  • Did the spend increase come from more traffic, or from a less efficient deploy?
  • Which specific service's cost per request went up, and when did that start?
  • Is a new feature driving proportional cost growth, or is something quietly leaking resources?
Without correlation, cost conversations stay reactive: someone notices the bill at month end, then reconstructs a timeline after the fact. With correlation, the same anomaly detection and dashboarding discipline used for reliability can catch a cost regression the same week it starts, not the month after.



The Architecture: Spend as a Telemetry Signal

[ AWS Cost and Usage Report (CUR) ] [ Application Metrics (CloudWatch/OTel) ]
| |
(daily Python ETL job) (existing metrics pipeline)
| |
v v
[ Normalized cost-per-service, [ requests, latency, error rate,
cost-per-hour time series ] deploy events, per service ]
| |
+-------------------+----------------------+
|
v
[ Join on service name + timestamp ]
|
v
[ Derived metric: cost per request, cost per unit of work ]
|
v
[ Dashboard + anomaly alert: cost-per-request trend by service ]
The core idea is a single derived metric: cost per unit of work, not just raw spend. Raw spend naturally rises with traffic growth, that's expected and healthy. Cost per request rising is the actual signal worth alerting on, because it means something got less efficient, not just busier.



Step 1: Pulling and Normalizing Cost Data

AWS Cost Explorer's API provides programmatic access to cost data broken down by service and time granularity:
python
import boto3
from datetime import datetime, timedelta

ce = boto3.client("ce") # Cost Explorer

def get_daily_cost_by_service(days_back=30):
end = datetime.utcnow().date()
start = end - timedelta(days=days_back)

response = ce.get_cost_and_usage(
TimePeriod={"Start": start.isoformat(), "End": end.isoformat()},
Granularity="DAILY",
Metrics=["UnblendedCost"],
GroupBy=[{"Type": "TAG", "Key": "service-name"}],
)

records = []
for day in response["ResultsByTime"]:
date = day["TimePeriod"]["Start"]
for group in day["Groups"]:
service = group["Keys"][0].split("$")[-1] or "untagged"
cost = float(group["Metrics"]["UnblendedCost"]["Amount"])
records.append({"date": date, "service": service, "cost": cost})

return records
This assumes resources are tagged with a  service-name  tag, which is worth calling out explicitly: cost observability is only as good as your tagging discipline. Untagged resources show up as an unattributed bucket that no amount of clever correlation logic can fix. If tagging enforcement isn't already in place, that's a prerequisite step, not an optional nice to have.



Step 2: Pulling Request Volume for the Same Services

Using the same service names from cost tagging, pull request counts from CloudWatch for the matching time window:
python
import boto3
from datetime import datetime, timedelta

cloudwatch = boto3.client("cloudwatch")

def get_daily_request_count(service_name, days_back=30):
end = datetime.utcnow()
start = end - timedelta(days=days_back)

response = cloudwatch.get_metric_statistics(
Namespace="AWS/ApplicationELB",
MetricName="RequestCount",
Dimensions=[{"Name": "ServiceName", "Value": service_name}],
StartTime=start,
EndTime=end,
Period=86400, # daily
Statistics=["Sum"],
)

return {
dp["Timestamp"].date().isoformat(): dp["Sum"]
for dp in response["Datapoints"]
}



Step 3: Joining Cost and Usage into a Single Derived Metric

This is where the actual insight comes from, joining the two series on service name and date, then computing cost per request:
python
from collections import defaultdict

def compute_cost_per_request(cost_records, request_counts_by_service):
"""cost_records: output of get_daily_cost_by_service()
request_counts_by_service: {service_name: {date: count}}
"""
grouped_cost = defaultdict(dict)
for record in cost_records:
grouped_cost[record["service"]][record["date"]] = record["cost"]

results = []
for service, cost_by_date in grouped_cost.items():
request_by_date = request_counts_by_service.get(service, {})
for date, cost in cost_by_date.items():
requests = request_by_date.get(date)
if requests and requests > 0:
cost_per_request = cost / requests
results.append({
"service": service,
"date": date,
"cost": cost,
"requests": requests,
"cost_per_request": cost_per_request,
})

return results

def detect_cost_efficiency_regression(results, service_name, threshold_pct=25):
"""Flags when cost-per-request has risen significantly vs a 14-day trailing baseline."""
service_series = sorted(
[r for r in results if r["service"] == service_name],
key=lambda r: r["date"],
)
if len(service_series) < 15:
return None

baseline = service_series[-15:-1]
baseline_avg = sum(r["cost_per_request"] for r in baseline) / len(baseline)
latest = service_series[-1]["cost_per_request"]

pct_change = ((latest - baseline_avg) / baseline_avg) * 100
if pct_change > threshold_pct:
return {
"service": service_name,
"baseline_cost_per_request": baseline_avg,
"latest_cost_per_request": latest,
"pct_increase": pct_change,
}
return None
Running  detect_cost_efficiency_regression  daily across all tagged services turns cost monitoring from a monthly retrospective into an ongoing check, the same discipline already applied to latency or error rate.



Real World Scenario: What This Catches

A service's total daily spend grows by 12% over two weeks, which looks unremarkable since traffic to that service also grew. Cost-per-request tells a different story: it rose 34% over the same window, well past the 25% threshold. The regression turns out to trace back to a deploy two weeks earlier that switched a caching layer to a larger, more expensive instance type without a corresponding drop in request latency to justify it, meaning the org was paying more per request for no measurable performance gain. Without the cost-per-request view, this would have been invisible inside a growth trend that looked entirely normal on a raw spend chart.



Common Pitfalls

  • Alerting on raw spend instead of a normalized rate. Raw spend rising with traffic is healthy and expected. Cost per unit of work rising is the actual signal, always normalize before alerting.
  • Weak tagging discipline. Cost observability is entirely dependent on consistent service level tagging. Untagged or inconsistently tagged resources silently break the correlation.
  • Ignoring Cost Explorer API latency. Cost and Usage Report data typically lags by up to 24 hours, this pipeline is well suited to daily batch analysis, not real time alerting.
  • Comparing across service types. A stateless API service and a data pipeline job have fundamentally different cost per unit of work profiles, always compare a service against its own historical baseline, not against other services.
  • Treating this as a replacement for FinOps tooling. This pattern complements existing cost management tools like AWS Cost Anomaly Detection, it adds the engineering-side correlation those tools typically don't provide out of the box.



Closing Thought

Cost observability is less about new infrastructure and more about a shift in framing: treating cloud spend as a telemetry signal that belongs alongside latency, error rate, and throughput, not as a separate monthly report. The technical lift is modest, a daily ETL job and a join on service name and date, but it converts cost management from a reactive, after the fact exercise into the same proactive, alert driven discipline DevOps teams already apply to reliability. For engineering organizations under real cost pressure, that shift alone often surfaces savings a dashboard of raw spend never would have shown.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article