AWS Builder Center

Your CloudWatch Alarm Isn't Silent — You're Just Not Listening Per-Resource

CloudWatch Metrics Insights alarms with GROUP BY already notify you per breaching resource, even while the alarm stays in ALARM state — most teams don't realize this and jump straight to building EventBridge + Lambda pipelines they don't need yet. This post covers what CloudWatch handles natively, exact payload examples (SNS vs. EventBridge), a minimal EventBridge + Lambda extension that also handles recovery events, and a clear decision table for when the extra layer is actually worth adding.

Cloud Architect
If you run CloudWatch Metrics Insights alarms with  GROUP BY  across a fleet—instances, accounts, or anything else—you've probably hit this: the alarm goes to  ALARM , your on‑call gets pinged, and then… silence. Even though three more resources breach the same threshold ten minutes later, nobody hears about it, because the alarm is "already in ALARM state." From the outside, it looks quiet. It isn't.
I ran into this while setting up CPU monitoring across a small multi‑account setup: a handful of dev and staging accounts, roughly 50–80 EC2 instances total, all needing a simple "high CPU" alarm. The docs showed how to write a Metrics Insights query and slap a  GROUP BY  on it, but they didn't say much about what happens when five instances cross the threshold at once, or what happens when they recover. Those gaps are where things get interesting.

The part CloudWatch handles natively

A Metrics Insights alarm with  GROUP BY  doesn't treat your fleet as one blob. Internally, it evaluates each unique dimension combination—each contributor—on its own. So if your query groups by  AccountId, InstanceId , every instance in every account is tracked independently, even though they all roll up into a single alarm resource.
The useful part: alarm actions (SNS, Lambda) fire per contributor, not just on the alarm's overall state transition. That means if Instance A breaches at 2:00 AM and Instance B breaches at 2:05 AM while the alarm is still technically "in ALARM," you still get a second, distinct notification for Instance B—with its own  AlarmContributorId  and attributes telling you exactly which resource and account it was.
If all you need is "tell me which resource breached," this is already enough. No extra infrastructure required.
In my case, that looked like a single CloudWatch alarm with a Metrics Insights query like:
sql
SELECT AVG(CPUUtilization)
FROM SCHEMA("AWS/EC2", InstanceId)
WHERE AccountId IN ('111111111111', '222222222222', '333333333333')
GROUP BY AccountId, InstanceId
Threshold:  AVG(CPUUtilization) > 80  for 3 consecutive 1‑minute periods. Action: send to an SNS topic that pages on‑call.
For a small team where everyone just needs to know "something's hot and here's which instance," that setup is genuinely sufficient. The alarm might be "already in ALARM," but each new breaching instance still triggers its own SNS notification with its own contributor details.
One thing worth knowing before you rely on this in production: the CloudWatch console's alarm history shows the overall alarm's state transitions clearly, but per‑contributor breach/recovery history is not easy to browse visually. You're mostly reading it off the notifications themselves, or querying it back out via the API. If you want a durable, queryable record of which contributor did what and when, that's another good reason to route events through EventBridge—even if you don't act on every one of them.

Where it stops being enough

The native notification is a fixed JSON blob. If you want to:
  • format a human‑readable Slack message instead of raw JSON,
  • route different accounts or resource types to different channels,
  • deduplicate noisy flapping resources,
  • or kick off automated remediation,
…you need something programmable sitting between the alarm and your notification channel. That's where EventBridge comes in. CloudWatch emits a Contributor State Change event to EventBridge for every breach or recovery, independently of the alarm actions—same per‑resource detail, but as a structured event you can route anywhere.
This is where I ended up adding EventBridge and Lambda, not because I needed per‑resource visibility (CloudWatch already gave me that), but because I needed to do more with each breach than dump JSON into one SNS topic.
Here's the difference in what you're actually working with. A native SNS notification for a single contributor breach looks roughly like this:
json
{
"AlarmName": "HighCPU-PerInstance-MultiAccount",
"NewStateValue": "ALARM",
"AlarmContributorId": "AccountId=222222222222,InstanceId=i-0abc123def456",
"AlarmContributorAttributes": {
"AccountId": "222222222222",
"InstanceId": "i-0abc123def456"
},
"Trigger": { "MetricName": "CPUUtilization", "Threshold": 80 }
}
The equivalent EventBridge Contributor State Change event carries the same core facts, but as a structured detail block you can pattern‑match on in a rule:
json
{
"source": "aws.cloudwatch",
"detail-type": "CloudWatch Alarm Contributor State Change",
"account": "222222222222",
"detail": {
"alarmName": "HighCPU-PerInstance-MultiAccount",
"state": { "value": "ALARM", "timestamp": "2026-08-04T02:05:11Z" },
"alarmContributor": {
"id": "AccountId=222222222222,InstanceId=i-0abc123def456",
"attributes": {
"AccountId": "222222222222",
"InstanceId": "i-0abc123def456"
}
}
}
}
Functionally equivalent data, but the EventBridge version is what lets you filter, route to multiple targets, and layer enrichment on top without touching the alarm itself.

The shape of the pipeline (with a minimal example)


The pattern is simple once you see it:
  1. CloudWatch alarm (Metrics Insights,  GROUP BY ) evaluates every contributor each period.
  1. EventBridge rule matches on  "detail-type": "CloudWatch Alarm Contributor State Change" , filtered to your alarm name.
  1. Lambda receives one invocation per breaching resource—never a batch—pulls out  alarmContributor.attributes , enriches it (tags, account alias, whatever context you want), and formats a real message.
  1. SNS / Slack / wherever gets the enriched, per‑resource alert.
One detail worth internalizing: if five resources breach in the same evaluation window, you get five separate events, each with exactly one contributor. Your Lambda doesn't need to loop over anything—it's invoked five times.
Here's the EventBridge rule pattern I used — note it matches both  ALARM  and  OK , not just breaches:
json
{
"detail-type": ["CloudWatch Alarm Contributor State Change"],
"detail": {
"alarmName": ["HighCPU-PerInstance-MultiAccount"],
"state": {
"value": ["ALARM", "OK"]
}
}
}
That last point matters more than it looks. It's easy to build the whole pipeline around the  ALARM  path and forget that contributors also emit an  OK  event when they recover — and if your Slack channel only ever says "🚨 breach," nobody knows when the instance is actually fine again. The Lambda below branches on state so both directions get a message:
python
import json
import os
import requests

SLACK_WEBHOOK = os.environ["SLACK_WEBHOOK"]

def handler(event, context):
# One event = one contributor breach or recovery
detail = event["detail"]
state = detail["state"]["value"]
contributor = detail["alarmContributor"]
attrs = contributor.get("attributes", {})

account_id = attrs.get("AccountId", "unknown")
instance_id = attrs.get("InstanceId", "unknown")
alarm_name = detail["alarmName"]

icon = "🚨" if state == "ALARM" else "✅"
message = {
"text": (
f"{icon} {alarm_name}\n"
f"Account: {account_id}\n"
f"Instance: {instance_id}\n"
f"State: {state}"
)
}

requests.post(SLACK_WEBHOOK, json=message)
One clear Slack message per contributor transition, in either direction, with account and instance IDs front and center.
A couple of production notes before anyone copies this verbatim: give the Lambda a dead‑letter queue so a Slack outage doesn't silently drop a breach notification, and treat each invocation as safe to retry. That's already true here—posting the same Slack message twice is harmless—but it changes if you swap Slack for something like auto‑remediation, where you'd want an idempotency key such as the contributor ID + timestamp.
If you're spanning multiple AWS accounts, the other piece worth planning up front is IAM: the alarm, EventBridge rule, and Lambda typically live in one monitoring account. If your Lambda needs to enrich the event with anything from the source account (instance tags, for example), it needs a cross‑account role to assume into each workload account—scoped to read‑only, not the broad access it's tempting to grant "just in case."

Where this actually pays off

For a single SNS topic with a small fleet, don't bother—the native contributor‑level alarm action is genuinely sufficient, and adding EventBridge and Lambda for no reason is just more to maintain. In my own setup, I only added the EventBridge + Lambda layer once we hit two concrete needs:
  • Different teams wanted different Slack channels for the same alarm (dev vs. staging vs. prod).
  • Raw JSON in email was becoming unreadable as the number of contributors grew.
It earns its keep once you have:
  • multiple teams that need different routing for the same alarm,
  • a cardinality high enough that raw JSON notifications become unreadable,
  • or a remediation step you want triggered automatically (a Systems Manager runbook, for example) rather than just an alert.
Need
Native alarm actions enough?
Add EventBridge + Lambda?
Just know which resource breached
✅
❌
Different Slack channels per team/account
❌
✅
Auto‑remediation (SSM, Lambda fix)
❌
✅
Dedup / flapping control
❌
✅
Notify on recovery, not just breach
⚠️ (basic)
✅ (customizable)
Worth flagging on cost: EventBridge management events are free, but at real scale the Lambda/SNS cost tracks contributor count, not alarm count. A thousand contributors breaching daily is roughly 30k Lambda invocations and 30k SNS publishes a month—trivial in dollars, but worth knowing before you assume "one alarm" means "one bill line."

Takeaway

The instinct is to reach for EventBridge and Lambda the moment you need per‑resource visibility. Don't CloudWatch already gives you that for free at the alarm‑action level. Reach for the extra plumbing only when you need to do something smarter with each breach (or recovery) than send it to one topic.
In practice, that meant: start with native SNS notifications, watch how the team actually uses them, then add EventBridge + Lambda only when the workflow demands it. That kept the initial design simple and made the extra complexity easy to justify when the time came.
About the author: Abdul Hayee is a Cloud/DevSecOps Engineer at CNS Engineering, focusing on cloud operations and DevSecOps on AWS.
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article