
My Pipeline Broke at 9 AM: Here's How I Fixed It in 5 Minutes (CodePipeline + CodeCommit + CodeBuild + EC2)
A pipeline that stops triggering fails silently — no red X, no alert, just nothing happening. Here's the real diagnostic path I used to find a deleted EventBridge rule, the actual console/CLI fix, and why "no failures" isn't the same signal as "everything's fine."
No dramatic outage, no root-cause postmortem deck. Just a normal Tuesday: a teammate pushes a one-line config fix to
main , and the pipeline that's deployed clean for three weeks straight... does nothing. No new execution. No error banner. Just silence. Here's exactly what I saw, what I checked, and the actual fix — in the order I did it, including the dead end.(Quick note before we start: this walkthrough uses CodeCommit because that's what the existing repo was on. If you're setting up a new pipeline today, AWS stopped onboarding new CodeCommit customers back in mid-2024 — GitHub or GitLab via CodeStar Connections is the right call for anything new. Everything below still applies if you're maintaining an existing CodeCommit-backed pipeline, which plenty of teams still are.)
Minute 0: the symptom
$ git log --oneline -1 origin/maina3f9c12 fix: correct env var name in worker config
Commit's on
main . I open the CodePipeline console.Latest execution: 3 hours ago. Status: Succeeded. Nothing new since. No failed run, no stuck stage — the pipeline just never started.
That's the tell. A failed stage screams at you in red. A pipeline that silently never triggers is quieter and, honestly, worse — because for three hours the team assumed the fix was live.
Minute 1: rule out the obvious
First check — is this actually a trigger problem, or did the push just not land?
$ aws codecommit get-branch \ --repository-name worker-service \ --branch-name main{ "branch": { "branchName": "main", "commitId": "a3f9c12..." }}
Commit's there, CodeCommit sees it. So the repo is fine. The break is somewhere between "commit lands in CodeCommit" and "CodePipeline notices."
Minute 2: check the actual trigger mechanism
CodePipeline doesn't poll CodeCommit by magic — it's wired through an EventBridge (CloudWatch Events) rule that watches for
CodeCommit Repository State Change events and calls StartPipelineExecution . If that rule is gone, disabled, or mis-scoped, your pipeline sits there forever looking healthy while doing nothing.$ aws events list-rules --name-prefix codepipeline-worker-service{ "Rules": []}
Empty. There's supposed to be a rule there. It's gone.
Root cause found: someone had run a "clean up unused CloudWatch resources" script the day before as part of a cost-hygiene pass. The rule wasn't tagged, its name didn't obviously scream "CodePipeline depends on this," and it got swept up as apparent dead weight. Classic — the automation that keeps your pipeline alive is invisible right up until it's deleted.
Minute 3–4: the actual fix
Two options here. The slow-but-clean one: go into the CodePipeline console, edit the pipeline's source stage, and let AWS regenerate the EventBridge rule for you automatically (it does this on save if the rule is missing). That's genuinely the fastest fix in the console — no CLI needed:
- CodePipeline → your pipeline → Edit
- Edit the Source stage → don't change anything, just click through the same CodeCommit repo/branch selection again
- Save → Release change
Saving the source action re-provisions the underlying EventBridge rule. If you want to confirm it's back without waiting:
$ aws events list-rules --name-prefix codepipeline-worker-service{ "Rules": [ { "Name": "codepipeline-worker-service-main", "State": "ENABLED", "EventPattern": "{\"source\":[\"aws.codecommit\"],\"detail-type\":[\"CodeCommit Repository State Change\"],...}" } ]}
Rule's back, enabled, scoped to the right repo and branch.
Minute 5: don't wait for the next commit: trigger it now
The fix restores triggering going forward. It doesn't retroactively fire for the commit that already landed. Don't wait around for someone to push a dummy commit, just start the execution directly:
$ aws codepipeline start-pipeline-execution --name worker-service-pipeline{ "pipelineExecutionId": "8f2e1a4b-..."}
Watch it go:
$ aws codepipeline get-pipeline-state --name worker-service-pipeline \ --query 'stageStates[*].[stageName,latestExecution.status]' --output table
Source → Build → Deploy, green all the way. Fix was live inside 5 minutes of me actually looking at it — the annoying part was the 3 hours nobody noticed it wasn't running at all.
The actual takeaway: silence is the dangerous failure mode
A CodeBuild failure is self-announcing: red X, error in the logs, someone gets paged or at least sees it in Slack if you've wired notifications right. A pipeline that stops triggering is invisible by default. Two things fix that permanently:
- Tag pipeline-managed EventBridge rules (or better: don't let cleanup scripts touch anything with
codepipeline-in the name without a human check) so this doesn't get deleted as "unused" again.
- Add a CloudWatch alarm on pipeline execution cadence: if you deploy multiple times a day normally, an alarm like "zero
SUCCEEDEDorFAILEDpipeline executions in the last 4 hours" catches exactly this failure mode, because a pipeline that isn't running doesn't emit a failure event for you to alert on. You have to alert on the absence of activity, which is the counterintuitive part people miss.
The fix itself was trivial. The lesson that your pipeline's trigger mechanism is a resource someone else's script can silently delete, and that "no failures" and "everything's fine" are not the same signal was the actual 5 minutes well spent.
Anyone else been burned by a "cleanup unused resources" script eating a CodePipeline-managed EventBridge rule? Curious how others tag around this.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article