What I Learned Building & Breaking a CI/CD Pipeline on AWS
An architectural review of an AWS CI/CD pipeline. Discover how to debug and fix a deployment rollback failure caused by IAM restrictions.
I'm still learning cloud and DevOps, so this isn't an expert guide. It's what I built, what failed, and what I want to improve next.
What I built
I wanted a pipeline where pushing code to GitHub deploys it to a live server automatically:
- GitHub holds the Java web app source.
- CodeArtifact serves the Maven dependencies.
- CodeBuild compiles and packages the app, driven by a
buildspec.yml. - CodeDeploy installs it on an EC2 instance running Tomcat, created from a CloudFormation template.
- CodePipeline connects the stages and starts a run on every push.
The first green run felt great. Then I wondered what happens when something goes wrong, so I tested rollback.
Where it broke
When I tested rollback, the deployment failed. The CodeDeploy deployment events showed a permissions problem, and it came from the CodeDeploy service role, not my EC2 instance profile or CodeBuild. The role didn't have permission to reach the earlier revision (the saved bundle of app files) it needed to roll back to.
That surprised me, because forward deployments worked fine. Deploying and rolling back don't exercise the same paths, so a pipeline can look healthy while its recovery path is broken. Test rollback before you need it.
I fixed it by troubleshooting with Amazon Q, which gave me a fine-grained IAM policy scoped to what the role actually needed. My takeaway is that AI tools make debugging faster, but you should still read every permission before you attach it.
One honest note: I tore down the environment to stop charges before saving the logs, so I can't quote the exact error. Next time I'll save the error text and the CloudTrail record first.
If I hit this again, I'd check which role was acting (CodeDeploy uses a service role for itself and an instance profile for the servers), then read the deployment events, the agent log at
/var/log/aws/codedeploy-agent/codedeploy-agent.log, and CloudTrail.What rollback actually did
In my test, rolling back reverted only the Deploy stage. The server returned to the previous good version, because CodeDeploy redeploys the last successful revision, while the Source and Build stages still showed the latest commit.
That told me rollback buys time, but it isn't a fix. Unless the bad commit is reverted in Git, the next push would deploy the problem again.
What I want to improve next
While reviewing my own setup and reading the docs, I found things I got wrong or want to learn more about. I originally called this pipeline "zero-downtime", and it isn't: an in-place deployment stops Tomcat during the update.
| What I noticed | What I plan to try |
|---|---|
| In-place deploy on one instance causes downtime | Learn rolling or blue/green deployment behind a load balancer |
| Resource: "*" on my CodeArtifact permission is broader than needed | Scope it to the specific domain and repository |
| The build runs mvn clean install and then mvn package | Check whether the second step is redundant |
| I'm on Java 8 | Move to a newer Java runtime and test |
| SSH is open on the instance | Try SSM Session Manager instead |
| The pipeline may start on pull requests | Restrict triggers to pushes on the deploy branch |
| No tests or scans before deploy | Add tests, a dependency scan and an approval step |
Two notes from the docs. CodePipeline V2 can also start a pipeline when pull requests match the branch, file path and event filters you set, so if a pipeline deploys to production, those filters matter. And CodeBuild's Amazon Linux 2023 and Ubuntu standard images support Corretto 17 and 21, so moving off Java 8 looks straightforward.
If you've built this differently, I'd love to hear what you'd change.
What's next
I'm early in my DevOps journey, so I'd value advice from people who've done this for real. If you were me, which would you learn next: pipeline security, blue/green deployments, EKS hardening, or infrastructure as code? Vote here, or tell me in the comments if you'd start somewhere else: https://builder.aws.com/poll/3KB1ndal4R0mtfHGFegC4M8cYBp_po
Thank you for reading!
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article