AWS Builder Center

Builder Center

CI/CD Strategy Patterns and Common Pitfalls

A comprehensive guide to CI/CD strategy covering deployment patterns (rolling, blue/green, canary, feature flags, shadow, immutable), testing types, rollback safety, supply chain security, and common mistakes to avoid — informed by AWS Well-Architected guidance, the Builders' Library, and DORA research. Includes a five-level maturity model with checklists, a metrics framework, and practical recommendations for engineering teams.

Handa, Rahul

Table of Contents

  1. Core Principles and DORA Metrics
  1. CI/CD Pipeline Architecture
  1. Continuous Integration Patterns
  1. Deployment Strategy Patterns
  1. Testing Strategy
  1. Rollback and Safety Mechanisms
  1. Pipeline Security and Supply Chain
  1. Observability in the Pipeline
  1. Common Mistakes
  1. CI/CD Metrics Framework
  1. Maturity Model
  1. Key Recommendations
  1. References


1. Core Principles and DORA Metrics

Continuous Integration (CI) and Continuous Delivery (CD) are foundational practices for modern software engineering. CI ensures that code changes are frequently merged and validated through automated builds and tests. CD extends this by automating the deployment of validated changes through environments up to and including production.
The goal: deliver changes to production frequently, reliably, and with minimal risk.

The Four Key Metrics (DORA)

The DevOps Research and Assessment (DORA) program identified four metrics that reliably predict software delivery performance and organizational outcomes:
Metric
Definition
Elite
High
Medium
Low
Deployment Frequency
How often code is deployed to production
On-demand (multiple per day)
Weekly to Monthly
Monthly to Biannually
Less than once per 6 months
Lead Time for Changes
Time from code commit to running in production
Under 1 hour
1 day to 1 week
1 week to 1 month
Over 1 month
Change Failure Rate
Percentage of deployments causing production failure
0 to 15 percent
16 to 30 percent
16 to 30 percent
Over 30 percent
Mean Time to Recovery
Time to restore service after a failure
Under 1 hour
Under 1 day
1 day to 1 week
Over 1 month
These metrics are interconnected. Teams that deploy frequently tend to have lower change failure rates because each deployment contains fewer changes. Teams with fast recovery times are more willing to deploy frequently because the cost of failure is low.

Guiding Principles

  • Small, frequent changes — Reduce batch size to reduce risk. Each deployment should contain the minimum viable change.
  • Automate everything repeatable — If a human does it more than twice, automate it. Manual steps introduce variability and error.
  • Fast feedback loops — Developers should know within minutes if their change broke something. Slow feedback encourages batching.
  • Immutable artifacts — Build once, deploy the same artifact everywhere. Never rebuild for different environments.
  • Everything as code — Pipelines, infrastructure, configuration, and policies should be version-controlled and peer-reviewed.
  • Shift left — Move quality gates (testing, security, compliance) earlier in the pipeline where fixes are cheaper.


2. CI/CD Pipeline Architecture

Standard Pipeline Stages

A well-structured pipeline progresses through these stages, with each acting as a quality gate:
Plain text
Source > Build > Unit Test > Static Review > Security Scanning > Integration Testing > Staging Deploy > Acceptance Test > Production Deploy > Post-Deploy Validation

Pipeline Design Principles

  1. Every commit triggers the pipeline — No manual start buttons. The pipeline runs automatically on every push to the repository.
  1. Fail fast — Order stages by speed and likelihood of failure. Unit tests (seconds) run before integration tests (minutes) which run before end-to-end tests (minutes to hours).
  1. Artifacts are built once — The CI stage produces a versioned, immutable artifact (container image, JAR, ZIP). That exact artifact is promoted through staging to production. Never rebuild per environment.
  1. Configuration is injected, not baked — Environment-specific values (database URLs, feature flags, API keys) are injected at deploy time via environment variables, parameter stores, or config services. Artifacts contain no environment-specific configuration.
  1. Stages are isolated — Each stage runs in its own environment. A failure in integration testing does not affect the staging environment.
  1. Pipeline state is visible — Every team member can see the current state of the pipeline, what is deployed where, and what is queued.
  1. Pipeline is defined as code — The pipeline definition lives in the same repository as the application code, versioned and reviewed alongside it.

Branching Strategy Alignment

Strategy
Pipeline Trigger
Best For
Trunk-based
Every commit to main
High-performing teams, continuous deployment
GitHub Flow
PR merge to main
Teams needing code review gates
GitFlow
Merge to develop or release branches
Scheduled releases, multiple supported versions
Release branches
Tag or branch creation
Mobile apps, packaged software
Recommendation: Trunk-based development with short-lived feature branches (under 1 day) is the approach most strongly correlated with elite DORA performance.


3. Continuous Integration Patterns

3.1 Trunk-Based Development

All developers commit to a single main branch (trunk) frequently, at least daily. Short-lived feature branches (under 1 day) are acceptable for code review purposes.
How it works:
  • Developers pull from trunk, make small changes, and push back to trunk
  • Feature branches exist only for the duration of a code review (hours, not days)
  • Incomplete features are merged behind feature flags
  • The trunk is always in a deployable state
Benefits:
  • Eliminates merge conflicts and painful integration cycles
  • Provides continuous feedback on integration issues
  • Enables true continuous deployment
Prerequisites:
  • Strong automated test suite to keep trunk green
  • Feature flag infrastructure for incomplete work
  • Team discipline around small, incremental commits
  • Fast CI pipeline (under 10 minutes) to avoid queuing

3.2 Build Once, Deploy Many

Build the artifact exactly once during CI, then promote that identical artifact through test, staging, and production.
How it works:
  • CI produces a versioned artifact (for example, Docker image myapp:v1.2.3-abc123)
  • The artifact is stored in a registry (ECR, Artifactory, Nexus)
  • Each environment deployment pulls the same artifact from the registry
  • Environment-specific configuration is injected at deploy time
Why it matters:
  • Eliminates environment-specific build differences (different compiler versions, dependency resolution)
  • Enables artifact provenance tracking and signing

3.3 Shift-Left Testing

Move testing earlier in the pipeline. Run fast, cheap tests first; slow, expensive tests later.
The Test Pyramid:
  • Base (many, fast, cheap): Unit Tests — Verify individual functions and modules
  • Middle (moderate): Integration Tests — Verify component interactions
  • Top (few, slow, expensive): End-to-End and UI Tests — Validate critical user journeys
Guidelines:
  • Aim for 70 to 80 percent unit tests, 15 to 20 percent integration tests, 5 to 10 percent end-to-end tests
  • Unit tests should complete in under 2 minutes
  • Integration tests should complete in under 10 minutes
  • End-to-end tests should complete in under 30 minutes
  • If a bug is found in end-to-end tests, write a unit test to catch it earlier next time

3.4 Pipeline as Code

Define your entire pipeline in version-controlled files that live alongside application code.
Examples:
  • GitHub Actions: .github/workflows/ci.yml
  • GitLab CI: .gitlab-ci.yml
  • Jenkins: Jenkinsfile
  • AWS CodePipeline: buildspec.yml plus pipeline definition in CloudFormation or CDK
  • CircleCI: .circleci/config.yml
Benefits:
  • Pipeline changes go through the same code review process as application code
  • Pipeline history is tracked in version control
  • Pipelines are reproducible — any team can recreate the exact same pipeline
  • Enables pipeline templating and sharing across teams

3.5 Dependency Management

  • Pin dependency versions explicitly (lock files: package-lock.json, Pipfile.lock, go.sum)
  • Scan dependencies for known vulnerabilities (Software Composition Scanning)
  • Use private artifact repositories to cache and control approved dependencies
  • Generate Software Bill of Materials (SBOM) for every build
  • Regularly update dependencies — stale dependencies accumulate security debt


4. Deployment Strategy Patterns

4.1 Rolling Deployment

Incrementally replaces instances of the old version with the new version, one batch at a time.
How it works:
  1. Divide the fleet into batches (for example, 25 percent at a time)
  1. Deploy the new version to the first batch
  1. Run health checks on the updated batch
  1. If healthy, proceed to the next batch; if unhealthy, halt and roll back
  1. Repeat until all instances are updated
When to use:
  • Stateless services with robust health checks
  • When zero-downtime is required without extra infrastructure cost
  • Services behind a load balancer that can drain connections gracefully
Trade-offs:
  • Both versions run simultaneously during rollout — APIs and data formats need to be backward compatible
  • Rolling back requires redeploying the old version through the same rolling process (slower than blue/green)
  • Debugging is harder when two versions serve traffic simultaneously

4.2 Blue/Green Deployment

Maintains two identical production environments. Traffic is switched from the current (blue) to the new (green) after validation.
How it works:
  1. The current production environment is blue
  1. Deploy the new version to an identical green environment
  1. Run validation tests against the green environment (smoke tests, synthetic traffic)
  1. Switch the router or load balancer to direct traffic from blue to green
  1. Monitor the green environment; keep blue running as a rollback target
  1. After a bake period (for example, 30 to 60 minutes), decommission the blue environment
When to use:
  • When instant rollback capability is critical (switch traffic back to blue in seconds)
  • Regulated environments requiring pre-production validation of the exact production setup
  • When you can afford running two environments temporarily
Trade-offs:
  • Higher infrastructure cost during the transition window (2x compute)
  • Database schema changes require careful coordination — both environments share the same database
  • DNS-based switching has TTL propagation delays; load-balancer-based switching is instant
  • Stateful applications (WebSocket connections, in-memory sessions) may lose state during the switch
Database migration strategy:
  • Use backward-compatible schema changes (additive only — new columns, new tables)
  • Apply schema changes before deploying the new application version
  • Never drop columns or rename tables in the same release that changes application code

4.3 Canary Deployment

Routes a small percentage of traffic to the new version first, then gradually increases if metrics are healthy.
How it works:
  1. Deploy the new version alongside the current version
  1. Route a small percentage of traffic (for example, 5 percent) to the new version (the canary)
  1. Monitor key metrics: error rate, latency, CPU, memory, business KPIs
  1. If metrics are within thresholds, increase traffic (for example, 5 to 25 to 50 to 100 percent)
  1. If metrics breach thresholds, automatically route all traffic back to the stable version
When to use:
  • High-traffic services where you want real production validation before full rollout
  • When you have strong observability and can detect degradation in a small traffic slice
  • Services where synthetic tests cannot fully replicate production behavior
Trade-offs:
  • Requires robust observability to detect issues in a small traffic sample
  • More complex routing configuration (weighted target groups, service mesh)
  • Low-traffic services may not generate enough canary traffic for statistical significance
  • Both versions need to be backward compatible (same as rolling)
Canary evaluation automation:
  • Define baseline metrics from the stable version
  • Compare canary metrics against the baseline using statistical methods
  • Auto-promote if canary metrics are within acceptable deviation
  • Auto-rollback if canary metrics breach thresholds
  • Tools: AWS CodeDeploy (built-in), Flagger (Kubernetes), Argo Rollouts
Example traffic shifting schedule:
Time
Canary Traffic
Action
T+0 min
5 percent
Deploy canary, begin monitoring
T+10 min
25 percent
Promote if error rate under 1 percent, p99 latency under 200ms
T+20 min
50 percent
Promote if metrics hold
T+30 min
100 percent
Full promotion, decommission old version

4.4 Feature Flags (Progressive Delivery)

Deploys code to production with new functionality hidden behind toggles. Features are enabled independently of deployment.
How it works:
  1. Wrap new functionality in a conditional check against a feature flag service
  1. Deploy the code to production with the flag disabled — the new code path is dormant
  1. Enable the flag for a subset of users (internal team, beta users, 1 percent of traffic)
  1. Gradually increase the audience as confidence grows
  1. Once fully rolled out, remove the flag and the old code path (flag cleanup)
When to use:
  • Decoupling deployment from release — deploy anytime, release when ready
  • A/B testing and experimentation
  • Gradual rollout by user segment, geography, or account tier
  • Long-running features that span multiple deployments
Trade-offs:
  • Adds code complexity — every flag is a branch in your code
  • Flags need to be actively managed and retired; stale flags become technical debt
  • Testing combinatorial growth — N flags create 2 to the power of N possible states
  • Requires a feature flag management service (LaunchDarkly, AWS AppConfig, Unleash, Flagsmith)
Flag lifecycle management:
  1. Create — Document the flag purpose, owner, and expected lifetime
  1. Enable — Roll out progressively with monitoring
  1. Stabilize — Flag is fully enabled for all users
  1. Remove — Delete the flag, remove the conditional code, delete the old code path
  1. Target SLA: Flags should be removed within 30 days of full rollout

4.5 Shadow (Dark Launch) Deployment

The new version receives a copy of production traffic, but its responses are discarded. Used for validation without user impact.
How it works:
  1. Deploy the new version alongside the current version
  1. Mirror (fork) incoming requests to both versions
  1. Only the current version responses are returned to users
  1. Compare the new version responses, latency, and resource usage against the current version
  1. Promote the new version once confidence is established
When to use:
  • ML model deployments — compare prediction quality without affecting users
  • Major refactors — verify the new implementation produces identical results
  • Performance-sensitive changes — validate latency and throughput at production scale
Trade-offs:
  • Doubles compute cost for the shadowed service
  • Not suitable for write operations with side effects (creating orders, sending emails)
  • Request mirroring adds complexity to the routing layer
  • Response comparison requires tooling to diff and review results at scale

4.6 Immutable Deployment

Deploys to entirely new infrastructure rather than updating existing instances in place.
How it works:
  1. Provision new instances or containers with the new version
  1. Run health checks on the new infrastructure
  1. Shift traffic to the new infrastructure
  1. Terminate the old infrastructure
When to use:
  • Containerized workloads (this is the default model for containers)
  • When configuration drift is a concern
  • Infrastructure-as-code environments where infrastructure is disposable

Strategy Selection Guide

Factor
Rolling
Blue/Green
Canary
Feature Flag
Shadow
Immutable
Rollback speed
Medium
Fast (seconds)
Fast
Instant
N/A
Fast
Infrastructure cost
Low
High (2x)
Medium
Low
High (2x)
Medium
Implementation complexity
Low
Medium
High
Medium
High
Low
Real traffic validation
Yes
Post-switch
Yes (partial)
Yes (partial)
Yes (no impact)
Post-switch
Database migration friendly
With care
With care
With care
Yes
No
With care
Requires backward compatibility
Yes
No (post-switch)
Yes
Yes
No
No (post-switch)
Best for
Stateless services
Critical services
High-traffic APIs
Feature rollouts
ML models
Containers


5. Testing Strategy

A comprehensive testing strategy is the foundation that makes CI/CD possible. Without automated tests, frequent deployments are high-risk rather than agile.

Test Types by Pipeline Stage

Build Stage (seconds to minutes)

Test Type
Purpose
Tools
Run Time Target
Unit Tests
Verify individual functions and modules in isolation. Simulate all external dependencies.
JUnit, Jest, pytest, Go test
Under 2 minutes
Static Code Review
Enforce code quality, style, complexity rules
SonarQube, ESLint, Pylint, Checkstyle
Under 2 minutes
SAST (Static Security Testing)
Scan source code for security vulnerabilities (SQL injection, XSS)
Amazon CodeGuru, SonarQube, Checkmarx, Semgrep
Under 5 minutes
Secrets Detection
Scan for accidentally committed credentials, API keys, tokens
git-secrets, TruffleHog, Gitleaks
Under 1 minute
Dependency Scanning (SCA)
Scan dependencies for known CVEs
Snyk, Dependabot, OWASP Dependency-Check, Amazon Inspector
Under 2 minutes
SBOM Generation
Produce a Software Bill of Materials for the build
Syft, CycloneDX, Amazon Inspector
Under 1 minute

Staging Stage (minutes)

Test Type
Purpose
Tools
Run Time Target
Integration Tests
Verify component interactions against real dependencies
Cucumber, Testcontainers, vRest NG, AWS CDK integ-tests
Under 10 minutes
Acceptance Tests
Validate user requirements and business flows
Cypress, Selenium, Playwright
Under 15 minutes
Contract Tests
Verify API contracts between services have not broken
Pact, Spring Cloud Contract
Under 5 minutes
Performance Tests
Validate latency, throughput, resource usage under load
Apache JMeter, k6, Locust, Gatling
Under 30 minutes
DAST (Dynamic Security Testing)
Scan running application for security vulnerabilities
OWASP ZAP, HCL AppScan, PortSwigger
Under 15 minutes
Compliance Tests
Verify regulatory and policy requirements
Open Policy Agent (OPA), AWS Config Rules, cfn-nag
Under 5 minutes

Production Stage (continuous)

Test Type
Purpose
Tools
Frequency
Smoke Tests
Verify critical paths work immediately after deployment
Custom scripts, Cypress
Every deployment
Synthetic Tests
Generate continuous traffic to verify system health
Amazon CloudWatch Synthetics, Datadog Synthetic
Continuous (every 1 to 5 min)
Canary Evaluation
Compare new version metrics against baseline
AWS CodeDeploy, Flagger, Argo Rollouts
During canary window
Resilience Tests
Inject failures to identify risk areas
AWS Fault Injection Service, Gremlin, Litmus
Scheduled (weekly or monthly)

Minimum Viable Test Suite

At minimum, every CI/CD pipeline should run:
  1. Unit tests and SAST on the code base (build stage)
  1. Integration tests and acceptance tests on a test environment (staging stage)
  1. Smoke tests after production deployment


6. Rollback and Safety Mechanisms

Every deployment needs a tested, automated rollback path. The guiding principle is: avoid one-way doors — stay away from choices that are hard to reverse.

Rollback Strategies by Deployment Type

Deployment Type
Rollback Mechanism
Speed
Complexity
Rolling
Redeploy previous version through rolling update
Minutes
Low
Blue/Green
Switch traffic back to the blue environment
Seconds
Low
Canary
Route 100 percent traffic back to stable version
Seconds
Low
Feature Flag
Disable the flag
Seconds
Low
Immutable
Point load balancer back to old instances
Seconds
Low
Database migration
Requires forward-fix or backward-compatible schema
Minutes to Hours
High

Automatic Rollback Triggers

Configure automated rollback based on alarms linked to key metrics:
  • Error rates — 4xx and 5xx error rates exceed baseline
  • Latency — p50, p95, or p99 latency exceeds threshold
  • CPU and Memory — Resource utilization spikes beyond normal range
  • Business KPIs — Conversion rate drops, order volume decreases
  • Health checks — Instances failing health checks
  • Log errors — Error log volume exceeds threshold

The Two-Phase Deployment Technique

For changes that alter data formats, protocols, or serialization (the most common cause of rollback failures), use a two-phase approach:
Phase 1 — Prepare: Deploy code that can read the new format but continues to write the old format. This is safe to roll back because nothing has changed operationally.
Phase 2 — Activate: Deploy code that writes the new format. All instances can already read it (from Phase 1). Rolling back to Phase 1 is safe because Phase 1 can read both formats.
Example: Migrating from XML to JSON storage:
  • Phase 1: Deploy code that reads both XML and JSON, but writes XML only
  • Phase 2: Deploy code that writes JSON (all instances can already read JSON from Phase 1)

Rollback Safety Checklist

Before every deployment, verify:
  • The previous version artifact is available in the artifact registry
  • Database schema changes are backward compatible (additive only)
  • API changes are backward compatible (new fields optional, no removed fields)
  • Configuration changes are backward compatible
  • Rollback has been tested in a pre-production environment
  • Automated rollback alarms are configured and tested
  • The team knows the rollback procedure and has practiced it

Bake Time

After a deployment completes, maintain a monitoring window (bake time) before declaring success:
Risk Level
Bake Time
When to Use
Low
15 to 30 minutes
Minor config changes, copy updates
Medium
1 to 2 hours
Feature additions, dependency updates
High
4 to 24 hours
Infrastructure changes, data format changes, security changes
During bake time: keep the previous version infrastructure available for instant rollback, monitor all alarm thresholds, and avoid starting another deployment.


7. Pipeline Security and Supply Chain

Integrating security into the CI/CD pipeline (DevSecOps) ensures vulnerabilities are caught early when they are cheapest to fix, rather than discovered in production.

Security Scanning Integration Points

Plain text
Source: Secrets Detection, Commit Signing
Build: SAST, Dependency Scanning, SBOM Generation, Container Image Scanning
Deploy: IaC Scanning, Artifact Signature Verification, DAST
Runtime: Runtime Protection, Continuous Monitoring

Software Supply Chain Security

1. Trusted Artifact Repositories
Deploy exclusively from trusted, controlled repositories. Restrict the pipeline to using only approved registries enforced through allow lists, IP restrictions, or authentication controls.
2. Cryptographic Signing
  • Sign code artifacts after each build to ensure authenticity and integrity
  • Enforce signature verification before deploying signed artifacts
  • Use commit signing to enhance traceability
  • Tools: AWS Signer, Sigstore/Cosign, Notation
3. Software Bill of Materials (SBOM)
Generate an SBOM for every build to maintain a complete inventory of components and dependencies. This enables rapid response when new vulnerabilities are disclosed (for example, which of our services use a specific vulnerable library).
4. Dependency Management
  • Use private artifact repositories (AWS CodeArtifact, Artifactory, Nexus) to cache and control approved packages
  • Pin dependency versions with lock files
  • Automate dependency updates with tools like Dependabot or Renovate
  • Block builds that introduce dependencies with critical CVEs
5. IaC Security Scanning
Scan infrastructure-as-code templates before deployment:
  • CloudFormation and Terraform: cfn-nag, Checkov, tfsec, Bridgecrew
  • Kubernetes manifests: Kyverno, OPA/Gatekeeper, Datree
  • Check for: public S3 buckets, overly permissive IAM policies, unencrypted storage, open security groups

Least Privilege for Pipelines

  • Pipeline service accounts should have the minimum permissions needed for each stage
  • Use separate roles for build, test, and deploy stages
  • Never store long-lived credentials in pipeline configuration — use IAM roles, OIDC federation, or short-lived tokens
  • Audit pipeline permissions regularly


8. Observability in the Pipeline

Deploying without observability means deploying without visibility. Every deployment should be monitored automatically, and the pipeline itself should be observable.

Deployment Observability

Pre-deployment:
  • Record the current baseline metrics (error rate, latency, throughput) before the deployment starts
  • Verify all monitoring and alerting is functional
  • Confirm rollback alarms are armed
During deployment:
  • Track deployment progress (percentage of instances updated, health check status)
  • Compare real-time metrics against the pre-deployment baseline
  • Monitor canary metrics if using canary deployment
Post-deployment (bake period):
  • Run automated smoke tests and synthetic monitors
  • Compare post-deployment metrics against baseline
  • Watch for delayed-onset issues (memory leaks, connection pool exhaustion, cache warming)
  • Monitor business KPIs (conversion rate, order volume, API call volume)

Key Metrics to Monitor Post-Deployment

Category
Metrics
Alarm Threshold Example
Availability
Error rate (4xx, 5xx), health check failures
Over 1 percent 5xx rate for 5 minutes
Latency
p50, p95, p99 response time
p99 over 2x baseline for 5 minutes
Throughput
Requests per second, transactions per minute
Under 80 percent of baseline for 10 minutes
Resources
CPU, memory, disk, connection pool utilization
Over 85 percent for 10 minutes
Business
Conversion rate, revenue per minute, order volume
Over 10 percent drop from baseline
Downstream
Dependency error rates, dependency latency
Any dependency over 5 percent error rate

Pipeline Observability

Monitor the pipeline itself as a production system:
  • Build success rate — percentage of builds that complete all stages
  • Pipeline duration — Time from commit to production (track p50 and p95)
  • Queue time — Time builds spend waiting for runners or agents
  • Flaky test rate — percentage of tests that intermittently fail without code changes
  • Stage failure distribution — Which stages fail most often (indicates where to invest)


9. Common Mistakes

9.1 Big Batch Deployments

Problem: Deploying all accumulated changes to production at once after weeks or months of development.
Why it is harmful: Large changesets are harder to test, harder to debug when failures occur, and carry significantly higher risk. Rollback is complex because you cannot isolate which change caused the issue.
Symptoms:
  • Release day is a stressful, all-hands event
  • Deployments require a detailed runbook and war room
  • Change failure rate over 30 percent
  • Rollbacks are rare because they are too risky
Fix: Deploy small, incremental changes frequently. Aim for multiple deployments per week, progressing toward multiple per day. Use feature flags to decouple deployment from release.

9.2 Long-Lived Feature Branches

Problem: Branches that live for weeks or months before merging back to the main branch.
Why it is harmful: The longer a branch lives, the more it diverges from trunk. Merge conflicts become painful and error-prone. Integration issues are discovered late.
Symptoms:
  • Merge day is a recurring event
  • Developers dread rebasing
  • Integration bugs appear only after merging
  • CI pipeline rarely runs on the actual integrated codebase
Fix: Adopt trunk-based development. Feature branches should live for hours, not days. Use feature flags to merge incomplete work safely.

9.3 Manual Deployments

Problem: Requiring manual steps — SSH into servers, run scripts, click through consoles — for every deployment.
Why it is harmful: Manual steps introduce human error, create bottlenecks, and discourage frequent releases. Teams batch changes to avoid the painful manual process, which increases risk.
Symptoms:
  • Deployment runbooks are multi-page documents
  • Only specific people know how to deploy
  • Deployments happen during maintenance windows (often nights or weekends)
  • Deployment frequency is monthly or less
Fix: Automate the deployment pipeline end-to-end. Replace manual gates with automated quality gates (tests, security scans, metric checks). Reserve manual approval only for regulatory or compliance requirements.

9.4 No Rollback Strategy

Problem: Deploying without a tested, automated way to revert to the previous version.
Why it is harmful: When a deployment fails, teams scramble to push a hotfix under pressure, often making things worse. Without a rollback path, every deployment is a one-way door.
Symptoms:
  • The response to a failed deployment is to push a hotfix
  • Previous artifacts are not retained or easily accessible
  • Nobody has practiced a rollback
  • Database migrations are irreversible
Fix: Every deployment strategy needs a documented and tested rollback mechanism. Practice rollbacks regularly (at least quarterly). Retain previous artifacts. Use backward-compatible database migrations.

9.5 Environment Drift

Problem: Development, staging, and production environments diverge in configuration, dependencies, OS versions, or infrastructure topology.
Why it is harmful: Bugs appear only in production because environments are not equivalent. Root cause investigation is complicated by environmental differences.
Symptoms:
  • Production-only bugs that cannot be reproduced in staging
  • Environment setup is manual and undocumented
  • Staging has different instance types, fewer instances, or different network topology
Fix: Use Infrastructure as Code (IaC) to provision all environments from the same templates with environment-specific parameters. Use containers to ensure runtime consistency. Regularly audit environment parity.

9.6 Testing in Production Only

Problem: Skipping pre-production testing and relying on production monitoring to catch issues.
Why it is harmful: Customers become your QA team. Issues are discovered after they have already caused impact.
Fix: Implement the test pyramid. Use canary deployments for production validation as an additional safety net, but never as a replacement for pre-production testing.

9.7 Snowflake Pipelines

Problem: Every team builds a completely custom pipeline with no shared patterns, libraries, or standards.
Why it is harmful: Duplicated effort across teams. Inconsistent quality gates. Difficult to audit or enforce organizational standards.
Symptoms:
  • No two teams pipelines look alike
  • Security team cannot verify that all pipelines include required scans
  • New teams spend weeks building a pipeline from scratch
  • Pipeline maintenance is a significant time sink for every team
Fix: Establish shared pipeline templates or libraries that teams can extend. Standardize mandatory quality gates (test coverage thresholds, security scan requirements, artifact signing) across the organization.

9.8 Slow Pipelines

Problem: Pipelines that take 45 or more minutes to complete, with no effort to optimize.
Why it is harmful: Slow pipelines discourage frequent commits, reduce developer productivity, and create long feedback loops.
Symptoms:
  • Developers push changes and go to lunch
  • Multiple commits are batched into a single pipeline run
  • Developers skip the pipeline and deploy manually to save time
Fix: Run independent test stages in parallel. Cache dependencies and build layers. Use incremental builds. Set a pipeline SLO (for example, under 10 minutes for CI, under 30 minutes for full CD). Profile the pipeline to find bottlenecks.

9.9 In-Place Patching

Problem: Updating software on existing running instances rather than replacing with new instances.
Why it is harmful: Configuration drift accumulates over time — instances that have been patched 50 times are in an unpredictable state. Rollback requires reverse-patching, which is error-prone.
Fix: Use immutable deployments — deploy to fresh infrastructure and decommission the old. Containers make this the default pattern.

9.10 Tightly Coupled Deployments

Problem: Multiple services or components that need to be deployed together in a specific order.
Why it is harmful: Deployment coordination becomes a bottleneck. A failure in one component blocks all others. Teams cannot deploy independently, reducing velocity.
Symptoms:
  • Deployment requires coordination across multiple teams
  • A single CI/CD pipeline deploys multiple services
  • API changes break downstream consumers
Fix: Architect loosely coupled systems with well-defined API contracts. Each service should be independently deployable. Use API versioning and backward-compatible changes. Implement contract testing between services.

9.11 No Observability Post-Deployment

Problem: Deploying without automated health checks, metric monitoring, or alerting.
Why it is harmful: You do not know if a deployment succeeded or degraded the service until customers report issues.
Fix: Implement automated post-deployment validation: smoke tests, synthetic monitors, error rate monitoring, latency checks. Configure deployment alarms to auto-rollback on metric breaches.


10. CI/CD Metrics Framework

Track these metrics to measure and improve your CI/CD practice.

Delivery Performance (DORA)

Metric
How to Measure
Target
Deployment Frequency
Count production deployments per time period
Multiple per day (elite)
Lead Time for Changes
Timestamp: commit to production deployment
Under 1 hour (elite)
Change Failure Rate
Failed deployments divided by total deployments
Under 15 percent (elite)
Mean Time to Recovery
Timestamp: incident start to service restored
Under 1 hour (elite)

Pipeline Health

Metric
How to Measure
Target
Pipeline Stability
Percentage of pipeline runs that complete without failure
Over 95 percent
Mean Time to Production (MTTP)
Average time from merge to production
Under 1 hour
Operator Interventions
Count of deployments requiring manual steps
0 (fully automated)
Changes per Release
Count of commits or PRs in each deployment
1 to 3 (small batches)
Pipeline Duration (CI)
Time from trigger to build plus test completion
Under 10 minutes
Pipeline Duration (Full CD)
Time from trigger to production deployment
Under 30 minutes
Flaky Test Rate
Percentage of test failures not caused by code changes
Under 2 percent
Queue Wait Time
Time builds spend waiting for runners
Under 2 minutes

Quality Indicators

Metric
How to Measure
Target
Test Coverage
Percentage of code covered by automated tests
Over 80 percent (unit), over 60 percent (integration)
Security Scan Completion Rate
Percentage of builds completing all security scans
Over 98 percent
Rollback Frequency
Rollbacks divided by total deployments
Under 5 percent
Escaped Defects
Bugs found in production divided by total bugs found
Under 10 percent


11. Maturity Model

Use this to evaluate where your team is and identify the next improvement area. Progress is incremental — complete each level before advancing.

Level 1 — Foundation

  • Source code in version control (Git)
  • Automated build triggered on every commit
  • Basic unit tests running in CI
  • Manual deployment to production using documented steps
  • Artifacts stored in a registry
  • Basic monitoring in production (uptime, error logs)
Key outcome: Builds are automated and repeatable. Deployments are documented but still manual.

Level 2 — Standardized

  • Pipeline defined as code (Jenkinsfile, GitHub Actions, etc.)
  • Automated deployment to staging environment
  • Integration and acceptance tests automated in the pipeline
  • Artifact built once and promoted through environments
  • Basic rollback capability (manual but documented)
  • Static code review and linting in CI
  • Dependency scanning for known vulnerabilities
  • Environment provisioned via Infrastructure as Code
Key outcome: The path to production is automated through staging. Quality gates are enforced automatically.

Level 3 — Optimized

  • Automated deployment to production (true CD)
  • Safe deployment strategy implemented (blue/green, canary, or rolling)
  • Security scanning integrated (SAST, SCA, container scanning, IaC scanning)
  • Post-deployment automated validation (smoke tests, synthetic monitors)
  • Automated rollback on alarm breach
  • Pipeline runs in under 10 minutes (CI) and under 30 minutes (full CD)
  • DORA metrics tracked and visible to the team
  • Secrets management via vault or parameter store (no hardcoded secrets)
Key outcome: Production deployments are automated, safe, and observable. Rollback is automated.

Level 4 — Advanced

  • Trunk-based development with feature flags
  • Canary deployments with automated metric-based promotion and rollback
  • Resilience engineering integrated into pipeline (scheduled)
  • DORA metrics reviewed regularly and driving improvement
  • Pipeline templates shared across teams with mandatory quality gates
  • Artifact signing and verification enforced
  • SBOM generated for every build
  • Contract testing between services
  • Deployment frequency: multiple per week
Key outcome: Deployments are routine, low-risk events. Cross-team standards are enforced. Supply chain security is addressed.

Level 5 — Elite

  • Multiple production deployments per day
  • Lead time under 1 hour from commit to production
  • Change failure rate under 15 percent
  • Mean time to recovery under 1 hour
  • Full GitOps — infrastructure and application managed declaratively via Git
  • Progressive delivery with automated experimentation
  • Pipeline performance continuously optimized
  • Zero manual interventions in the deployment path
Key outcome: Deployment is a non-event. The team focuses on building features, not managing releases.


12. Key Recommendations

For Teams Just Starting

  1. Start with CI — Get automated builds and unit tests running on every commit before tackling CD
  1. Automate one environment first — Automate deployment to a staging environment, then extend to production
  1. Invest in tests early — Automated tests are the foundation. Without them, CD is just continuous deployment of bugs
  1. Use managed services — Do not build your own CI/CD platform. Use GitHub Actions, GitLab CI, AWS CodePipeline, or similar

For Teams Maturing Their Practice

  1. Measure with DORA — Track the four key metrics. You cannot improve what you do not measure
  1. Shrink batch size — The single most impactful change is deploying smaller changes more frequently
  1. Automate rollback — Manual rollback under pressure is error-prone. Configure alarm-based auto-rollback
  1. Shift security left — Integrate SAST, SCA, and secrets detection into CI. Do not bolt security on at the end
  1. Standardize across teams — Create shared pipeline templates with mandatory quality gates

For Advanced Teams

  1. Adopt progressive delivery — Combine canary deployments with feature flags for maximum control
  1. Practice resilience engineering — Regularly inject failures to validate system resilience
  1. Optimize pipeline speed — Set and enforce pipeline SLOs. Profile and eliminate bottlenecks
  1. Manage feature flag lifecycle — Flags are powerful but become debt if not retired. Set expiration dates
  1. Treat the pipeline as a product — Assign ownership, track metrics, invest in improvements

Universal Principles

  1. Make deployments boring — Frequent, small, automated deployments should be a non-event
  1. Own your rollback — Every deployment needs a tested rollback path
  1. Avoid deploying before weekends — Unless your rollback is automated and your monitoring is excellent
  1. Blameless retrospectives — When deployments fail, focus on improving the system, not blaming individuals
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article