AWS CodeBuild vs. Self-Hosted GitHub Actions Runners: Lessons from Scaling CI/CD
AWS CodeBuild offers a managed approach to running GitHub Actions workflows, but scaling CI/CD introduces challenges around runner startup time, compatibility, queue delays, webhook limits, and reliability. Explore the key lessons from a real-world engineering experience.
Introduction
Modern DevOps teams want CI/CD infrastructure that is scalable, reliable, secure, and cost-effective. GitHub Actions makes it easier to automate software delivery, while AWS provides managed services that reduce the operational burden of maintaining build infrastructure.
At first glance, AWS CodeBuild for GitHub Actions appears to be a natural solution for teams that want self-hosted runners without managing their own servers.
However, the requirements of a large-scale engineering environment can be very different from those of a small project.
When workloads involve hundreds or thousands of concurrent jobs, even small limitations in startup time, capacity, compatibility, or monitoring can become significant operational problems.
This article explores the main lessons from using CodeBuild for GitHub Actions runners at scale and explains why self-managed EC2-based runners can sometimes be a better fit.
1. Why CodeBuild Looked Like the Right Choice
Managing CI/CD infrastructure requires engineering effort. Teams must provision machines, maintain operating system images, monitor resource usage, handle scaling, and ensure that build environments remain isolated.
A managed service can reduce much of this work.
AWS CodeBuild for GitHub Actions provides a way to connect GitHub workflows to build environments launched by CodeBuild. This approach allows teams to use AWS infrastructure without building a complete runner-management platform themselves.
For teams prioritizing simplicity, this can be attractive.
The challenge emerges when the system must satisfy demanding requirements, such as:
- Supporting around 1,000 concurrent builds.
- Handling sudden bursts of approximately 200 jobs.
- Starting runners within seconds.
- Supporting both x86 and ARM workloads.
- Providing an isolated environment for each job.
- Ensuring that every triggered job eventually receives a runner.
- Maintaining reasonable cost per build minute.
A solution that works well for moderate workloads may not automatically satisfy these requirements.
2. Runner Startup Time Affects Developer Productivity
CI/CD pipelines are part of the developer feedback loop. When a developer pushes a change, they expect the build and tests to start quickly.
With CodeBuild's on-demand runner approach, the GitHub Actions runner must be downloaded during startup. The source article reports an additional startup delay of approximately 10–20 seconds.
That delay may seem insignificant for a single build. Across thousands of jobs, however, unnecessary startup time can affect the overall development experience.
Reserved capacity fleets offer additional customization options, but they introduce trade-offs when teams require a strictly isolated, one-job-per-runner environment.
Lesson: Evaluate the complete time required for a job to become productive, not just the time required to provision compute resources.
3. Compatibility Is More Than Operating System Support
GitHub Actions workflows frequently depend on third-party actions, scripting environments, language runtimes, and helper functions.
A runner environment must support these requirements consistently.
The source describes several compatibility limitations encountered with CodeBuild's EC2-based environments, including:
- Node.js not being installed in the tested images.
- JavaScript and TypeScript-based actions failing as a result.
- Limitations involving ARM image availability.
- A reported problem with GitHub Actions'
hashFileshelper function. - Differences between EC2, container, and Lambda execution environments.
Custom container images can address some environment requirements, but they do not remove every limitation associated with the available execution modes.
Lesson: Before migrating a CI/CD platform, test representative workflows rather than validating only that a simple pipeline can run.
Include dependency installation, caching, architecture requirements, third-party actions, and real production build scripts in the evaluation.
4. Compute Capacity Must Match the Workload
Build jobs can have very different resource requirements.
A small unit-test job might run comfortably on a modest instance, while a large compilation or container build may need significantly more CPU and memory.
The source reports that the tested EC2-based CodeBuild options offered a limited range of instance sizes, while container-based options had larger gaps between available resource configurations.
These constraints can create two problems.
First, an undersized environment can increase build duration or cause failures. Second, selecting a larger environment than necessary can increase costs.
Reserved capacity fleets provide more flexibility in instance selection, but their characteristics may not align with every team's isolation and availability requirements.
Lesson: Compare available CPU, memory, architecture, startup behavior, and isolation guarantees against actual workload requirements.
The best compute configuration is the one that meets performance and security requirements without paying for unnecessary capacity.
5. Webhook Limits Can Become a Scaling Bottleneck
GitHub Actions relies on events to trigger workflows. In a large organization, many repositories and workflows can generate events at the same time.
The source describes an account-wide webhook-processing limit encountered during the CodeBuild integration. According to that experience, the limit was not clearly surfaced in the standard quota information, and addressing it required contacting AWS Support.
Multiple CodeBuild projects listening to organization-wide events could increase the number of webhook events processed.
Possible responses included requesting a limit increase or distributing workloads across AWS accounts.
This illustrates an important scaling principle: the compute layer is only one part of the system.
A deployment platform may have sufficient CPU and memory but still experience delays or failures because its event-processing and orchestration layers cannot keep up.
Lesson: Investigate event throughput, quotas, retry behavior, and failure visibility before choosing a runner architecture for a high-volume organization.
6. Monitoring Is Essential for Runner Reliability
One of the most important findings in the source concerns jobs that were triggered but never picked up by a runner.
A CI/CD system needs more than a successful response from an integration endpoint. It must provide evidence that a workflow event was received, a runner was started, and the intended job actually executed.
The source describes difficulty monitoring GitHub webhook delivery centrally and identifying cases where jobs remained waiting for runners.
Without suitable monitoring, a team may discover the problem only when a developer notices a workflow stuck in the queue.
A robust monitoring strategy should track:
- Workflow jobs waiting for runners.
- Runner startup failures.
- Queue duration.
- Webhook delivery failures.
- Build failures and retries.
- Job execution time.
- Compute utilization and cost.
Where supported, automated alerts and retries can reduce the time required to detect and recover from failures.
Lesson: Treat CI/CD infrastructure as a production system. Monitor not only whether jobs fail, but also whether jobs start when expected.
7. Runner Isolation and Job Assignment Matter
Security is another important consideration when running self-hosted CI/CD infrastructure.
Some teams require immutable runners, where each runner handles only one job and does not retain state that could affect subsequent jobs.
The source reports that CodeBuild's runner-label behavior created a job-assignment challenge. Because runners are matched using labels, insufficiently specific labels could allow a job to be assigned to an unintended runner while another job remained waiting.
The described workaround was to assign each job a unique custom label in addition to the CodeBuild label.
This is a useful reminder that runner provisioning and job scheduling are separate concerns.
Starting a runner does not necessarily guarantee that the correct job will be assigned to it.
Lesson: Test isolation, label matching, cleanup, and job-to-runner assignment under concurrent workloads. Document the required workflow configuration so that individual teams do not accidentally introduce reliability problems.
8. Cost Requires More Than Comparing Service Prices
Choosing a CI/CD platform based only on its published compute price can lead to misleading conclusions.
The source identifies several cost considerations, including per-minute billing for the relevant CodeBuild EC2 compute options, a minimum billing period for reserved capacity fleet instances, and differences in the available pricing models of alternative compute services.
However, managed services also reduce the engineering effort required to maintain infrastructure.
Self-hosted runners introduce their own costs:
- EC2 compute and storage.
- Machine image maintenance.
- Autoscaling configuration.
- Monitoring and alerting.
- Security updates.
- Failure recovery.
- Engineering time spent operating the platform.
The source reports that the team ultimately found its own EC2-based runner setup cheaper per build minute for its particular workload.
That result should not be treated as a universal pricing conclusion. Actual costs depend on utilization, build duration, concurrency, instance selection, operational overhead, and workload characteristics.
Lesson: Compare total cost of ownership, not just the price of a single build minute.
9. Why the Team Moved to Self-Hosted EC2 Runners
According to the source, the deciding factor was not simply startup speed or compute pricing. It was reliability.
Repeated cases of jobs waiting for runners created a serious problem for the team's development workflow. In a repository with frequent commits and a merge queue, one stuck job could delay the work behind it and require manual investigation.
The team chose to build its own runner infrastructure using Auto Scaling groups and EC2 instances.
This approach required the team to take responsibility for operating the infrastructure. In return, it gained greater control over machine images, instance types, startup behavior, and monitoring.
The source reports that the resulting setup could scale to thousands of instances and handle large bursts of new runners per minute.
These results reflect that team's environment and implementation rather than a guarantee for every EC2-based architecture.
The broader takeaway is that managing infrastructure directly can be justified when the operational requirements are specific enough that a managed service's constraints become more costly than the work of operating a custom solution.
10. Does This Mean CodeBuild Is the Wrong Choice?
No.
The source explicitly recognizes that CodeBuild can remain a good fit for teams with moderate build volumes, more flexible startup requirements, and workflows compatible with the available execution environments.
A managed service may be preferable when reducing operational overhead is more important than controlling every aspect of the runner environment.
Self-hosted runners may be more appropriate when a team needs:
- Highly customized machine images.
- Specific CPU and memory configurations.
- Strict one-job-per-runner isolation.
- Predictable startup performance.
- High concurrency and burst capacity.
- Detailed control over runner monitoring and recovery.
The decision should be based on workload characteristics, not on the assumption that managed or self-managed infrastructure is always superior.
11. A Practical Evaluation Checklist
Before selecting a runner platform, evaluate it against real workflow requirements.
| Area | Questions to ask |
|---|---|
| Startup | How long does a job wait before execution begins? |
| Scale | Can the platform handle peak concurrency and sudden bursts? |
| Compatibility | Do required actions, runtimes, architectures, and scripts work? |
| Isolation | Can jobs be kept isolated according to security requirements? |
| Scheduling | Is every job assigned to the intended runner? |
| Reliability | Are runner startup failures and stuck jobs detected automatically? |
| Monitoring | Can queue time, webhook delivery, and runner health be monitored? |
| Cost | What is the total cost, including operational effort? |
| Recovery | Can failed jobs be detected, retried, and recovered safely? |
Run a realistic load test before committing to the architecture. A small proof of concept can validate basic functionality, but only a representative workload can reveal many scaling and reliability issues.
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article