AWS Builder Center
Measure AI developer productivity from idea to production

Measure AI developer productivity from idea to production

In a Harvard study of 700+ firms, AI coding agents raised lines of code 30% while completed features didn't rise. See five practices for the full lifecycle.

Series: A study on AI in the product lifecycle (4 articles)

  1. 1
    Measure AI developer productivity from idea to production This article
Global Sr. Solutions Architect, Private Equity
Does this sound familiar? You adopted AI coding tools and your engineers say they feel faster. However, your delivery metrics haven't moved. In a  2025 randomized trial , experienced developers using AI tools took 19% longer to finish tasks while believing they were 20% faster. By  early 2026, the effect may have reversed , but the study could no longer measure it cleanly, because many developers declined to take part rather than work without AI.
I spent two years working with engineering teams through this transition. The teams seeing a return on investment (ROI) changed how work moves through their organization before they picked a tool. In this article, I walk through five principles that move AI investment from activity metrics, such as tokens generated and lines suggested, to delivery outcomes, meaning value that reaches customers faster.

1. Treat the lifecycle as a loop

Most teams picture delivery as a pipeline. Requirements go in, code comes out, and AI goes in the middle to make it faster. The teams getting results treat it as a loop instead. They decide what to build, build it, ship it, measure what it did, and feed what they learned into the next decision, so each pass starts from evidence.
This is the scientific method applied to product development. The software delivery lifecycle sits inside it and supplies the speed and stability. The shift to microservices over the past 10 years supports this approach, because it lets teams deliver in smaller increments.

Circular diagram of five stages in a loop: decide, build, ship, measure, and learn, with evidence fl

Figure 1: The five-stage product development loop, from decide to learn.
Code authoring is  25–35% of the idea-to-launch span . The rest is review, provisioning, deployment, and waiting. If you speed up only the coding, you risk a backlog in review and delivery pipelines that weren't built for the added load. A  2026 Harvard study of more than 700 firms  found that AI coding agents increased lines of code by 30% and the time from pull request submission to merge by 49%, while completed features didn't rise.
Apply this to a whole organization. If about a fifth of its people write code, and those engineers spend half their time coding, a 35% coding speedup works out to about 3.5% of the organization's total time. That's a real gain, but it's too small to move delivery metrics on its own.
The AWS Executives in Residence team reached the same conclusion in  Reimagine: Turning AI into Value . Their advice is to find a constraint that exists only because a person needed time, remove it, and redesign the work around its absence.

2. Shift left twice

AI gives you two places to move work earlier in the loop.
The first is the product decision. You can validate a use case before anyone writes code, since the cheapest fix is deciding not to build something. AI speeds up that validation by reading customer signals and producing quick prototypes.
The second is quality. Security and site reliability engineering (SRE) checks can run inside the coding agent as it writes, replacing reviewer queues that wait on time zones. Veracode's  2026 GenAI Code Security Report  found that about 44% of AI code generation tasks in its tests produced code with a known vulnerability, and the average security pass rate barely moved, from 55% in its first report to 56%.
Review is where the volume piles up.  LinearB telemetry across 8.1 million pull requests from 4,800 organizations, compiled in Encore 2026 , shows AI-generated pull requests (PRs) wait 4.6 times longer for first review and are accepted at only 32.7%, compared with 84.4% for human-written PRs. A  July 2026 Carnegie Mellon and Stanford study  of 802 developers and nearly 200,000 PRs at one company found the share of pull requests getting any written human feedback fell from 39% to 21%.
Teams moved code generation earlier without moving review to match.

Circular diagram of five stages in a loop: decide, build, ship, measure, and learn, with evidence fl

Figure 2: Code volume grows faster than review capacity, and shipped work doesn't rise to match.

3. How do you measure AI's impact on delivery?

Start with the four delivery metrics from the  Accelerate  research. They are deployment frequency, lead time for changes, change failure rate, and failed deployment recovery time, which earlier reports called mean time to restore. The 2024 DevOps Research and Assessment (DORA) report set the bands below, and  only 19% of organizations reached elite .
In 2024, medium performers had a lower change failure rate than high performers. The report describes them as slower but more stable.
Metric
Elite
High
Medium
Low
Deployment frequency
On demand
Daily to weekly
Weekly to monthly
Monthly to every six months
Lead time for changes
Less than 1 day
1 day to 1 week
1 week to 1 month
1–6 months
Change failure rate
5%
20%
10%
40%
Failed deployment recovery time
Less than 1 hour
Less than 1 day
Less than 1 day
1 week to 1 month
These bands matter more once AI arrives, because AI speeds up whatever system it lands in, including the broken parts. A 2026 DORA report,  The ROI of AI-Assisted Software Development , estimates what that costs. In its illustrative scenario, a change failure rate rising from 5% to 6% after AI adoption adds $344,000 in downtime for a single organization. The report calls this added downtime cost the instability tax.
DORA metrics cover the middle of the loop, from commit to production. To see whether AI changes outcomes, extend your measurement across the entire product lifecycle.
Stage
Metric
What it shows
Decide
Idea-to-spec cycle time
Whether product decisions keep pace with build capacity
Decide
Spec one-shot rate
Share of specs that produce a mergeable PR without revision
Ship
Review queue depth
Open PRs waiting on human review, per reviewer
Measure
Feature instrumentation rate
Share of shipped features with usage analytics attached
Learn
Feature adoption rate
Share of features reaching a usage threshold within 60 days
Full loop
Roadmap conversion
Share of the quarterly roadmap committed, shipped, and adopted
To do this, measure the full span from ticket created to running in production, including PR review, quality assurance waits, and approval queues. That span is your AI ROI denominator. Capture the baseline before you change anything, so you can see how each change affects your end goals.
When leadership asks for one number, give them cost per shipped unit of value, meaning fully loaded engineering cost per merged PR or per shipped feature, trended against that baseline and shown next to adoption breadth.
Most companies can't produce that number yet. On Q2 2026 S&P 500 earnings calls,  Goldman Sachs Research  found that 11% of companies quantified a specific use-case impact of AI, and 2% quantified an effect on earnings. Quantifying an effect on earnings takes a baseline like this one.

How do you measure the agents themselves?

Once agents complete tasks end to end, you also need to measure the agent itself.
Metric
What it measures
Level
Tool selection accuracy
Did the agent pick the correct tool for the intent?
Tool call
Goal success rate
Did the agent achieve the stated objective?
Session
Trajectory match
Did the agent follow the expected sequence of steps?
Session
Faithfulness
Is the response grounded in tool results?
Trace
These metrics are the evaluation framework that decides how much autonomy you grant your agents in Principle 4.

4. When can an agent work on its own?

Trust in AI builds in three levels on a maturity curve: targeted comes first, then one-shot, then autonomous. Teams that struggle usually skipped one, jumping from autocomplete straight to autonomous agents without the methodology in between.

Circular diagram of five stages in a loop: decide, build, ship, measure, and learn, with evidence fl

Figure 3: The three trust levels, from targeted to one-shot to autonomous.
At the targeted level, an engineer asks and the agent assists. In the 2025 DORA report,  90% of technology professionals  said they use AI at work. Individual tasks get faster, yet organizational delivery stays flat, because coding is a small share of the idea-to-launch span (Principle 1).
At the one-shot level, you give the agent full task context, meaning requirements, constraints, and acceptance criteria, and it completes a whole unit of work for your review. Getting here depends more on documentation than on the model. Your organizational context and institutional knowledge need to be written into structured files that agents read directly. That practice is spec-driven development, where you write reviewable requirements, design, and task definitions before any code exists. AWS open-sourced an  AI-Driven Development Lifecycle  for structuring it, and  Kiro  formalizes the spec workflow from requirements through tasks.
At the autonomous level, agents pick up tickets and open PRs, which takes evaluated trust. Measure the agent with the goal success rate and trajectory match from Principle 3. On each type of task, its error rate should fall below the human error rate from Principle 5 before you widen its scope. Stripe's internal coding agents ( Minions ) produce more than 1,000 merged pull requests each week that contain no human-written code, and a person reviews each one. Ramp reports that its internal coding agent writes roughly 30% of merged pull requests across its frontend and backend repositories, supported by a verification loop that runs tests, checks monitoring, and joins code reviews before a human approves ( InfoQ ).
You can see a team getting ready to move up a level by watching how much work an agent runs before a person steps in, and how many PRs merge without edits. The attempts it takes to reach a merged PR should fall toward one, and the share of an engineer's day spent supervising an agent should drop.

5. Capture the pre-AI wins first

Some of the largest gains are not AI-driven. In the 2026 State of DevOps report,  70% of organizations  said DevOps maturity meaningfully influences their AI success. Encore's 2026 delivery research found that  moving from low to moderate automation  doubles the likelihood of realizing AI velocity gains, from 26% to 57%.
Start by removing manual gates and consider replacing human review chains on application-level changes with one peer reviewer plus evaluated security and SRE agents.
Next, as Principle 4 noted, write down what your team knows. An agent works only from what it can read, so move undocumented conventions into versioned Markdown in the repository. For large bodies of knowledge and skills, consider progressive disclosure. Keep a lightweight map that points to deeper docs the agent fetches on demand. That keeps context more focused than one exhaustive file. In my experience, tests and documented context do more for one-shot success than anything else in the repository.  AWS Prescriptive Guidance on coding agents  covers how agents gather and act on repository context.
Then define good before you automate. Collect representative samples of good and bad output for each gate, and measure the human error rate. That rate becomes the baseline you compare agents against.

Where AWS fits the loop

The five principles don't depend on any one tool, nor should your current toolset dictate your practice.

Circular diagram of five stages in a loop: decide, build, ship, measure, and learn, with evidence fl

Figure 4: Where AWS products map onto the product development loop.
 Amazon Quick  supports the first and last stages of the loop. At the decide stage, it grounds the choice in your own data through research, voice of customer, and validation. At measure and learn, it reads analytics and experiment results and routes the evidence into the next decision, for business and technology teams together. The build stage runs on Kiro. Kiro generates the requirements, design, and task files from a prompt, and once you review them, it implements the tasks. That's the one-shot level from Principle 4. The  AI-Driven Development Lifecycle  structures that handoff.
The managed agents work across build, ship, and measure.  AWS Continuum  finds risks in designs and code before release, runs on-demand penetration tests, and takes vulnerabilities from discovery through remediation across the loop. A person reviews and merges each fix it proposes.  AWS DevOps Agent  returns release-readiness verdicts and investigates incidents. Measure their effect the same way as any other change, with the review queue depth, change failure rate, and failed deployment recovery time from Principle 3. Underneath them, delivery services carry the work.  AWS CodeCommit  holds the source.  AWS CodePipeline ,  AWS CodeBuild , and  AWS CodeDeploy  run the build-and-deploy pipeline.  AWS AppConfig  handles feature-flagged release, and  Amazon CloudWatch  telemetry feeds the measure stage.
You also choose where a person stays in the loop. The AWS Security blog's  control framework for AI coding agents  separates author-time controls, covering what the agent produces in the integrated development environment (IDE), from build-time controls, covering what reaches production. The framework reserves human review for decisions that need judgment. That lightens review on low-risk changes and focuses it on high-risk ones. This risk-based split works only after you've set the human baseline (Principle 5) and the agent meets the threshold (Principle 4).

Conclusion

As  O'Reilly's 2026 analysis  put it, "The ceiling isn't code authoring speed anymore; it actually never was." The return on AI comes from your pipeline, your measurement, and the structured context your agents work from.
You can start this week by measuring the full ticket-to-production span. This month, remove one manual review gate and document one set of architectural constraints in a format agents can read.
On AWS, the AI-Driven Development Lifecycle and Kiro support the spec-driven practice from Principle 4. AWS Continuum and AWS DevOps Agent add checks at the review and release stages, and Amazon Quick helps business and technology teams measure what each change produced, which closes the loop.

Further reading

About the Author

Michael Purpura is a Senior Global Private Equity Solutions Architect at AWS. He works with private equity firms and their portfolio companies to structure AI adoption around measurable business outcomes.

Series: A study on AI in the product lifecycle (4 articles)

  1. 1
    Measure AI developer productivity from idea to production This article
Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.
Enjoyed reading this content? Let the author know!

Your likes, comments, shares, and saves help creators reach more builders.

Loading recommendations

Loading article