
Measure AI developer productivity from idea to production
In a Harvard study of 700+ firms, AI coding agents raised lines of code 30% while completed features didn't rise. See five practices for the full lifecycle.
Series: A study on AI in the product lifecycle (4 articles)
- 1Measure AI developer productivity from idea to production This article
Does this sound familiar? You adopted AI coding tools and your engineers say they feel faster. However, your delivery metrics haven't moved. In a 2025 randomized trial , experienced developers using AI tools took 19% longer to finish tasks while believing they were 20% faster. By early 2026, the effect may have reversed , but the study could no longer measure it cleanly, because many developers declined to take part rather than work without AI.
I spent two years working with engineering teams through this transition. The teams seeing a return on investment (ROI) changed how work moves through their organization before they picked a tool. In this article, I walk through five principles that move AI investment from activity metrics, such as tokens generated and lines suggested, to delivery outcomes, meaning value that reaches customers faster.
1. Treat the lifecycle as a loop
Most teams picture delivery as a pipeline. Requirements go in, code comes out, and AI goes in the middle to make it faster. The teams getting results treat it as a loop instead. They decide what to build, build it, ship it, measure what it did, and feed what they learned into the next decision, so each pass starts from evidence.
This is the scientific method applied to product development. The software delivery lifecycle sits inside it and supplies the speed and stability. The shift to microservices over the past 10 years supports this approach, because it lets teams deliver in smaller increments.

Figure 1: The five-stage product development loop, from decide to learn.
Code authoring is 25–35% of the idea-to-launch span . The rest is review, provisioning, deployment, and waiting. If you speed up only the coding, you risk a backlog in review and delivery pipelines that weren't built for the added load. A 2026 Harvard study of more than 700 firms found that AI coding agents increased lines of code by 30% and the time from pull request submission to merge by 49%, while completed features didn't rise.
Apply this to a whole organization. If about a fifth of its people write code, and those engineers spend half their time coding, a 35% coding speedup works out to about 3.5% of the organization's total time. That's a real gain, but it's too small to move delivery metrics on its own.
The AWS Executives in Residence team reached the same conclusion in Reimagine: Turning AI into Value . Their advice is to find a constraint that exists only because a person needed time, remove it, and redesign the work around its absence.
2. Shift left twice
AI gives you two places to move work earlier in the loop.
The first is the product decision. You can validate a use case before anyone writes code, since the cheapest fix is deciding not to build something. AI speeds up that validation by reading customer signals and producing quick prototypes.
The second is quality. Security and site reliability engineering (SRE) checks can run inside the coding agent as it writes, replacing reviewer queues that wait on time zones. Veracode's 2026 GenAI Code Security Report found that about 44% of AI code generation tasks in its tests produced code with a known vulnerability, and the average security pass rate barely moved, from 55% in its first report to 56%.
Review is where the volume piles up. LinearB telemetry across 8.1 million pull requests from 4,800 organizations, compiled in Encore 2026 , shows AI-generated pull requests (PRs) wait 4.6 times longer for first review and are accepted at only 32.7%, compared with 84.4% for human-written PRs. A July 2026 Carnegie Mellon and Stanford study of 802 developers and nearly 200,000 PRs at one company found the share of pull requests getting any written human feedback fell from 39% to 21%.
Teams moved code generation earlier without moving review to match.

Figure 2: Code volume grows faster than review capacity, and shipped work doesn't rise to match.
3. How do you measure AI's impact on delivery?
Start with the four delivery metrics from the Accelerate research. They are deployment frequency, lead time for changes, change failure rate, and failed deployment recovery time, which earlier reports called mean time to restore. The 2024 DevOps Research and Assessment (DORA) report set the bands below, and only 19% of organizations reached elite .
In 2024, medium performers had a lower change failure rate than high performers. The report describes them as slower but more stable.
Metric | Elite | High | Medium | Low |
|---|---|---|---|---|
Deployment frequency | On demand | Daily to weekly | Weekly to monthly | Monthly to every six months |
Lead time for changes | Less than 1 day | 1 day to 1 week | 1 week to 1 month | 1–6 months |
Change failure rate | 5% | 20% | 10% | 40% |
Failed deployment recovery time | Less than 1 hour | Less than 1 day | Less than 1 day | 1 week to 1 month |
These bands matter more once AI arrives, because AI speeds up whatever system it lands in, including the broken parts. A 2026 DORA report, The ROI of AI-Assisted Software Development , estimates what that costs. In its illustrative scenario, a change failure rate rising from 5% to 6% after AI adoption adds $344,000 in downtime for a single organization. The report calls this added downtime cost the instability tax.
DORA metrics cover the middle of the loop, from commit to production. To see whether AI changes outcomes, extend your measurement across the entire product lifecycle.
Stage | Metric | What it shows |
|---|---|---|
Decide | Idea-to-spec cycle time | Whether product decisions keep pace with build capacity |
Decide | Spec one-shot rate | Share of specs that produce a mergeable PR without revision |
Ship | Review queue depth | Open PRs waiting on human review, per reviewer |
Measure | Feature instrumentation rate | Share of shipped features with usage analytics attached |
Learn | Feature adoption rate | Share of features reaching a usage threshold within 60 days |
Full loop | Roadmap conversion | Share of the quarterly roadmap committed, shipped, and adopted |
To do this, measure the full span from ticket created to running in production, including PR review, quality assurance waits, and approval queues. That span is your AI ROI denominator. Capture the baseline before you change anything, so you can see how each change affects your end goals.
When leadership asks for one number, give them cost per shipped unit of value, meaning fully loaded engineering cost per merged PR or per shipped feature, trended against that baseline and shown next to adoption breadth.
Most companies can't produce that number yet. On Q2 2026 S&P 500 earnings calls, Goldman Sachs Research found that 11% of companies quantified a specific use-case impact of AI, and 2% quantified an effect on earnings. Quantifying an effect on earnings takes a baseline like this one.
How do you measure the agents themselves?
Once agents complete tasks end to end, you also need to measure the agent itself.
Metric | What it measures | Level |
|---|---|---|
Tool selection accuracy | Did the agent pick the correct tool for the intent? | Tool call |
Goal success rate | Did the agent achieve the stated objective? | Session |
Trajectory match | Did the agent follow the expected sequence of steps? | Session |
Faithfulness | Is the response grounded in tool results? | Trace |
These metrics are the evaluation framework that decides how much autonomy you grant your agents in Principle 4.
4. When can an agent work on its own?
Trust in AI builds in three levels on a maturity curve: targeted comes first, then one-shot, then autonomous. Teams that struggle usually skipped one, jumping from autocomplete straight to autonomous agents without the methodology in between.

Figure 3: The three trust levels, from targeted to one-shot to autonomous.
At the targeted level, an engineer asks and the agent assists. In the 2025 DORA report, 90% of technology professionals said they use AI at work. Individual tasks get faster, yet organizational delivery stays flat, because coding is a small share of the idea-to-launch span (Principle 1).
At the one-shot level, you give the agent full task context, meaning requirements, constraints, and acceptance criteria, and it completes a whole unit of work for your review. Getting here depends more on documentation than on the model. Your organizational context and institutional knowledge need to be written into structured files that agents read directly. That practice is spec-driven development, where you write reviewable requirements, design, and task definitions before any code exists. AWS open-sourced an AI-Driven Development Lifecycle for structuring it, and Kiro formalizes the spec workflow from requirements through tasks.
At the autonomous level, agents pick up tickets and open PRs, which takes evaluated trust. Measure the agent with the goal success rate and trajectory match from Principle 3. On each type of task, its error rate should fall below the human error rate from Principle 5 before you widen its scope. Stripe's internal coding agents ( Minions ) produce more than 1,000 merged pull requests each week that contain no human-written code, and a person reviews each one. Ramp reports that its internal coding agent writes roughly 30% of merged pull requests across its frontend and backend repositories, supported by a verification loop that runs tests, checks monitoring, and joins code reviews before a human approves ( InfoQ ).
You can see a team getting ready to move up a level by watching how much work an agent runs before a person steps in, and how many PRs merge without edits. The attempts it takes to reach a merged PR should fall toward one, and the share of an engineer's day spent supervising an agent should drop.
5. Capture the pre-AI wins first
Some of the largest gains are not AI-driven. In the 2026 State of DevOps report, 70% of organizations said DevOps maturity meaningfully influences their AI success. Encore's 2026 delivery research found that moving from low to moderate automation doubles the likelihood of realizing AI velocity gains, from 26% to 57%.
Start by removing manual gates and consider replacing human review chains on application-level changes with one peer reviewer plus evaluated security and SRE agents.
Next, as Principle 4 noted, write down what your team knows. An agent works only from what it can read, so move undocumented conventions into versioned Markdown in the repository. For large bodies of knowledge and skills, consider progressive disclosure. Keep a lightweight map that points to deeper docs the agent fetches on demand. That keeps context more focused than one exhaustive file. In my experience, tests and documented context do more for one-shot success than anything else in the repository. AWS Prescriptive Guidance on coding agents covers how agents gather and act on repository context.
Then define good before you automate. Collect representative samples of good and bad output for each gate, and measure the human error rate. That rate becomes the baseline you compare agents against.
Where AWS fits the loop
The five principles don't depend on any one tool, nor should your current toolset dictate your practice.

Figure 4: Where AWS products map onto the product development loop.
Amazon Quick supports the first and last stages of the loop. At the decide stage, it grounds the choice in your own data through research, voice of customer, and validation. At measure and learn, it reads analytics and experiment results and routes the evidence into the next decision, for business and technology teams together. The build stage runs on Kiro. Kiro generates the requirements, design, and task files from a prompt, and once you review them, it implements the tasks. That's the one-shot level from Principle 4. The AI-Driven Development Lifecycle structures that handoff.
The managed agents work across build, ship, and measure. AWS Continuum finds risks in designs and code before release, runs on-demand penetration tests, and takes vulnerabilities from discovery through remediation across the loop. A person reviews and merges each fix it proposes. AWS DevOps Agent returns release-readiness verdicts and investigates incidents. Measure their effect the same way as any other change, with the review queue depth, change failure rate, and failed deployment recovery time from Principle 3. Underneath them, delivery services carry the work. AWS CodeCommit holds the source. AWS CodePipeline , AWS CodeBuild , and AWS CodeDeploy run the build-and-deploy pipeline. AWS AppConfig handles feature-flagged release, and Amazon CloudWatch telemetry feeds the measure stage.
You also choose where a person stays in the loop. The AWS Security blog's control framework for AI coding agents separates author-time controls, covering what the agent produces in the integrated development environment (IDE), from build-time controls, covering what reaches production. The framework reserves human review for decisions that need judgment. That lightens review on low-risk changes and focuses it on high-risk ones. This risk-based split works only after you've set the human baseline (Principle 5) and the agent meets the threshold (Principle 4).
Conclusion
As O'Reilly's 2026 analysis put it, "The ceiling isn't code authoring speed anymore; it actually never was." The return on AI comes from your pipeline, your measurement, and the structured context your agents work from.
You can start this week by measuring the full ticket-to-production span. This month, remove one manual review gate and document one set of architectural constraints in a format agents can read.
On AWS, the AI-Driven Development Lifecycle and Kiro support the spec-driven practice from Principle 4. AWS Continuum and AWS DevOps Agent add checks at the review and release stages, and Amazon Quick helps business and technology teams measure what each change produced, which closes the loop.
Further reading
- Reimagine: Turning AI into Value (AWS Executives in Residence)
- Agentic AI: Bridging the Widening Gap Between Ambition and Execution (AWS Enterprise Strategy)
About the Author
Michael Purpura is a Senior Global Private Equity Solutions Architect at AWS. He works with private equity firms and their portfolio companies to structure AI adoption around measurable business outcomes.
Series: A study on AI in the product lifecycle (4 articles)
- 1Measure AI developer productivity from idea to production This article
Enjoyed reading this content? Let the author know!
Your likes, comments, shares, and saves help creators reach more builders.
Loading recommendations
Loading article