By Julia Iv

AI coding agents can produce more code, open more pull requests, and take on longer implementation tasks. For an engineering team, the harder question is how much of that activity becomes reliable software that people actually use.

The research reveals a gap between those outcomes. One study covering 718 firms associated agent adoption with roughly 30% more code and 23% more pull requests, alongside a 49% increase in review time. Measured gains in downstream software output were small and statistically insignificant.

That finding does not establish that AI coding agents are ineffective. It shows why implementation speed alone is an incomplete measure of their value.

For software engineers, the transition changes both the work they delegate and the work they must evaluate. Understanding that shift requires looking at production deployments, productivity measurements, software quality, and the development of engineering skills together.

AI Agents Are Already Part of Production Engineering

Agent-based engineering extends delegation across the development process. An engineer supplies objectives, repository context, constraints, and acceptance criteria. The agent explores the codebase, edits files, runs tools, and proposes a change. Tests, reviewers, and operational controls determine whether that change moves forward.

Several examples show how this works in practice:

These examples establish different kinds of capability. AlphaEvolve’s specialized algorithm search is different from an application agent handling an issue. Stripe’s pull request volume does not reveal the share of all engineering work automated or the net human time saved.

The environment also matters. An agent operating inside a mature organization can benefit from established documentation, standard tools, isolated execution, observability, and reliable deployment procedures. Its results cannot simply be transferred to an unfamiliar repository with weak tests and undocumented behavior.

For a team evaluating adoption, the surrounding engineering infrastructure is part of the system being evaluated.

Productivity Depends on What You Count

The studies become easier to interpret once their measures are separated. Time spent on a selected task, code volume, pull requests, releases, and application use describe different outcomes.

METR illustrates how sensitive task-level results can be. Its earlier randomized study in 2025 found that developers took 19% longer to complete the selected work with AI assistance. A February 24, 2026 update reported later samples with estimated time reductions of 18% and 4%. Both confidence intervals included zero, so those estimates did not establish a clear speedup.

The update also identified selection problems and difficulties measuring work involving concurrent agents. Neither the earlier slowdown nor the later estimates provides a universal productivity multiplier for software engineering.

Chen and Stratton examined what happens across firms. Their August 4 working-paper version covered 718 firms and associated agent adoption with approximately 30% more code, 23% more pull requests, and 49% more review time. Measured increases in downstream software output were small and statistically insignificant.

This was an observational study using an identification strategy, rather than random assignment. Its measures also could not capture every kind of customer or business value. Still, it identifies a practical concern: additional implementation activity can create additional work elsewhere in the organization.

Demirer, Musolff, and Yang found a similar gap between activity and delivery. Their June 21 author summary reported estimated commit increases reaching 180% for asynchronous agents, while releases increased approximately 30%. Growth in activity and releases substantially exceeded measured growth in application use.

Historical controls and matching helped address selection differences, but the analysis was not a randomized experiment.

Taken together, these results make a claim such as “AI makes engineers three times more productive” incomplete. The claim needs an outcome, a comparison group, a quality standard, and a time period.

For an engineering manager, the useful question is what improved between the original request and stable use of the resulting software.

The Work Shifts Toward Delegation and Evaluation

AI can take on implementation work across many engineering specialties. The decisions surrounding that work still need strong evaluation and clear ownership.

For routine applications and frontend development, delegation can include scaffolding, CRUD operations, forms, components, styling variants, and accessibility checks. Evaluation must still address whether the application solves a real need, handles data appropriately, and provides a coherent experience for actual users.

For backend development and API integrations, agents can implement endpoints, validation, service wiring, clients, and adapters. The consequential questions include transaction boundaries, authorization, failure behavior, capacity, rate limits, changing API versions, and third-party outages.

For databases and infrastructure, agents can construct queries, draft migrations, suggest indexes, prepare configurations, and propose deployment plans. Data correctness, concurrency, access scope, production changes, rollback, and recovery require particular attention.

For testing and security, agents can generate tests, perform analysis, suggest patches, and maintain regression coverage. Someone still needs to establish whether those checks represent the requirements, address relevant threats, and test the business’s actual authorization rules.

For systems programming and architecture, delegation can include bounded optimizations, ports, instrumentation, dependency analysis, and migration plans. Evaluation extends to memory behavior, concurrency, hardware assumptions, performance under load, and tradeoffs spanning teams and future changes.

For legacy maintenance, requirements, and distributed debugging, agents can navigate code, propose refactors, draft specifications, analyze logs, and generate hypotheses. The difficult questions involve undocumented behavior users depend on, conflicting stakeholder needs, and causal diagnosis from incomplete observations.

This is a map of work suited to delegation and work requiring careful judgment. It is not a measured percentage of engineering already automated, or evidence that any particular activity will remain exclusively human.

Passing Tests Can Hide a Security Gap

A change can function as requested while failing an important security requirement.

The October 10 Last Secure Code Benchmark preprint evaluated 450 tasks derived from 150 real vulnerabilities. Functional pass rates ranged from 73.6% to 91.3%. The best result satisfying both functional and security checks was 25.1%.

That difference is substantial, but its scope matters. The benchmark deliberately concentrates on security-related tasks. It does not establish that three-quarters of all AI-generated software is insecure.

Its practical value is more specific: successful functional behavior cannot stand in for security validation.

The concern becomes greater when an agent generates the implementation, tests, and explanation from the same mistaken assumption. Agreement among those outputs may give reviewers confidence without providing an independent check of the requirement.

A useful review therefore asks what the tests establish, which assumptions they share with the implementation, and whether consequential failure modes received separate scrutiny.

More Code Can Outrun a Team’s Understanding

Source code can remain readable while the pace of change exceeds the team’s ability to understand its interactions.

Individual patches may pass tests while their combined effect makes the architecture harder to modify. An engineer may understand each local change without understanding what the accumulating changes do to the system.

A March 30 maintainability preprint examined 304,362 AI-attributed commits across 6,275 repositories. It identified substantial static-analysis issues, with 24.2% of the identified issues persisting to the latest observed revision.

Static-analysis warnings are not all runtime defects. The study also did not establish that equivalent human work would contain fewer problems. Its relevance is that acceptance of a generated change does not end the maintenance question.

Teams need to evaluate what remains after the pull request is merged: dependencies, interactions, unresolved warnings, and the effort required to make the next change.

Human review only creates meaningful accountability when reviewers can evaluate consequential behavior. A signature on an incomprehensible change provides little operational protection.

AI Assistance Can Affect How Engineers Learn

The future of software engineering also depends on how developers acquire the knowledge needed to evaluate generated work.

Anthropic’s January 29 randomized study involved 52 mostly junior Python developers learning a new library. Average quiz scores were 50% with AI assistance and 67% without it: a difference of 17 percentage points. The time advantage from AI assistance was not statistically significant.

Participants who actively requested explanations tended to learn more than those who delegated extensively. However, those comparisons between usage patterns were not themselves randomized.

One short exercise cannot establish permanent loss of skill. It identifies a concern worth testing over longer periods: completing a task with assistance and learning enough to perform or evaluate similar work are separate outcomes.

Anthropic’s much larger Claude Code session study also found that experienced users directed work differently and achieved better measured outcomes. Those success measures came from interaction and tool signals, rather than independent measures of customer value or future incident rates.

The evidence supports the usefulness of expertise in the observed workflows. It does not guarantee that expertise will retain the same economic premium indefinitely.

For training programs, the practical implication is to assess understanding alongside task completion. Reading code, modeling state changes, anticipating failures, and constructing meaningful tests remain central to evaluating what an agent produces.

Benchmark Scores Need Their Own Review

Benchmarks influence expectations about how much work an agent can handle. Their construction deserves scrutiny too.

OpenAI’s February review of SWE-bench Verified identified problems with tests and concerns about contamination. Its reported 59.4% defect figure applied to a selected set of 138 difficult tasks. It did not describe the entire benchmark.

That distinction changes what can reasonably be concluded from the number.

When comparing agents, teams should examine what the benchmark tasks represent, how success is checked, and whether the evaluation resembles their own work. A leaderboard cannot answer questions its underlying tests were not designed to measure.

Evaluate the Entire Change, From Request to Stable Use

The practical response is to measure and control the whole engineering workflow.

Track total human time, elapsed delivery time, review burden, rework, incidents, and user adoption. Code volume and token throughput can help explain activity and operating costs, but they cannot establish whether the organization achieved its objective.

Support review with independent checks: executable contracts, representative tests, access limits, staged releases, and observable production behavior. Adjust the depth of review to the consequences of failure.

Keep ownership explicit. Teams should be able to explain important system invariants, identify who owns a service, and recover when the automated workflow fails.

The evidence supports a substantial change in how engineering work gets performed. It also shows why the value of that change has to be measured beyond implementation. For software engineers and the organizations employing them, the useful result is a change that reaches users, behaves as intended, and remains maintainable after the agent finishes.