The current impact of AI on engineering velocity: What 400 companies are seeing (Abi Noda & Brian Houck)

This section introduces the podcast's setting at the DX Annual event, with host Justin Riyak welcoming guests Abi Noda and Brian Houck. Abi Noda shares the motivation behind DX's recent research into AI’s impact on engineering velocity, driven by the gap between industry hype and real-world observations. DX’s study aims to understand what organizations are actually experiencing as they scale AI adoption, especially after observing modest gains in throughput during the past year.

Abi Noda details how DX approached the study, selecting companies with mature AI adoption—defined as over 75% of developers actively using AI tools—and with more than 100 engineers. They excluded companies affected by M&A or IPO events to avoid regulatory confounders. The research emphasizes longitudinal data analysis at the organizational level to better understand causality, as opposed to cross-sectional analysis, to gauge the true impact of AI on engineering throughput over time.

The research reveals that AI adoption has led to a modest increase in throughput—typically around 8-11%, or roughly 10-15%, far lower than the often-promised 10x gains. This discrepancy is attributed to various factors, including the realization that coding is only about 14% of developers' work, along with new bottlenecks introduced by automation, cultural resistance, and friction in adoption. The findings challenge the hype of dramatic productivity boosts and highlight the complexity of measuring true value.

The discussion covers why AI productivity gains tend to be lower than expectations. Follow-up interviews with developers suggest that much of their time isn't spent coding, which diminishes the impact of AI tools focused solely on code. Developers are often using saved time for reviewing, managing technical debt, or waiting, which impacts overall productivity metrics. Abi emphasizes that these nuanced realities explain the modest gains observed.

The podcast explores potential downside effects of increased velocity, such as quality degradation, rising costs, and the concept of 'false velocity'—where teams appear faster but fail to deliver real business value. Cultural challenges, like prioritizing speed over meaningful innovation, are discussed as crucial considerations for engineering leaders to avoid unintended long-term risks like technical and cognitive debt.

Guests recommend that organizations look beyond just improving coding efficiency. They suggest improving the entire Software Development Life Cycle (SDLC), enhancing developer experience, and exploring autonomous engineering ideas. Emphasis is placed on applying AI to optimizing workflows, documentation, and creating more autonomous agents that augment human work in parallel, rather than replacing it.

The conversation emphasizes the human costs of AI adoption, including cognitive debt—the mental load and understanding gaps in complex systems. Influenced by Margaret Ann Storrey’s research, the discussion considers how AI can lead to a loss of human mastery over systems and the importance of measuring and managing these human factors to sustain long-term quality and innovation.

Abi Noda discusses how DX is updating its measurement frameworks to better evaluate both human and agent productivity. New metrics include tracking AI agent effectiveness and user feedback, as well as distinguishing between acceleration of human effort versus augmentation through autonomous agents. This approach aims to provide a clearer picture of real productivity and ROI in AI-enhanced engineering.