DORA’s latest research on AI impact

Key takeaways

There is often a discrepancy between leaders' perceptions of AI's impact and the experiences of individual developers.
Measuring productivity in the context of AI requires rigor and sensitivity to how survey questions are framed.
A nuanced understanding of 'time' is crucial when assessing productivity in software development.
The preliminary findings suggest that while AI can boost individual productivity, this may not always translate into team performance or throughput.
Organizations should focus on aligning productivity measures with their larger goals while leveraging AI.

Summary

Measuring Productivity with DX Core 4

The episode kicks off with the introduction of the DX Core 4, a productivity measurement framework developed from extensive research and practical experiences with prominent companies like Dropbox and Etsy. Listeners are encouraged to explore how this framework can assist in evaluating productivity in the context of AI's impact on software development.

AI's Discrepant Impact on Leaders and Developers

Derek discusses the differences in perception regarding AI's benefits. While leaders believe that AI enhances productivity significantly, individual developers often feel that the gains are overstated. This gap in understanding reveals the challenges of navigating AI's integration into workflows.

Insights from DORA's GenAI Report

The conversation dives into the methodology behind the latest DORA report, emphasizing how survey crafts need to evolve to accommodate AI's nuances. Particular attention is given to how survey questions are developed, iterated, and tested for clarity and precision.

The Complex Nature of Measuring Time

The podcast conducts a thorough examination of how time spent on different tasks is measured, highlighting both the percentage of time allocated to toil versus valuable work. The complexities and subjectiveness of these measurements are noted, especially in recognizing what constitutes valuable work.

Productivity Gains and Their Realities

Derek elaborates on the findings that a 25% increase in AI adoption can correlate with a 2.1% increase in productivity. This data prompts a broader discussion regarding the importance of looking beyond individual metrics to understand the potential corporate-wide effects of AI.

Navigating Decreased Throughput and Stability

Surprisingly, the report also indicates a slight decrease in software delivery throughput and stability with increased AI usage. Various interpretations of this outcome are explored, including the learning curve associated with adopting new technologies.

Recommendations for Measuring AI Impact

The episode wraps up with practical advice for leaders on aligning metrics with organizational goals. The emphasis is on creating an effective feedback loop that incorporates data analysis into actionable strategies.

Show notes

In this episode, Abi Noda speaks with Derek DeBellis, lead researcher at Google’s DORA team, about their latest report on generative AI’s impact on software productivity.

They dive into how the survey was built, what it reveals about developer time and “flow,” and the surprising gap between individual and team outcomes. Derek also shares practical advice for leaders on measuring AI impact and aligning metrics with organizational goals.

Where to find Derek DeBellis:

• LinkedIn: https://www.linkedin.com/in/derekdebellis/

Where to find Abi Noda:

• LinkedIn: https://www.linkedin.com/in/abinoda

In this episode, we cover:

(00:00) Intro: DORA’s new Impact of Gen AI report

(03:24) The methodology used to put together the surveys DORA used for the report

(06:44) An example of how a single word can throw off a question

(07:59) How DORA measures flow

(10:38) The two ways time was measured in the recent survey

(14:30) An overview of experiential surveying

(16:14) Why DORA asks about time

(19:50) Why Derek calls survey results ‘observational data’

(21:49) Interesting findings from the report

(24:17) DORA’s definition of productivity

(26:22) Why a 2.1% increase in individual productivity is significant

(30:00) The report’s findings on decreased team delivery throughput and stability

(32:40) Tips for measuring AI’s impact on productivity

(38:20) Wrap up: understanding the data

Referenced: