Measuring the impact of AI on software engineering – with Laura Tacho · The Pragmatic Engineer summary

Key takeaways

Developers are often left dissatisfied as AI tools take over enjoyable coding tasks, leaving them with less fulfilling work.
Media plays a significant role in shaping perceptions around AI, often creating sensationalized narratives that misrepresent reality.
Effective measurement of AI's impact on developer productivity requires a framework that goes beyond traditional metrics to include utilization, impact, and cost.
Structured rollouts and thorough training are crucial for successful AI tool adoption.
As AI tools evolve, the focus should not just be on speed but also on maintaining quality and reducing cognitive load for developers.

Summary

Understanding AI's Impact on Developer Satisfaction

Laura Tacho shares insights into the paradox of AI in software engineering—while AI tools can save developers time, many express dissatisfaction as the tools often take over the tasks they enjoy. Developers find themselves left with the 'toil' tasks like meetings and administrative work instead of creative coding.

The Role of Media in Shaping Perceptions of AI

Laura comments on how sensationalist media headlines create false narratives around AI's impact, leading to misunderstandings about job security and the true capabilities of AI tools. She stresses the importance of media and data literacy for developers and engineering leaders.

Real Metrics for Measuring AI Effectiveness

The conversation emphasizes the inadequacies of traditional metrics like lines of code and acceptance rates to gauge productivity. Instead, Laura introduces the DX AI Measurement Framework, which encompasses utilization, impact, and cost, facilitating a comprehensive evaluation of AI tools.

Case Studies in AI Rollouts

Insights from case studies, particularly those from Booking.com and WorkHuman, illustrate effective strategies for AI tool adoption. Both companies found that structured rollouts and focusing on developer experience significantly enhance the benefits of AI.

The Future of Developer Workflows and Productivity

As AI tools evolve, Laura suggests that they could transform coding workflows, encouraging faster testing and reducing migration complexities. However, she warns of potential issues like feature bloat and stresses that improvements need to focus on quality alongside speed.

The Intersection of AI and Organizational Change

The discussion explores how AI adoption should be seen as an organizational challenge, requiring systematic changes in culture, training, and how success is measured.

Show notes

Supported by Our Partners

•⁠ Statsig ⁠ — ⁠ The unified platform for flags, analytics, experiments, and more.

• Graphite — The AI developer productivity platform.

—

There’s no shortage of bold claims about AI and developer productivity, but how do you separate signal from noise?

In this episode of The Pragmatic Engineer, I’m joined by Laura Tacho, CTO at DX, to cut through the hype and share how well (or not) AI tools are actually working inside engineering orgs. Laura shares insights from DX’s research across 180+ companies, including surprising findings about where developers save the most time, why devs don’t use AI at all, and what kinds of rollouts lead to meaningful impact.

We also discuss:

• The problem with oversimplified AI headlines and how to think more critically about them

• An overview of the DX AI Measurement framework

• Learnings from Booking.com’s AI tool rollout

• Common reasons developers aren’t using AI tools

• Why using AI tools sometimes decreases developer satisfaction

• Surprising results from DX’s 180+ company study

• How AI-generated documentation differs from human-written docs

• Why measuring developer experience before rolling out AI is essential

• Why Laura thinks roadmaps are on their way out

• And much more!

—

Timestamps

(00:00) Intro

(01:23) Laura’s take on AI overhyped headlines

(10:46) Common questions Laura gets about AI implementation

(11:49) How to measure AI’s impact

(15:12) Why acceptance rate and lines of code are not sufficient measures of productivity

(18:03) The Booking.com case study

(20:37) Why some employees are not using AI

(24:20) What developers are actually saving time on

(29:14) What happens with the time savings

(31:10) The surprising results from the DORA report on AI in engineering

(33:44) A hypothesis around AI and flow state and the importance of talking to developers

(35:59) What’s working in AI architecture

(42:22) Learnings from WorkHuman’s adoption of Copilot

(47:00) Consumption-based pricing, and the difficulty of allocating resources to AI

(52:01) What DX Core 4 measures

(55:32) The best outcomes of implementing AI

(58:56) Why highly regulated industries are having the best results with AI rollout

(1:00:30) Indeed’s structured AI rollout

(1:04:22) Why migrations might be a good use case for AI (and a tip for doing it!)

(1:07:30) Advice for engineering leads looking to get better at AI tooling and implementation

(1:08:49) Rapid fire round

—

The Pragmatic Engineer deepdives relevant for this episode:

• AI Engineering in the real world

• Measuring software engineering productivity

• The AI Engineering stack

• A new way to measure developer productivity – from the creators of DORA and SPACE

—

See the transcript and other references from the episode at ⁠⁠https://newsletter.pragmaticengineer.com/podcast⁠⁠

—

Production and marketing by ⁠⁠⁠⁠⁠⁠⁠⁠https://penname.co/⁠⁠⁠⁠⁠⁠⁠⁠. For inquiries about sponsoring the podcast, email podcast@pragmaticengineer.com.

Get full access to The Pragmatic Engineer at newsletter.pragmaticengineer.com/subscribe

Measuring the impact of AI on software engineering – with Laura Tacho