Measuring AI code assistants and agents with the AI Measurement Framework

Key takeaways

AI is a valuable tool for enhancing developer experiences, but it isn't a one-size-fits-all solution.
Adopting a structured measurement approach with the AI Measurement Framework is critical for understanding AI's impact on productivity.
Flawed metrics such as acceptance rates may misrepresent the true value of AI-generated code, necessitating a broader evaluation of quality and maintainability.
Use a combination of methods such as API data, surveys, and experience sampling to gather comprehensive insights on AI tool adoption.
Redefine the notion of who can be considered a developer in the context of AI contributions.
Monitor long-term effects of AI tools on code quality and maintainability to avoid potential pitfalls like technical debt.
Clear communication and supportive environments are essential for successfully integrating AI tools into development workflows.

Summary

The Role of AI in Developer Productivity

In the age of AI, organizations must understand how to measure developer productivity effectively. The speakers stress that while AI is a helpful tool for developers, it is not a magic solution that will drastically cut development time. There's potential for AI to improve organizational outcomes by enhancing developer experiences, but the challenges of measuring this impact remain significant.

The AI Measurement Framework

Laura and Abi introduce the AI Measurement Framework, designed to track AI adoption and its effectiveness through established metrics. They emphasize the importance of using this framework in practice to provide clarity on how AI is influencing developer productivity, particularly in regards to adoption rates, quality, and cost associated with AI tools.

Flaws in Metrics: Acceptance Rate

The hosts delve into the pitfalls of relying on flawed metrics such as acceptance rates to measure AI's contribution to coding. They argue acceptance rates may not accurately reflect the quality or effectiveness of AI-generated code, and suggest evaluating other factors like code maintainability and developer satisfaction.

Gathering Measurement Data

Discussing methods for gathering relevant data, the podcast highlights three main approaches: utilizing APIs from AI tools, conducting periodic surveys to measure developer experience, and employing experience sampling for real-time feedback during coding tasks. Each method provides invaluable insights that inform organizations about the impact of AI tools.

Agentic Workflows and New Definitions of Developer

As organizations begin to employ agentic AI tools that can autonomously generate code, there is a need to redefine what constitutes a developer. The speakers suggest encompassing a broader range of contributors who may not traditionally fall under the developer role but can now engage with code due to AI-enabled capabilities.

Long-Term Impact and Second-Order Effects

The discussion touches on the second-order effects of increased AI adoption, such as potential technical debt and impacts on code review processes. The importance of monitoring long-term effects beyond immediate productivity gains is stressed, with recommendations to utilize comprehensive metrics to track overall quality and maintainability over time.

Final Thoughts on AI Integration

In conclusion, the podcast stresses the importance of a data-driven approach to integrating AI in software development. Organizations must communicate clearly about the use of data and ensure employees feel supported in their use of new tools, enabling a smooth transition into the AI era.

Show notes

In this episode of Engineering Enablement, DX CTOLaura Tacho and CEO Abi Noda break down how to measure developer productivity in the age of AI using DX’s AI Measurement Framework. Drawing on research with industry leaders, vendors, and hundreds of organizations, they explain how to move beyond vendor hype and headlines to make data-driven decisions about AI adoption.

They cover why some fundamentals of productivity measurement remain constant, the pitfalls of over-relying on flawed metrics like acceptance rate, and how to track AI’s real impact across utilization, quality, and cost. The conversation also explores measuring agentic workflows, expanding the definition of “developer” to include new AI-enabled contributors, and avoiding second-order effects like technical debt and slowed PR throughput.

Whether you’re rolling out AI coding tools, experimenting with autonomous agents, or just trying to separate signal from noise, this episode offers a practical roadmap for understanding AI’s role in your organization—and ensuring it delivers sustainable, long-term gains.

Where to find Laura Tacho:

• X: https://x.com/rhein_wein

• LinkedIn: https://www.linkedin.com/in/lauratacho/

• Website: https://lauratacho.com/

Where to find Abi Noda:

• LinkedIn: https://www.linkedin.com/in/abinoda

• Substack: https://substack.com/@abinoda

In this episode, we cover:

(00:00) Intro

(01:26) The challenge of measuring developer productivity in the AI age

(04:17) Measuring productivity in the AI era — what stays the same and what changes

(07:25) How to use DX’s AI Measurement Framework

(13:10) Measuring AI’s true impact from adoption rates to long-term quality and maintainability

(16:31) Why acceptance rate is flawed — and DX’s approach to tracking AI-authored code

(18:25) Three ways to gather measurement data

(21:55) How Google measures time savings and why self-reported data is misleading

(24:25) How to measure agentic workflows and a case for expanding the definition of developer

(28:50) A case for not overemphasizing AI’s role

(30:31) Measuring second-order effects

(32:26) Audience Q&A: applying metrics in practice

(36:45) Wrap up: best practices for rollout and communication

Referenced: