Research briefing with Brian Houck: Measuring AI agents and revisiting the Core 4

Justin Riyak introduces the podcast episode and highlights the evolution of the conversation around developer experience over nearly five years. He emphasizes how recent advancements, particularly in AI, have shifted the focus towards agent experience and the need to measure this emerging domain effectively. The episode sets the stage for a deep dive into the intersection of developer productivity, AI fluency, and the tools that support these efforts.

Brian Houck explains the concept of agent experience as an extension of developer experience, focusing on whether AI agents have the optimal environment to perform their tasks. He discusses how agent experience encompasses clarity of requirements, data accuracy, and the ability for humans and agents to collaborate effectively. The importance for engineering leaders to care about this area lies in the fact that poor agent setup can hinder human productivity, making it crucial to measure and improve the collaboration and conditions around AI agents.

The speakers explore ways to enhance agent experience by applying principles similar to those used in traditional developer experience—defining it, measuring it, and iterating improvements. They highlight key factors such as clear and unambiguous context, data validation, shared prompts, and the importance of platform guardrails. There is also a discussion about how improvements in agent experience benefit human developers and the significance of platform-level protections to ensure effective and predictable AI behavior.

The discussion delves into the complex relationship between developer experience and agent experience, emphasizing how they are interconnected and increasingly inseparable. They consider research findings indicating a paradox where AI makes developers feel more productive but also leads to a decline in overall developer experience. This highlights the need to measure and optimize both, as improvements in agent experience can positively influence overall productivity while avoiding negative impacts like burnout or decreased code quality.

Attention shifts to organizational readiness for AI and agents, emphasizing three layers: tooling, culture, and infrastructure. Cultural readiness, including leadership advocacy, training, and social proof, is identified as the most impactful for adoption. Infrastructure involves measurement and feedback loops to gauge progress. The conversation underscores the importance of managing expectations and fostering psychological safety, along with developing metrics such as documentation quality, PR size, token usage, and the innovative 'idea to customer' measure, to evaluate AI's impact effectively.

Brian Houck discusses how traditional metrics like PR throughput remain relevant but need reinterpretation in the AI context. He advocates for a focus on outcome-oriented outcomes—like the DX Core 4 (speed, effectiveness, quality, business impact)—and stresses that diagnostic metrics should adapt to new patterns of work and AI involvement. He warns against relying solely on input metrics like token counts, emphasizing triangulation with broader outcome and system health metrics.

The podcast explores the limitations of token usage as a metric, deeming it primarily a cost indicator rather than a reflection of productivity or quality. They propose focusing on higher-level metrics like 'idea to customer' to assess the true flow of value, advocating for cross-referencing cost metrics with outcome data. The conversation also emphasizes avoiding gaming of metrics like PR size and underscores the importance of holistic, multiple-metric approaches.

The hosts respond to questions about keeping documentation up to date, noting its crucial role in onboarding and efficiency, especially for AI. They highlight ongoing research to automate and improve documentation practices. The topic of psychological safety is discussed in relation to AI-induced layoffs and burnout, emphasizing the importance of transparency and leadership support to foster a healthy work environment. Brian shares upcoming research plans on agent experience, further exploring how organizations can optimize human-AI collaboration.