Uber’s journey of measuring AI impact on developer productivity

The podcast explores Uber's journey of developing metrics to measure AI's impact on developer productivity across four distinct eras: pre-AI, AI-assisted, agentic, and the emerging software factory. Ty Smith and Abhishek Tibrewal discuss how internal metrics have evolved to keep pace with rapid technological changes, such as the shift from manual code writing to fully autonomous agent work. They emphasize that traditional metrics like PR throughput and developer hours are now insufficient to capture true productivity, especially as AI tools are integrated into the SDLC, prompting Uber to develop new frameworks like feature velocity.

Throughout their journey, Uber faced significant challenges in quantifying AI's benefits. Early qualitative methods relied on surveys and experience sampling, but these were supplemented (and at times replaced) by telemetry data. As AI tools scaled, traditional activity-based metrics like PR count and suggestion frequency became unreliable indicators of value because activity often inflated without corresponding to real progress. The hosts shared lessons on unlearning assumptions, such as equating activity with output, and highlighted the importance of outcome-based metrics that tie directly to business value, like feature velocity.

Uber introduced a new KPI called 'feature velocity,' aimed at measuring the number of features shipped over time—independent of who authored the PR, whether human or AI. This metric, along with PR classification and flow efficiency, is designed to more accurately reflect meaningful product progress. The podcast details how Uber is working towards an 'AI native engineer' model, measuring the breadth, depth, and consistency of agent usage across the SDLC. These efforts aim to better understand if AI truly accelerates the product roadmap and adds substantial value.

The hosts acknowledge that as the industry moves towards more automated, agent-driven development, many questions remain unanswered. These include how to measure judgment when most of the code is written by AI, how to track tech debt accumulation, and how to attribute value when tasks are autonomously executed. They emphasize the importance of building adaptable, outcome-focused metrics and recognize that the 'software factory' era remains an open frontier for research and measurement innovation.