How Meta reduced diff authoring time by 40%

The episode begins with host Brian Houck discussing the centrality of code diffs at Meta, emphasizing their importance in daily engineering conversations and decision-making. The focus is on how the business wanted to understand the time spent on diffs, which led to the creation of the Diff Authoring Time metric. This metric measures the active time engineers spend creating, reviewing, or preparing diffs. Brian notes that diffs are considered the 'currency' of code development at Meta, and the metric was born from the need to quantify that activity. The conversation highlights the significance of measuring engineering activity in a company where the flow of change is continuous and critical to product development.

Moritz Beller explains that a diff is essentially a self-contained code change, similar to a pull request, including associated summaries and test plans. He details how Diff Authoring Time captures the active work involved in creating and reviewing these diffs, providing a more precise measure than simple averages, which previously obscured variability. Moritz points out that this metric enables Meta to run controlled experiments, akin to A/B testing, which was not feasible with crude previous metrics. He shares an example where improved memoization in React reduced diff authoring time by around 30%, illustrating the metric’s potential to evaluate internal tooling.

The discussion shifts to practical uses of the metric, such as guiding engineering tool improvements and informing leadership about development efficiency. Moritz highlights that Diff Authoring Time is not used as a performance metric but helps identify large gains from specific tool changes, like AI-enhanced features, which can lead to strategic investments. The metric also helps categorize diffs, distinguishing exploratory work or technical debt from regular feature development, thereby informing project prioritization. Brian emphasizes that Diff Authoring Time contributes to understanding system efficiency and how automation and tooling influence the overall productivity and flow of work.

The conversation explores how AI-driven tools have drastically reduced Diff Authoring Time—by over 40%—while increasing the number of diffs produced, indicating greater efficiency. Moritz discusses how developer time savings are shifting to other activities, such as gathering context or planning, which are less visible but crucial. They debate whether lines of code still hold value as a productivity measure, with Moritz suggesting a relationship exists between coding activity and perceived productivity. The episode also considers how AI might affect differences across experience levels, with initial findings showing senior developers adopting these tools more effectively, though this may evolve.

Moritz reflects on his earlier “Mind the Gap” study, which bridged quantitative activity data with developers' self-perceived productivity. He recalls that only a small percentage of the workday is spent actively coding, with other factors like interruptions and on-call duties influencing productivity perceptions. He discusses how modern telemetry, including sleep data from fitness trackers, could add depth to future research. The panel anticipates that new forms of workload measurement, such as analyzing collaboration and informal interactions, will become increasingly important, especially as AI changes what constitutes valuable developer work. They highlight the need for comprehensive metrics that incorporate human and interpersonal factors.

Closing the episode, Brian raises the challenge of measuring softer aspects like collaboration, argumentation, and informal communication—areas still poorly understood but vital for effective engineering. Moritz agrees and emphasizes the importance of understanding how developers interact and coordinate, especially as organizational complexities grow. They discuss potential future tools, like AI synthesizing meeting transcripts, to capture unseen effort. Both agree that understanding human dynamics and social interactions remains a frontier in developer productivity research. The episode ends with optimism about advancing measurement techniques to better support engineering teams in navigating a rapidly evolving landscape.