Inside the DevEx Metrics Compass with André Meyer
· Summarized
Key takeaways
- AI adoption, acceptance rates, lines of code, token spend, and pull-request counts show activity, not whether AI created customer or business value.
- DevEx measurement should begin with a specific objective and be tailored to the organization’s friction, initiatives, and context rather than copied from a universal metric set.
- Collaboration is underrepresented in current dashboards even though developers may spend 30–50% of their time communicating, coordinating, and attending meetings.
- A useful measurement system combines qualitative feedback with telemetry across value, quality, experience, and activity, examined as trends against a baseline.
- Metrics should be reviewed and sometimes retired when they are no longer monitored, actionable, relevant, or trusted, while a small set of guardrail measures remains in place.
- Good developer days come from meaningful progress, efficient use of time, manageable interruptions, and the ability to retain ownership of important technical decisions even when AI writes code.
- Teams can improve workdays by batching collaboration and focus time and by establishing clear AI-assisted pull-request responsibilities, risk thresholds, and review expectations.
Summary
Why AI dashboards rarely prove value
Brian Houck opens by asking André Meyer why leaders with AI dashboards still struggle to answer whether their investment helped. Meyer says most dashboards measure adoption and activity—daily active users, acceptance rates, lines of code, token spend, throughput, pull requests, and deployment frequency—because these are easy to collect. They reveal that AI is being used, not whether it creates value or delivers return on investment. Since producing code is now cheap, output metrics inflate and are easy to game. Meyer notes that qualitative measures such as satisfaction, friction, and reported time savings remain useful complements.
The DevEx Metrics Compass maps a fragmented field
Meyer explains that he and collaborators Thomas Fritz, Gail Murphy, and Patrick Meyer collected more than 120 measures from research and from roughly 40 companies. The resulting DevEx Metrics Compass helps teams navigate overlapping frameworks such as SPACE, DORA, and EngFlow, compare their practices with similar companies, and receive an actionable report suggesting where to begin. Houck praises the visual tool’s breadth. Meyer cautions that the compass is not a prescription to adopt every metric; teams should first define their measurement goal, then filter the landscape to metrics relevant to that objective.
Collaboration is the biggest blind spot
Meyer says most published DevEx dashboards measure development activity while barely measuring collaboration, even though developers may spend 30–50% of their time communicating and coordinating. He cannot name a mature collaboration metric, but suggests measuring the balance between collaborative and individual work, meeting quality, whether meetings are scattered across the day, focus, interruptions, and after-hours messages. He considers after-work email or chat a possible sign that something is wrong. Houck adds that meeting clustering can predict completed work better than simply counting meetings.
Google and Microsoft show why metrics need context
Although Google and Microsoft each publicly shared between 20 and 24 metrics, only about six overlapped. Meyer says this comparison is limited to what each company chose to disclose, but the difference supports tailoring measurement to a company’s goals rather than copying a fixed industry set. New teams can begin with generic measures; more mature teams should define metrics around a specific friction and use them to test whether an intervention improves developers’ work. He acknowledges that the Compass itself may overwhelm users, so leaders should select metrics only after defining the outcome they want.
Public reporting gaps are checked against interviews and research
Meyer says the Compass’s apparent gaps, including the lack of collaboration measures, are probably real, although companies may omit sensitive subjects such as burnout and attrition or disclose only metrics mature enough to share. The team interviewed more than 20 engineering leaders, who largely confirmed that the most common public metrics are also the most actionable. Research papers show more attention to cognitive load, stress, and burnout than industry reporting does. Anonymous Compass usage may reveal further demand: early users searched for measures of joy and meaning, concepts rarely found in existing publications.
Outcome and flow metrics are difficult to act on
Meyer says a metric’s usefulness depends on its level of abstraction and how narrowly it is analyzed. Flow can guide a small experiment, such as grouping meetings instead of scattering them, but becomes difficult to act on across a large organization because people’s conditions for focus differ. Value-based measures are especially rare: companies often fall back on deployment frequency, customer-reported bugs, promoter scores, AI spending, or revenue because they cannot connect engineering, customer-support, project, and financial data. Meyer suggests measuring developer time spent on new features alongside adoption, satisfaction, retention, and time from idea to customer delivery.
Metrics should be retired, but guardrails should remain
Meyer rejects the idea that DevEx measurement is a one-time project. Problems, tools, and teams change, while continually adding metrics creates clutter and increases the risk of gaming or turning a measure into a harmful target under Goodhart’s law. Teams should ask whether anyone monitors a metric, whether it influenced a recent decision, whether it is actionable, whether it solved the problem it was introduced to track, and whether it remains trustworthy. He still favors a stable baseline of guardrail measures reviewed over time, with a yearly check for metrics that no longer serve a purpose.
A metric hierarchy keeps dashboards usable
When Houck asks how many measures are enough, Meyer says there is no universal number because measurement maturity and the ability to act vary. Houck offers his experience: three to five top-level metrics can provide checks and balances, supported by dozens of lower-level diagnostic measures used to investigate patterns and guide initiatives. Meyer agrees that a hierarchy makes sense: high-level measures should be durable, while telemetry, interviews, and self-reports provide detail beneath them. He distinguishes this from displaying every SPACE dimension or every available metric at the same level.
AI impact requires objectives, qualitative feedback, and trends
For measuring AI beyond activity counts, Meyer recommends first defining whether the goal is better DevEx, more feature value, or higher quality. Teams should combine surveys covering satisfaction, ease of delivery, time savings, and productivity with serious analysis of reported friction. Telemetry should span multiple dimensions rather than only output or pull requests, and should be examined as trends against a baseline. Quality remains difficult, but rollback rates and post-release failures can help. Houck notes that across hundreds of organizations, change-failure rate has not clearly improved as AI adoption rises.
Good workdays depend on value, progress, and protected focus
Meyer revisits the “Today Was a Good Day” study, conducted through observation and later tool-usage analysis. Developers described good days as days when they created something valuable, made meaningful progress, used time efficiently, faced few blockers or interruptions, and had more time to code—about 30% of the day in the study. Observation exposed extensive context switching and showed why telemetry alone is insufficient: switching from an IDE to documentation may be productive, while rebuilding a mental model after a disruptive task change is costly. Waiting for a build can also be a useful moment for email or review.
AI can remove coding toil without removing ownership
Meyer says the rewarding part of development is not merely typing code; it is deciding what to build, designing a solution, and creating something useful. AI can preserve that satisfaction if developers remain in the driver’s seat and understand the result. He worries when AI produces larger pull requests that developers cannot comprehend, increasing cognitive and intent debt. Delegation is therefore a judgment call: developers should decide what to hand off while retaining key decisions and enough understanding of the code. Houck connects this tension to the productivity-experience paradox between increased output and developer sentiment.
Teams should batch collaboration and define AI review etiquette
Meyer says meetings and interruptions are consistently reported as the main impediment, but collaboration still occupied roughly one-third of even good workdays. The problem is usually fragmented timing, poor preparation, and weak moderation—not collaboration itself. He prefers team-wide schedules that group focus blocks, stand-ups, and collaboration rather than relying on isolated no-meeting days. His second recommendation is an AI-specific PR etiquette: authors must understand and own generated changes, identify risk or confidence, use automated review for low-risk work where appropriate, and require human review for high-risk or low-confidence changes. Teams should also agree on reviewable PR sizes and turnaround times.
Sponsors
- CAFES framework · The host promoted CAFE(S): Your Agent Is Only as Good as Its Context, a research paper from researchers at DX, Capital One, GitHub, the University of Victoria, and Google. It examines context quality as a bottleneck in agentic workflows and introduces five properties intended to make AI agents more effective.
Show notes
Are engineering teams measuring the impact of AI, or just how much it's being used?
In this episode of Engineering Enablement, host Brian Houck talks with researcher André Meyer about his DevEx Metrics Compass, why traditional productivity metrics often miss what matters, and how to measure the value AI actually delivers.
They also revisit André's Today Was a Good Day study to explore what makes developers' work rewarding, how AI is changing that experience, and practical ways to balance focused work, collaboration, and growing code review demands.
Where to find André Meyer-Baron:
• LinkedIn: https://www.linkedin.com/in/andrenmeyer
• Website: https://andre-meyer.ch and www.devexcompass.com
Where to find Brian Houck:
• LinkedIn: https://www.linkedin.com/in/brianhouck
In this episode, we cover:
(00:00) Intro
(02:42) Why AI dashboards fail to capture actual value
(05:08) What the DevEx Metrics Compass is and why it was created
(06:58) Why measuring collaboration matters
(09:47) Determining the right metrics for your organization
(12:03) Distinguishing measurement gaps from gaps in public reporting
(14:35) Why some metrics are harder to act on at scale
(18:11) Why customer and business value are difficult to measure
(20:03) How to know when it's time to retire a metric
(24:20) Hierarchies of metrics
(27:31) How to measure AI's impact beyond activity metrics
(30:20) André's Good Day study: What makes a good workday for developers
(31:56) What observing developers revealed about context switching
(34:28) What makes developers' work rewarding in the age of AI
(36:27) The tradeoffs of delegating coding to AI
(38:01) Balancing collaboration with focused work
(42:11) Two ways to improve developers' workdays
Referenced:
• CAFE(S): Your Agent Is Only as Good as Its Context
• Too Many DevEx Metrics, Too Little Guidance
• Today was a Good Day: The Daily Life of Software Developers
• From Technical Debt to Cognitive and Intent Debt: Rethinking Software Health in the Age of AI
Follow this show on AudioBrevity and new episodes get summarized for you. Create a free account