Running data-driven evaluations of AI engineering tools

Key takeaways

Robust evaluation frameworks are crucial for selecting AI engineering tools.
Regularly revisiting and re-evaluating incumbent tools is essential in rapidly changing tech landscapes.
Cohort diversity in trials enhances data accuracy and leads to better decision-making.
Support and training are integral to the successful deployment of new tools.
Establishing clear, data-driven goals is necessary for evaluating tool effectiveness.

Summary

Evaluating AI Engineering Tools

The episode emphasizes the rapid evolution of AI engineering tools and the need for rigorous evaluations to ensure effective integration into existing workflows. Laura Tacho and Abi Noda discuss how tools should be evaluated based on real-world metrics and data rather than superficial measures.

Challenges and Considerations in Tool Selection

Key challenges include the fear of missing out on new capabilities and the risk of sticking too long with incumbent tools. The hosts highlight the importance of evaluating not just new tools, but also reassessing currently used tools to ensure they meet evolving organizational needs.

Structured Evaluations and Cohort Selection

The discussion dives into the importance of structuring evaluations around clear research questions and selecting diverse cohorts for trials. This ensures robust and reliable data collection, which is critical for making informed purchasing decisions.

Support, Enablement, and Measuring Impact

Success isn't just about the tools themselves, but also how well organizations support their teams through training and enablement. The AI Measurement Framework is introduced as a vital tool for measuring the impact of new AI technologies effectively.

Continuous Evaluation and Data-Driven Decisions

The hosts stress the importance of continuous measurement and establishing baselines for every evaluation phase. They suggest that organizations regularly revisit their tool selections to stay current and effective in a rapidly changing market.

Show notes

AI engineering tools are evolving fast. New coding assistants, debugging agents, and automation platforms emerge every month. Engineering leaders want to take advantage of these innovations while avoiding costly experiments that create more distraction than impact.

In this episode of the Engineering Enablement podcast, host Laura Tacho and Abi Noda outline a practical model for evaluating AI tools with data. They explain how to shortlist tools by use case, run trials that mirror real development work, select representative cohorts, and ensure consistent support and enablement. They also highlight why baselines and frameworks like DX’s Core 4 and the AI Measurement Framework are essential for measuring impact.

Where to find Laura Tacho:

• LinkedIn: https://www.linkedin.com/in/lauratacho/

• X: https://x.com/rhein_wein

• Website: https://lauratacho.com/

• Laura’s course (Measuring Engineering Performance and AI Impact): https://lauratacho.com/developer-productivity-metrics-course

Where to find Abi Noda:

• LinkedIn: https://www.linkedin.com/in/abinoda

• Substack: https://substack.com/@abinoda

In this episode, we cover:

(00:00) Intro: Running a data-driven evaluation of AI tools

(02:36) Challenges in evaluating AI tools

(06:11) How often to reevaluate AI tools

(07:02) Incumbent tools vs challenger tools

(07:40) Why organizations need disciplined evaluations before rolling out tools

(09:28) How to size your tool shortlist based on developer population

(12:44) Why tools must be grouped by use case and interaction mode

(13:30) How to structure trials around a clear research question

(16:45) Best practices for selecting trial participants

(19:22) Why support and enablement are essential for success

(21:10) How to choose the right duration for evaluations

(22:52) How to measure impact using baselines and the AI Measurement Framework

(25:28) Key considerations for an AI tool evaluation

(28:52) Q&A: How reliable is self-reported time savings from AI tools?

(32:22) Q&A: Why not adopt multiple tools instead of choosing just one?

(33:27) Q&A: Tool performance differences and avoiding vendor lock-in

Referenced: