Why AI evals are the hottest new skill for product builders | Hamel Husain & Shreya Shankar (creators of the #1 eval course) · Lenny's Podcast: Product | Career | Growth summary

Key takeaways

Evaluations (evals) are crucial for systematically improving AI applications.
Human involvement in error analysis is essential; AI should not completely replace it.
Evals can be implemented with minimal time investment once set up.
Understanding the debate around 'vibes' versus structured evals can enhance product reliability.
Continuous learning and adaptation are vital for product success in AI.

Summary

Introduction to Evals

In this episode, Hamel Husain and Shreya Shankar, educators of the world's leading course on AI evals, explain the importance of evals for AI product builders. They emphasize that evals are essential for systematically measuring and improving AI applications, which have skyrocketed in their relevance due to the rise of AI technologies. Both guests highlight that even though evals have gained traction recently, they were previously considered obscure but are now vital for any AI product team.

Understanding Evals

Evals, short for evaluations, are outlined as tools for systematically analyzing and enhancing AI applications. Hamel explains that evals are foundational for ensuring that AI applications operate effectively, using data analysis techniques to identify errors and improve performance. Shreya provides context around common misconceptions, particularly the belief that AI can completely automate the evaluation process.

Step-by-Step Guide to Evals

The duo walks through the practical steps involved in developing an effective eval, referencing specific examples such as a property management AI. They explain error analysis, open coding, and axial coding—methods for categorizing and synthesizing errors found in logs. The conversation highlights the importance of human involvement in error analysis while advocating for the use of LLMs as automated evaluators for certain metrics.

The Great Debate: Evals vs. Vibes

A significant portion of the conversation explores the ongoing debate between 'vibes'—a more subjective, human-centered approach to evaluating AI applications—and systematic evals. Hamel and Shreya argue for the value of structured methodologies while acknowledging the arguments against them prevalent in some tech communities. They address common critiques and emphasize the role of error tracking in AI development, especially when quality assurance is paramount.

The Importance of Continuous Learning and Adaptation

Finally, the episode emphasizes that building effective evals is an ongoing process that benefits from constant iteration and learning. They share tips for implementing evals in a time-efficient manner, urging product managers to dedicate at least 30 minutes weekly to the process once initial setup is done. They highlight the importance of tracking metrics to achieve sustainable product quality and growth.

Show notes

Hamel Husain and Shreya Shankar teach the world’s most popular course on AI evals and have trained over 2,000 PMs and engineers (including many teams at OpenAI and Anthropic). In this conversation, they demystify the process of developing effective evals, walk through real examples, and share practical techniques that’ll help you improve your AI product.

What you’ll learn:

1. WTF evals are

2. Why they’ve become the most important new skill for AI product builders

3. A step-by-step walkthrough of how to create an effective eval

4. A deep dive into error analysis, open coding, and axial coding

5. Code-based evals vs. LLM-as-judge

6. The most common pitfalls and how to avoid them

7. Practical tips for implementing evals with minimal time investment (30 minutes per week after initial setup)

8. Insight into the debate between “vibes” and systematic evals

—

Brought to you by:

Fin—The #1 AI agent for customer service

Dscout—The UX platform to capture insights at every stage: from ideation to production

Mercury—The art of simplified finances

—

Where to find Shreya Shankar

• X: https://x.com/sh_reya

• LinkedIn: https://www.linkedin.com/in/shrshnk/

• Website: https://www.sh-reya.com/

• Maven course: https://bit.ly/4myp27m

—

Where to find Hamel Husain

• X: https://x.com/HamelHusain

• LinkedIn: https://www.linkedin.com/in/hamelhusain/

• Website: https://hamel.dev/

• Maven course: https://bit.ly/4myp27m

—

In this episode, we cover:

(00:00) Introduction to Hamel and Shreya

(04:57) What are evals?

(09:56) Demo: Examining real traces from a property management AI assistant

(16:51) Writing notes on errors

(23:54) Why LLMs can’t replace humans in the initial error analysis

(25:16) The concept of a “benevolent dictator” in the eval process

(28:07) Theoretical saturation: when to stop

(31:39) Using axial codes to help categorize and synthesize error notes

(44:39) The results

(46:06) Building an LLM-as-judge to evaluate specific failure modes

(48:31) The difference between code-based evals and LLM-as-judge

(52:10) Example: LLM-as-judge

(54:45) Testing your LLM judge against human judgment

(01:00:51) Why evals are the new PRDs for AI products

(01:05:09) How many evals you actually need

(01:07:41) What comes after evals

(01:09:57) The great evals debate

(1:15:15) Why dogfooding isn’t enough for most AI products

(01:18:23) OpenAI’s Statsig acquisition

(1:23:02) The Claude Code controversy and the importance of context

(01:24:13) Common misconceptions around evals

(1:22:28) Tips and tricks for implementing evals effectively

(1:30:37) The time investment

(1:33:38) Overview of their comprehensive evals course

(1:37:57) Lightning round and final thoughts

—

LLM Log Open Codes Analysis Prompt:

Please analyze the following CSV file. There is a metadata field which has an nested field called z_note that contains open codes for analysis of LLM logs that we are conducting. Please extract all of the different open codes. From the _note field, propose 5-6 categories that we can create axial codes from.

—

Referenced:

• Building eval systems that improve your AI product: https://www.lennysnewsletter.com/p/building-eval-systems-that-improve

• Mercor: https://mercor.com/

• Brendan Foody on LinkedIn: https://www.linkedin.com/in/brendan-foody-2995ab10b

• Nurture Boss: https://nurtureboss.io/

• Braintrust: https://www.braintrust.dev/

• Andrew Ng on X: https://x.com/andrewyng

• Carrying Out Error Analysis: https://www.youtube.com/watch?v=JoAxZsdw_3w

• Julius AI: https://julius.ai/

• Brendan Foody on X—“evals are the new PRDs”: https://x.com/BrendanFoody/status/1939764763485171948

• Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences: https://dl.acm.org/doi/abs/10.1145/3654777.3676450

• Lenny’s post on X about evals: https://x.com/lennysan/status/1909636749103599729

• Statsig: https://statsig.com/

• Claude Code: https://www.anthropic.com/claude-code

• Cursor: https://cursor.com/

• Occam’s razor: https://en.wikipedia.org/wiki/Occam%27s_razor

• Frozen: https://www.imdb.com/title/tt2294629/

• The Wire on HBO: https://en.wikipedia.org/wiki/The_Wire

—

Recommended books:

• Pachinko: https://www.amazon.com/Pachinko-National-Book-Award-Finalist/dp/1455563935

• Apple in China: The Capture of the World’s Greatest Company: https://www.amazon.com/Apple-China-Capture-Greatest-Company/dp/1668053373/

• Machine Learning: https://www.amazon.com/Machine-Learning-Tom-M-Mitchell/dp/1259096955