Why experts writing AI evals is creating the fastest-growing companies in history | Brendan Foody (CEO of Mercor) · Lenny's Podcast: Product | Career | Growth summary

Key takeaways

Evals are becoming the primary bottleneck for AI progress, serving as critical criteria for AI model success.
Mercor achieved unprecedented growth by focusing on high-caliber expert labor for AI labs, identifying a major market opportunity.
AI is transforming the labor market, with new roles emerging that leverage specific human skills and capabilities.
A long road lies ahead for developing AGI, with significant reliance on human expertise and evals.
The success of companies like Mercor hinges on key cultural values: can-do attitude, high standards, and a focus on output.

Summary

The Rise of Evals in AI

Brendan Foody discusses how the emergence of 'evals', or evaluations, is becoming a critical component in AI development. He explains that evals serve as the Product Requirement Document (PRD) for AI models, outlining the criteria for success. This evaluation process is essential for measuring how well AI can automate tasks that humans have traditionally performed, making it a burgeoning frontier for companies in the AI industry.

Mercor's Remarkable Growth Story

Brendan shares the astonishing growth trajectory of Mercor, which soared from $1M to a $500M revenue run rate in just 17 months, marking it as one of the fastest-growing companies in history. This growth is attributed to recognizing a pivotal market opportunity where AI labs require high-caliber experts to help improve their models. Brendan emphasizes that understanding customer needs and relentlessly focusing on product excellence were key to this success.

AI's Impact on the Labor Market

The conversation shifts towards the changing landscape of labor in an AI-driven economy. Brendan asserts that while AI might displace some jobs, it is also creating new opportunities, particularly in roles that require a nuanced understanding of AI evals. He discusses how traditional job roles are evolving, recommending that individuals invest in skills that will enhance their productivity alongside AI tools.

Future of AI and Consciousness

Brendan provides insights into the future of AI, expressing skepticism about the imminent arrival of AGI (Artificial General Intelligence) or superintelligence. He believes that while AI will improve significantly over the next decade, it will still rely heavily on human expertise for defining success and learning effective behaviors through evals.

Core Values Driving Mercor's Success

Brendan highlights three core values that have propelled Mercor to success: a can-do attitude, maintaining high standards in hiring, and fostering an intense work culture. He explains how these principles promote a focused environment where employees are motivated to meet ambitious goals and deliver exceptional outcomes.

Show notes

Brendan Foody is the CEO and co-founder of Mercor, the fastest-growing company in history to go from $1M to $500M in revenue (in just 17 months!). At 22, he is also the youngest American unicorn founder ever. Mercor works with 6 of the Magnificent 7 and all top 5 AI labs to help them hire experts to create evaluations and training data that improve their models. In this conversation, Brendan explains why evals have become the critical bottleneck for AI progress, how he discovered this massive opportunity, and what the future of work might look like in an AI-driven economy.

What you’ll learn:

1. Why evals are becoming the primary bottleneck for AI progress and what this means for AI startups

2. How Mercor grew to $500M revenue in 17 months (fastest in history)

3. Brendan’s meeting with xAI that changed his company’s trajectory

4. Which skills and jobs will remain most valuable as AI continues to advance (hint: jobs with “elastic” demand)

5. Why Brendan believes AGI and superintelligence are not happening anytime soon

6. The three unique core values that drove Mercor’s success

7. How Harvard Lampoon writers are making Claude funnier

—

Brought to you by:

WorkOS—Modern identity platform for B2B SaaS, free up to 1 million MAUs

Jira Product Discovery—Atlassian’s new prioritization and roadmapping tool built for product teams

Enterpret—Transform customer feedback into product growth

—

Transcript: https://www.lennysnewsletter.com/p/experts-writing-ai-evals-brendan-foody

—

My biggest takeaways (for paid newsletter subscribers): https://www.lennysnewsletter.com/i/173303790/my-biggest-takeaways-from-this-conversation

—

Where to find Brendan Foody:

• X: https://x.com/BrendanFoody

• LinkedIn: https://www.linkedin.com/in/brendan-foody-2995ab10b/

—

Where to find Lenny:

• Newsletter: https://www.lennysnewsletter.com

• X: https://twitter.com/lennysan

• LinkedIn: https://www.linkedin.com/in/lennyrachitsky/

—

In this episode, we cover:

(00:00) Introduction to Brendan Foody and Mercor

(05:38) The “era of evals”

(09:26) Understanding the AI training landscape

(17:10) The future of work and AI

(25:54) The evolution of labor markets

(29:55) Understanding how AI models are trained

(38:58) Building Mercor

(53:27) Lessons from past ventures

(56:55) The future of AI and model improvement

(01:00:41) His personal use of AI and final thoughts

—

References: https://www.lennysnewsletter.com/p/experts-writing-ai-evals-brendan-foody

—

Production and marketing by https://penname.co/. For inquiries about sponsoring the podcast, email podcast@lennyrachitsky.com.

Lenny may be an investor in the companies discussed.

To hear more, visit www.lennysnewsletter.com

Why experts writing AI evals is creating the fastest-growing companies in history | Brendan Foody (CEO of Mercor)