Sonnet 5 review: I ran 64 generations to find out if it's worth it

Key takeaways

The host created a comprehensive, repeatable benchmark system called How I AI Bench to evaluate frontier AI models across multiple tasks.
Building the benchmark involved using Claude Code to generate, score, and visually compare model outputs, integrating human vibe scores with automated judgments.
Surprising leaderboard results showed Gemini 3 Pro and Sonnet 5 performing highly, contrary to expectations and initial claims.
Subjective taste and individual judgment differ from automated and model-generated scores, highlighting the importance of human oversight.
The evaluation process is ongoing, with plans to refine tasks, incorporate personal taste, and develop industry-standard benchmarks for consistent model comparison.

Summary

Introduction to Sonnet 5 and New Benchmarks

The episode begins with the announcement of Anthropic's new model, Sonnet 5, which claims to be the most agentic Sonnet yet and offers significant performance at a lower cost compared to Opus models. The host discusses his initial testing and boredom with traditional vibe checks, introducing a new, standardized benchmark system called the How I AI Bench designed to evaluate models on relevant tasks such as writing PRDs, solving bugs, and generating designs. This benchmark system is built live using Claude Code, emphasizing repeatability and subjectivity balanced with model judgments.

Construction of the How I AI Bench

The host details how he used Claude Code to develop the benchmark based on stored session histories, focusing on creating tasks that test the models' abilities to produce PRDs, prototypes, and assess agent personality. The process included designing an HTML interface for scoring outputs on gut feel, exporting JSON scores, and running blind tests across multiple models—Sonnet 4.6, Sonnet 5, Opus 4.8, GPT-5.5, and Gemini 3 Pro. This approach aims to bring a more subjective, human-like scoring method to model evaluation.

Benchmark Results and Insights

The results from the blind tests reveal surprising and somewhat counterintuitive leaderboard standings, with Gemini 3 Pro and Sonnet 5 ranking at the top, and Opus models and Sonnet 4.6 falling behind. The host discusses the discrepancies between automated scores and personal preferences, noting the models' tendency to rate outputs toward the middle of the spectrum. It highlights how individual taste and subjective judgment differ from automated benchmarks, emphasizing the need for continuous refinement and the importance of aligning benchmarks with human preferences.

Reflections on Model Performance and Future Improvements

The host reflects on the strengths and weaknesses of each model, emphasizing the importance of task-specific benchmarks and subjective assessments like vibe scores. He discusses plans to encode his taste more explicitly into the evaluation process, retire less effective tasks like agentic bug tracking, and develop a more nuanced, weighted model ranking system combining human and model judgments. Future improvements include refining task focus, better capturing subjective taste, and expanding the benchmark to evaluate new models with greater accuracy.

Show notes

I’ve been testing every major frontier model release since the start of the year, and when Anthropic dropped Sonnet 5, I wanted more than a vibe check. I got tired of one-off tests I couldn’t repeat or compare over time, so I built something better: the How I AI Bench, a repeatable eval harness I constructed live using Claude Code while recording this episode. I ran Sonnet 5 blind against four other frontier models (Sonnet 4.6, Opus 4.8, GPT-5.5, and Gemini 3 Pro) across PRD quality, prototype generation, agentic task completion, and agent personality. The results were not what I expected.

What you’ll learn:

What Anthropic claims Sonnet 5 improves over Sonnet 4.6, and where the benchmark data actually backs that up
How I built the How I AI Bench in under 45 minutes using Claude Code, starting from my own stored session history
Why I combined human vibe scoring (70%) with LLM as judge scoring (30%) instead of trusting either alone
How to set up a local HTML scoring page so you can rate AI outputs on gut feel and export those scores as JSON
Which model I recommend for PRDs, which for complex prototypes, and which for chatting with an agent daily

—

Brought to you by:

Runway—The creative AI platform for images, video and more

Hyperagent—Deploy fleets of agents that handle real work

—

In this episode, we cover:

(00:00) Sonnet 5 is out

(01:55) What Anthropic claims

(04:02) Why I’m done with one-off vibe checks

(05:05) Building the How I AI Bench live with Claude Code

(07:42) The scoring system

(10:43) Agent voice eval