Opus 5.5 vs. GPT-6 Sol: which model won my blind taste test?

The episode begins with the host explaining his anticipation for the release of Anthropic's Opus 55 and new models from OpenAI, which dropped unexpectedly on the same morning. He shares that he had planned to review Opus 5.5 in advance but will now do a live, unscripted evaluation instead, showcasing his process in real time. The host mentions three new models: Opus 5.5, GPT-6 Soul, and GPT-6 Luna, and discusses his broader testing framework called the How AI Bench, which will be performed live during the episode. This spontaneous approach aims to demonstrate how these models perform across various tasks and how subjective judgment plays a role in their evaluation.

The host provides a quick overview of the models' price points, noting that Opus 5.5 is twice as expensive as GPT-6 Soul. He emphasizes that these are not the fastest or cheapest models but are designed for core knowledge work and coding tasks. Special attention is given to Opus 5.5, which has shipping guardrails for safety and alignment, unlike earlier versions. The host comments on Anthropic’s cautious philosophy, describing Opus 5.5 as conservative and less likely to prompt unsafe behaviors. He also shares his personal experience with different models, explaining a recent shift from Claude to GPT-based models for better personality and ergonomics, though he acknowledges that Opus 5.5 has improved his daily workflow.

He delves into the testing methodology known as the How AI Bench, which involves blind evaluation of model outputs across multiple categories—emails, PRDs, frontend prototypes, backend code, SVGs, and video editing. The host describes his scoring system using a 1-5 scale, assessing clarity, helpfulness, and appropriateness without knowing the model behind each output. He discusses his expectations beforehand based on familiarity with the models, predicting that Astra would excel in frontend and creative tasks, while Soul would shine in clarity and cost efficiency. This section illustrates how subjective and flexible the evaluation process is, with the host frequently revising his opinions after seeing the outputs.

The host tests various models on practical tasks critical for daily work. For emails, models B, C, E, and G are compared, with B and G performing most favorably due to their readability and helpfulness, especially Model G, which delivered concise and helpful drafts. The frontend prototype assessment assigns low scores across models but notes that Model C is easiest to interpret visually. For backend coding, success is high across models, with a focus on readability and correctness, and the host prefers models that produce clear, easily digestible responses. SVG illustration tests reveal Astra and GPT-6 Soul's strength in generating appealing and accurate images, contradicting earlier expectations that Anthropic models would dominate in creative visuals.

The episode discusses the models' performance in video editing tasks, which all perform poorly, attributed to skill gaps in current AI capabilities. The host admits that the models' overlays and editing decisions were subpar, emphasizing that this is more a skill issue than a model issue. In creative tasks like generating 3D Barbie models, the models produce unrefined but improving results, highlighting ongoing challenges in AI rendering of complex objects. The host shares a humorous assessment of the Barbie benchmarks, noting the models' limitations in realistic human figures but acknowledging progress in artistic expressions like SVGs and illustrations. Overall, these results underscore the current boundary of AI-generated multimedia content.

In the concluding section, the host reveals his predictions based on the tests—favoring Astra for creative work, Opus 5.5 for overall robustness, with Sol providing a good balance of price and clarity. He compares his subjective rankings with those of an AI judge, which sometimes disagrees, especially regarding Astra. The host discusses how this disagreement highlights the subjective nature of evaluating generative models, depending heavily on specific needs and personal preferences. He also revisits the Barbie Bench as a recurring test case and emphasizes that Opus 5.5 performs well for long-term, complex tasks, though it still struggles with high-fidelity 3D rendering. The episode ends with the host expressing satisfaction with the new models and teasing future reviews, underscoring the rapid pace of AI development.