The AI Rankings Just Got Reshuffled

The episode opens with Google’s unveiling of Gemini 4 Carbon, just two weeks after Gemini 4 Argon. The host explains that once Google has completed the expensive base-model pretraining, it can iterate through reinforcement learning rather than waiting for another full pretraining run. Google CEO Sundar Pichai said on an earnings call that AI was increasing the company’s training and launch velocity and that he expected further acceleration over the next 12 months. According to the reported benchmarks, Carbon is a substantial improvement over Argon. It gained eight points on Terminal-Bench 4.0 and six on Frontier Suite for coding; 12 points on ExploitGem and three on CWE-Bench for cybersecurity; and seven points on OSWorld and eight on the Agents’ Last Exam for computer use. It led six of eight key benchmarks and reached first place on Artificial Analysis’s new Intelligence Index V5 with 59 points, nine points higher on the new scale and ahead of Opus 5.5. It ranked second on the Artificial Analysis coding index, tied with Sonnet 5.5 and three points behind Opus 5.5. The model’s measured cost was $7.76 per task—23% cheaper than Opus 5.5 and 20% more expensive than Astra. However, Carbon was not yet generally available. Google was initially giving access to selected cyber defenders and trusted partners while it gathered feedback and refined guardrails. Arena’s early render and game-design tests were judged less attractive than Opus 5.5, although Arena’s Peter Gostet said Carbon “feels a lot better than Argon” and represented a genuine comeback for Google. The host reserved judgment until broader hands-on testing but said the results looked promising.

The headline segment then turns to Microsoft and Google’s earnings. Microsoft reported that Azure grew 48% year over year to $33 billion, the first time it had disclosed Azure revenue as a separate line item. Azure helped drive total Microsoft revenue up 19% to $92.3 billion, slightly ahead of analyst estimates. Microsoft also said OpenAI contributed $10 billion to Azure revenue, roughly one-third of the division’s total. Microsoft shares rose as much as 4.8% in after-hours trading. Google Cloud nearly closed the gap, doubling year-over-year revenue to $30.7 billion and exceeding Wall Street’s high expectations. The problem for Google was that other businesses were slowing. Search revenue grew 12%, down from 17% in the prior quarter and 19% the quarter before. YouTube growth also slowed for a third consecutive quarter, reaching 9.2%. Executives avoided directly answering a question about TPU profit margins, and Google missed bottom-line earnings partly because of a markdown on its SpaceX stock. The host says this raises the possibility that Google is reaching the point where AI products cannibalize its search business. Pichai acknowledged that the rapid growth of AI Mode queries was accompanied by lower monetization. Although analyst Dan Ives emphasized Google’s “cloud print,” investors focused on the weaker search figures and pushed the stock down as much as 10% after hours. The episode also examines infrastructure spending. Google is guiding toward a 50% increase in capital expenditure next year, to $300 billion, but is currently undershooting this year’s targets. Microsoft, Oracle, and Meta face similar constraints. The host argues that the apparent slowdown is not primarily collapsing demand: hyperscalers would prefer to keep building faster but are constrained by components, electricity, labor, and community support.

The Department of Justice is questioning Nvidia’s proposed $13 billion acquisition of Hugging Face. Reuters reported that investigators are examining whether Nvidia ownership would let it control a major distribution point for AI models. Nvidia has said Hugging Face would remain free to work with model developers, chip manufacturers, and cloud providers, and that the transaction was expected to close in the first half of 2027 pending regulatory approval. The inquiry is part of a broader, multi-year antitrust investigation into Nvidia. Earlier reporting had suggested that the DOJ was focused on Nvidia’s licensing and talent-acquisition arrangement with chip startup Groq. That deal was structured as an “acquihire,” a format that has become common in AI and does not automatically trigger the same antitrust review as a traditional acquisition. The episode notes that Google used a similar structure in its 2024 Character AI deal. New reporting indicates the investigation predates the Groq arrangement and includes Nvidia’s acquisitions of data-center optimization companies SketchMD and RunAI. Those were conventional acquisitions rather than acquihires. Sources also said that DOJ questioning has covered Nvidia’s venture investments, sales practices, and whether the chip designer has pressured customers. The host stresses that no charges have been filed and that the investigation could end without action. Still, the apparent scope is more serious than a narrow inquiry into whether Nvidia is using acquihires to avoid review. It now appears to be a top-to-bottom examination of the dominant AI chipmaker’s role in the competitive landscape.

The main episode begins with the host explaining his position on benchmarks: they are inherently limited but still valuable because, without them, comparisons would be based mostly on vibes. A single score should not be treated as gospel, but well-designed evaluations can help indicate where a model may fit in a technology stack. Artificial Analysis has now released version 5 of its Intelligence Index, which combines roughly a dozen benchmarks into one score. The host traces the index’s evolution from the period when Chatbot Arena preferences dominated model evaluation, through the rise of objective coding and research tests, to repeated updates as benchmarks became saturated. Artificial Analysis also released an interim version 4.2 after GPT-6 Astra’s computer-use improvements exposed that the prior index placed too little weight on agentic capabilities. Version 5 relies on private datasets for the majority of its benchmarks, increasing the difficulty of “benchmark maxing”—training a model on test questions and answers so that a score reflects memorization rather than general ability. New or upgraded tests include Frontier Code 2.0, which evaluates whether a repository maintainer would merge the generated code; Terminal-Bench Science, an agentic scientific-research evaluation; and HLE Diamond, a harder version of Humanity’s Last Exam. AA also upgraded its tests for knowledge, hallucination, and physics, and replaced GDPVAL with AA Briefcase, Automation-Bench AA, and GDP.PDF for more granular white-collar work evaluation. The changes reflect a shift in what the host considers important. Long-context reasoning has become less decisive because improvements in compaction and memory have made it more manageable, while agentic coding and research have become central to real-world use. The index now weights studies, knowledge work, coding, and science/general knowledge equally, although the host questions whether science deserves 25% for the average user.

Before Google released Carbon, Opus 5.5 led the new index with 58 points. GPT-6 Astra moved into second with 57, despite previously ranking sixth, showing how much the new weighting favored computer use and scientific tasks. GPT-5.1 Soul rose to third from ninth, Sonnet 5.5 slipped from second to fourth, and GPT-6.1 Terra moved from tenth to fifth. Gemini 4 Argon rose to sixth, tied with Meta’s model, while Fable 5.1 fell to seventh or eighth and Kimi K3.1 Pro dropped sharply from fourth to ninth. Carbon subsequently took first place with 59. The most consequential result was the widening gap between open-weight and U.S. proprietary models. Artificial Analysis said the gap increased from three points to 12. Kimi K3.1 Pro remained the leading open-weight model at 46, while DeepSeek V4.1 Flash and GLM 5.3 scored 32 and 26, respectively—below Claude Haiku 5.5 at 36. Anthropic, OpenAI, Google, and Meta all had models scoring at least 50, followed by a ten-point gap between Kimi and the rest of the field. Several commentators said the new results matched their practical impressions better. Dean Ball called it the first Artificial Analysis ranking that roughly matched his subjective ordering, while Haseeb Qureshi said it cleared away some benchmark noise and showed larger gaps both between leading labs and between closed and open weights. Others defended Kimi’s narrower strengths. Brookings fellow Kyle Chan noted that Kimi scored only 34% on Terminal-Bench Science versus Astra’s 63%, but led all models on Terminal-Bench 4.0 coding with 67% and beat Claude on the private Automation-Bench AA test. His conclusion was that Chinese models can match or beat U.S. systems on specific agentic tasks, even though a major gap remains in advanced scientific reasoning and research workflows.

Artificial Analysis’s top ten models vary considerably in cost. GPT-6.1 Terra was the least expensive at $0.96 per task, while Fable 5.1 was the most expensive at $13.04. Comparing models of more similar size, Opus and Sonnet cost about $10 per task, GPT-6 Astra $6.24, and GPT-6.1 Terra about $1.27. Mews, Wonder, Grok 4.7, and Gemini 4 Carbon were in the high-$7 range, slightly above Astra. The host says the new index is useful but expects debate over its weighting. OpenAI and Anthropic have emphasized scientific discovery, yet science may matter less to an average user than agentic coding. One response is to adopt a different weighting; another is to ignore the headline score and compare only the benchmarks relevant to a particular workflow. The host also agrees with criticism that the leaderboards use maximum effort settings. Since many people do not run models at maximum effort, he would like Artificial Analysis to provide medium, high, and extra-high leaderboards, not just a model-level ranking. Artificial Analysis also introduced a Progress Index covering models released since November 2022. It is intended to show how each laboratory’s leading systems improve over time rather than merely ranking the current generation. Commentators used the underlying data to argue that OpenAI may have the strongest reinforcement learning—or that its models perform poorly without reasoning—and to estimate that the U.S.-China model gap is about 3.3 months and narrowing. The host’s final conclusion is that benchmarks matter precisely because they can be improved and debated, but they remain guides rather than definitive judgments. Users should treat individual capabilities, effort settings, cost, and their own real-world tasks as more important than a single leaderboard position.