Can Open Models Keep Up With the AI Frontier? | Nathan Lambert

The episode opens with Alicia Li introducing Nathan Lambert and framing the episode's core question: how durable is the gap between the leading AI labs (like OpenAI and Anthropic) and open-source models, particularly those in China? Lambert discusses his perspective that major AI labs have significant structural advantages: access to compute, capital, and pioneering training paradigms. He emphasizes that the open-source ecosystem, especially in China, has been remarkably capable of following the frontier models closely, often driven by talent and strategic priorities rather than just resources. Lambert also introduces his nonprofit, Trillium, which aims to increase transparency around AI development and deployment, focusing on open-sourcing RL environments and documenting AI's practical use in automating research. He underscores the importance of understanding how much open models can catch up, considering factors like distillation, talent, and data, amidst the growing costs and constraints faced by frontier labs. Lambert hints at the complex and nuanced dynamics of AI progress, where resource scarcity, talent, and innovative approaches like post-training and environment creation play critical roles.

Lambert discusses how large AI labs like OpenAI and Anthropic retain their lead through substantial investments in compute, talent, and innovative training methods, often pioneering new paradigms like reasoning models. He notes that these labs generate groundbreaking results, such as inference scaling laws, which dramatically reshape the understanding of model capabilities. Conversely, open-source labs, especially in China, are driven by pragmatic goals—building models that can be commercialized through APIs and capturing market share. He explains that while closed labs focus on high margins and proprietary models, open models tend to be more distributed, relying heavily on community contributions, flexible licensing, and iterative improvements. Lambert sees differences in the timelines—closed labs can quickly deploy scaled, optimized models, while open-source efforts are slower but benefit from broader participation. He argues that the actual talent pool in China, including skills in distillation and post-training, is underestimated and rivals that of leading Western labs, despite less compute.

Lambert provides a nuanced view on the significance of distillation in catching up with frontier models. He explains that while distillation is often hailed as a key lever for open-source models, its actual impact—particularly in post-training—is complex and potentially limited. He notes that distillation’s effectiveness depends on access to reasoning traces, which are hard to obtain and replicate, especially from high-performing proprietary models like Claude. He discusses how open-model builders are often stuck using older techniques and models due to the difficulty of effective distillation from the best closed models. Lambert also emphasizes that Chinese labs excel not just technically but in deploying outputs at scale, with significant data and API monetization strategies. He mentions that the pace of progress in open models is accelerating, driven by improved techniques, data workflows, and community efforts, but resource constraints remain a fundamental barrier.

Lambert stresses that much of AI progress now hinges on high-quality data, innovative environments for training, and post-training techniques like reinforcement learning. He describes how data industries are vital, with companies buying massive datasets and synthetic environments, sometimes cutting corners that lead to issues like reward hacking. Lambert supports open-source efforts to create and share complex RL environments, arguing they are crucial infrastructure for research and understanding AI capabilities. He advocates for transparency in documenting how models are trained and used, believing that sharing trajectories, tasks, and environments can demystify AI progress and help policymakers and researchers assess AI risks more accurately. Lambert also highlights the long-term importance of building a robust data economy, where improved data quality and cost efficiency can substantially accelerate AI development across sectors.

In the episode's final sections, Lambert discusses his initiative with Trillium to open-source components of AI training, especially RL environments, to foster transparency and collective understanding. He envisions documenting how AI models automate tasks and where they fall short as a way to clarify the pace and capabilities of AI development. Lambert emphasizes that this open approach can help depolarize debates about AI risks, including recursive self-improvement (RSI), by providing measurable data and shared benchmarks. He stresses the need for a diverse ecosystem of organizations—academic, nonprofit, and commercial—to track, report, and understand AI progress. Lambert notes that the current mystique around RSI and AI capabilities can be mitigated by openly sharing data, tasks, and results, thus enabling better policy and safety measures. He believes that this transparency will help society prepare for the future, reduce anxiety rooted in uncertainty, and advance more grounded discussions about AI risks.