Why Video Agent models are next — Ethan He, xAI Grok Imagine

The episode begins with an introduction to Ethan He, highlighting his trajectory from working on the NVIDIA Cosmos World Model to joining xAI and contributing to the rapid development of Grok Imagine in just three months. Ethan shares insights into his early research endeavors, including scale models of video foundation models, and his move from NVIDIA to xAI to work on multimodal models. The conversation emphasizes Ethan's collaborative approach, community engagement through paper clubs, and his broad experience across large-scale model training.

Ethan details the roadmap of building Grok Imagine from scratch within three months, underlining the importance of talented, cohesive teams with minimal communication overhead. He stresses infrastructure's role—especially high iteration speeds and robust data pipelines—in accelerating model development. Critical challenges include managing the vast data storage costs, inference speed optimization via techniques like step distillation, and model training complexities involving VAEs and diffusion transformers. Ethan emphasizes that most improvements stem from bug fixing and pipeline tweaks rather than novel algorithms, highlighting the iterative nature of cutting-edge AI development.

The discussion covers the exorbitant costs associated with training large-scale video models, comparable to those of language models. Ethan explains that storing and processing billions of videos—potentially petabytes—incurs millions of dollars monthly. Optimization strategies include data compression (e.g., VAEs and temporal compression) and inference speedups through distillation, which are necessary to make training feasible. The segment highlights the importance of infrastructure design and suggests that building in-house data centers might be more cost-effective than cloud storage, considering egress and storage expenses.

A deep dive into the technical aspects of training video models reveals how VAEs and patch-based tokenization create manageable latent spaces for models. Ethan discusses how training on synthetic paired data—gathered via captioning videos with vision-language models—enables effective learning despite the scarcity of text-video alignment in internet data. He describes how continuous latent spaces, along with techniques like diffusion and step distillation, allow the models to generate high-quality videos efficiently. The conversation also covers the importance of leveraging image models as a foundation for video, as they are cheaper to train and offer dense language-code connections.

The episode explores the challenges and future prospects of real-time, long-horizon interactive video models. Ethan envisions systems that respond in milliseconds, supporting interactive applications like gaming and immersive environments. Current approaches include video extension, reference tokens, and hierarchical compression to manage the enormous context length. He emphasizes that practical systems may rely on hybrid solutions—using tools like FFmpeg and editing models—rather than purely generative processes. The need for efficiency and the potential for generative UI, replacing traditional interfaces with AI-driven visuals, are highlighted as key future trends.

A recurring theme is the concept of world models—comprehensive, real-time, interactive systems capable of long-horizon reasoning. Ethan discusses how models can be made aware of their context, manage long histories, and incorporate recursive self-awareness. These systems could enable a level of interaction and decision-making comparable to human cognition, such as controlling embodied robots or simulating entire virtual environments. The conversation also covers architectural ideas like memory attention, attention pruning, and the integration of language, vision, and audio models, emphasizing that much of the perceived intelligence stems from language understanding and reasoning.

Ethan addresses the challenges of misinformation and misuse of generative media, discussing watermarking techniques like SynthID and the ongoing struggle to prevent reverse engineering. He notes that as models improve, detecting machine-generated content becomes more difficult, requiring continual advances in watermarking and validation methods. The ethical implications surrounding AI safety, privacy, and the transparency of AI-generated content are acknowledged, with instances of regulations, watermarking, and watermark removal techniques being part of the broader conversation.

The podcast concludes with Ethan reflecting on his decision to leave xAI to pursue research on language models, emphasizing that most recent breakthroughs and future potential lie in LLM improvements—particularly in managing self-awareness, context length, and recursive reasoning. He highlights that advances in language understanding will underpin the next wave of generative and interactive AI, including long-horizon video, robotics, and embodied AI. Ethan underscores his conviction that integrating these capabilities through better architecture, self-management, and multi-modal reasoning will define the coming years.