Everything you need to run Mission Critical Inference (ft. DeepSeek v3 + SGLang)

The podcast opens with a discussion of the launch of DeepSeek V3, a groundbreaking model boasting 671 billion parameters. Experts delve into its architecture, including fine-grained Mixture of Experts (MOE) and multi-token prediction objectives. DeepSeek V3 is positioned as the best open weights model globally in January 2025, ranked 7th on the LM Arena leaderboard. The conversation highlights the competitive landscape of large language models, particularly from Chinese labs, and the challenges of deploying these immense models.

The episode explores the difficulties associated with serving large language models like DeepSeek V3, focusing on the hardware requirements such as the H200 clusters with significant VRAM and bandwidth. The discussion emphasizes the importance of collaborative efforts between Baseten and DeepSeek in delivering these models effectively, along with the introduction of SGLang, a new VLLM alternative that enhances performance. The presenters touch on the foundational aspects of scaling models for inference and the technical hurdles that developers face.

A significant portion of the podcast revolves around the three critical pillars necessary for running mission-critical inference workloads: performance at the model level, performance at the cluster level, and performance at the region level. Each pillar is elaborated upon, noting that merely optimizing one aspect (like a single GPU model) is insufficient for handling high demand. Scaling solutions are necessary to ensure reliability and meet strict latency requirements across various regions, underscoring the infrastructure needed for efficient AI operations.

In discussions about recent advancements in model optimization, the presenters introduce SGLang, highlighting its innovative attributes and performance benefits over existing frameworks like TensorRT and vLLM. Key techniques such as speculative decoding and Radix attention for enhanced cache performance are presented as essential developments. The prospects of using reinforcement learning for fine-tuning also arise, reflecting ongoing trends in AI model training and deployment, and the importance of adaptability in evolving workloads.

The podcast concludes with an announcement about the upcoming AI Engineer Summit in NYC, inviting engineers and leaders in AI to engage in high-signal discussions at the event. The conversation emphasizes the significance of community and collaboration in advancing technology and fosters excitement about key speakers and sessions planned for the summit.