The Inference Engineering Masterclass — Philip Kiely & Ali Taha, Baseten

This section discusses the process involved when handling long context requests, such as 200,000 tokens. It covers topics like cache-aware routing, where cached inputs help reduce prefill costs, the separation of prefill and decode GPU tasks, and the use of speculative decoding models to increase throughput especially in coding tasks. It emphasizes how sophisticated management of GPU resources and caching strategies are essential for efficient large-scale inference, alongside the importance of dedicated deployment versus shared APIs.

Support efforts for new models are highly labor-intensive, requiring significant infrastructure work including quantization, calibration, and training of specialized speculative decoders. The discussion highlights the differences when supporting models like GLM-5.2 and Kimi K3, including architectural novelties like sparse attention. The process involves customizing support for different hardware architectures and retrofitting models with new components from open source initiatives, sometimes merging layers across models (‘franken-merges’). The challenges of quick support, support validation, and maintaining quality are also explored.

This section delves into the intricacies of quantization, a lossy compression technique used to speed up inference. It discusses advanced research showing how quantization errors can cancel out when carefully selected layer-wise, leading to better-than-expected fidelity. The importance of calibration, layer selection, and the impact of vendor support on inference quality are examined. The discussion also covers the potential for models to optimize themselves, such as generating their own kernel code or improving their performance through training-aware techniques.

The focus here is on hardware evolution, including the role of GPUs versus ASICs, the importance of interconnect bandwidth between nodes, and the push towards increasingly specialized hardware. Notable topics include the emergence of Rubin chips designed to resemble ASICs but still programmable, and how future hardware cycles are long, with designs made years in advance. The conversation emphasizes how infrastructure like tensor parallelism, model sharding, and inter-node transfer bottlenecks significantly influence AI scale and performance.

This part covers various parallelization strategies for large models, such as tensor, expert, and pipeline parallelism. It emphasizes the hardware considerations like interconnect speeds and the limitations of mega kernels and fused kernels. The shift towards hardware like Rubin that may function more like ASICs, and the importance of auto-tuning configuration setups for optimal performance, are also discussed. The segment highlights how these techniques aim to push model inference to higher speeds—up to 10×—through stacking various hardware-accelerated optimizations.

Looking ahead, the discussion touches on emerging AI modalities beyond text, such as audio and video, and the technical challenges in scaling these models. Topics include the quadratic attention bottleneck in long-form video and the prospects of combining autoregressive and diffusion architectures for coherent long sequences. The conversation emphasizes the exponential demand for AI due to increasing usage, better infrastructure, and new modalities, as well as the need for faster networking (e.g., high-speed NICs) to overcome current bottlenecks.

This theme focuses on the longevity of models like Llama 3, which remains in use long after release, emphasizing that open source allows for continued use and modification without reliance on newest versions. It discusses the economic and strategic considerations of continuously evolving models versus sticking with well-understood, trusted models, especially in enterprise settings where reliability and predictability are critical.

The final sections explore the blending of training and inference, where techniques like continual learning, post-training quantization, distillation, and dynamic inference adjustments are converging. The idea is that models will increasingly learn from inference data in real-time, leading to systems that optimize and improve themselves continuously. This includes the potential for feedback loops where models help optimize their own infrastructure and training pipelines, catalyzing a shift toward integrated training-inference pipelines.