Why AI Infrastructure must evolve for Agent Experience — Akshat Bubna, Modal CTO

The discussion begins with Akshat Bubna explaining Modal's origins as a better runtime designed to improve developer experience beyond traditional Kubernetes, which was not suited for bursty, compute-heavy AI workloads. Initially aimed at building primitives for bursty processes like ETL and data engineering, Modal later incorporated GPU workloads before the rise of ChatGPT, highlighting their early foresight. The emphasis is on building a platform optimized for AI workloads, moving from just a runtime to a comprehensive AI cloud that supports inference, training, and sandbox workloads.

Akshat discusses shifting their SDK focus from traditional developer experience to enhancing agent experience. This change is driven by the need for more programmatically controlled, elastic, and fast-iterating AI agents that co-locate infrastructure with code. They aim to reduce the friction in managing infrastructure through decorator-based APIs, enabling self-provisioning and real-time updates, which are critical for AI agents operating at scale. Observability remains vital, especially for interpretation and debugging of autonomous agent actions.

The episode explores Modal's primitive building blocks designed for AI workloads, such as sandboxes, GPU snapshotting, speculative decoding, and auto endpoints. Akshat details innovations like DeFlash (block-based speculator), private IPv6 overlay networks, RDMA networking, multi-node training, and a capacity pool spanning multiple clouds for elasticity and resiliency. These primitives support bursty inference, demand training, RL rollouts, and large-scale experiments, illustrating Modal’s focus on enabling flexible, high-performance AI pipelines.

The conversation shifts to Modal’s specialized solutions like elastic inference, GPU snapshotting, and multi-node training tailored for AI's bursty workloads. Akshat emphasizes that their approach can achieve frontier-level model performance by open-sourcing components like DeFlash. They also discuss how their primitives facilitate complex workflows like continual learning, model orchestration, and interactive research, highlighting their goal of making high-quality inference and training accessible and efficient at scale.

Modal has built primitives such as networked sandboxes, sidecars, private IPv6 overlay networks, and support for RDMA to enable sophisticated AI agent workflows. These solutions allow for low-latency, secure, and distributed operations. Akshat mentions that many primitives are discovered by users experimenting beyond official documentation, illustrating the flexibility of their platform. They aim to facilitate complex multi-node, multi-region, and infrastructure-aware AI workloads, including production agents and collaborative research.

Akshat emphasizes their focus on primitives that simplify the AI workflow across inference, training, and agent management, supporting a diverse set of verticals like bio, robotics, and multimedia. They are not limited to large language models but are expanding into real-time audio/video, computer vision, and beyond. They also retain their stance against providing simple model APIs, preferring more flexible, code-based solutions that can be integrated into sophisticated AI products. Their long-term vision involves enabling companies to build entire AI systems with their primitives, emphasizing a multi-cloud, capacity-aware, and scalable infrastructure approach.