Google's AI Infrastructure Chief, Amin Vahdat, on the Physics & Economics of Frontier AI

Amin Vahdat explains that AI and conventional data centers share concrete buildings, electrical and mechanical yards, cooling, power distribution, networking, and storage. The difference is specialization: traditional facilities are planned as 20–30-year buildings that can host many generations of servers, while AI facilities are increasingly co-designed with the hardware. A storage rack may consume 10–40 kilowatts, whereas a TPU or GPU rack already reaches hundreds of kilowatts and could reach megawatts. Designing a row for dozens of storage racks versus one or two AI racks changes the building’s power, cooling, networking, and physical layout.

Vahdat says FLOPS and other chip-centric metrics describe what hardware could theoretically deliver, but Google cares about performance on an actual workload. He calls the relevant measure “goodput”: useful work that contributes to completing the task, including the time lost to failures, recovery, checkpoint restoration, and repeated computation. At 100,000-accelerator scale, synchronous training, serving, and agentic workloads can stop when one component fails. A failure may occur multiple times per day or even multiple times per hour, and the causes form a long tail spanning chips, chiplets, HBM, networking, compilers, runtimes, operating systems, and models.

Google’s target is to double effective serving capacity roughly every six months, measured as token-generation capability rather than simply doubling FLOPS or accelerator count. Vahdat says the gains will come from dozens or hundreds of optimizations: model changes, runtime improvements, compiler work, and hardware. In his experience, most intelligence-per-watt gains come from model-side improvements, followed by software systems that use existing hardware more effectively. Hardware still provides a powerful multiplier, with year-over-year performance improvements of 2× or more possible, but its benefits depend on the rest of the stack.

Google began the TPU program in 2013, when conventional wisdom held that custom accelerators were too narrow and general-purpose processors would win through flexibility and Moore’s law. The first TPU targeted inference for language translation and voice recognition; the second extended the concept to training. Transformers then became a major turning point, while recommender systems made TPUs useful for advertising and related workloads. Vahdat describes the program’s evolution as an expansion from a few inference applications into training, transformers, recommendation, and increasingly large generative-AI systems.

Vahdat says specialization is worthwhile only when a workload is large, persistent, and predictable enough to repay the fixed cost of a new chip program. Google considered whether one chip should handle training and inference or whether the workloads justified separate designs. Its projections indicated that inference could represent 30–60% of the market over the hardware’s lifetime, leading to the 8i inference chip and 8t training chip. Both can still run the other workload, preserving flexibility if demand forecasts are wrong. Burning an entire model architecture into silicon would go further, but today’s hardware instead implements reusable linear-algebra primitives.

The host asks about designing chips, networks, and software together. Vahdat says abstraction layers make systems portable across clouds, hardware, software, and network topologies, but that flexibility leaves performance on the table. Co-design can remove impedance mismatches at every layer, producing gains that compound across compute, networking, storage, software, power delivery, and availability. The benefit may be 10%, 20%, or even 2× at individual layers; the cost is reduced portability and greater vendor dependence. Google therefore supports both vertical optimization and open interfaces rather than forcing customers into a single closed stack.

Vahdat describes Google’s collaboration with DeepMind as a daily, shoulder-to-shoulder partnership spanning models, hardware, consumer services, and cloud. If DeepMind discovers a model or mathematical optimization while a chip is still being designed, engineers may alter the architecture, negotiate a model change that captures most of the benefit, or even delay tape-out by a week or two. Google simultaneously manages chips in production, bring-up, implementation, design, and concept stages. Hardware planning runs two to five years ahead, so small late changes usually are not worth disrupting a chip, but large opportunities can be intercepted before manufacturing.

Vahdat says ordinary human interactions naturally rate-limit model requests: a person reads a response, thinks, and submits a follow-up. Long-horizon agents can move between requests in milliseconds, with CPUs parsing responses, maintaining state, and gathering context from DRAM, SSDs, or hard drives. That raises demand not only for GPUs and TPUs but also for CPUs, storage, and networking. It also complicates facility design: accelerators are dense and network-intensive, while CPUs and storage have different requirements. Google may either mix them in a less-specialized building or separate them into buildings and pay the networking and latency costs between them.

Google adopted wavelength-division multiplexing and optical circuit switching roughly 15–16 years ago. Instead of converting every packet into the electrical domain, optical circuit switches use programmable MEMS mirrors to direct light between fiber ports, creating shortcuts between rack groups and allowing the network spine to expand or contract without physically moving fiber. For TPU pods, the system can redirect light from a failed rack to a spare rack in milliseconds. Fiber remains necessary because free-space links across a large building would suffer greater attenuation and bandwidth challenges, although the light briefly travels through the switch via tiny mirrors.

Vahdat calls power the most fundamental long-term constraint, although networking, land, and other constraints shift over time. Google prefers utility-connected power and gives utilities years of notice for gigawatt-scale demand, while paying for transmission and infrastructure upgrades that its load requires. If a utility can provide only 700 megawatts by the needed date instead of 1 gigawatt, Google may wait or temporarily add solar, batteries, or other generation, potentially returning power to the grid during peak demand. Training favors large, concentrated clusters; serving requires globally distributed endpoints. Seven- and eight-year-old TPUs can remain fully utilized, but Google eventually replaces whole pods—sometimes 9,600 chips and roughly 152 racks—once efficiency and depreciation justify it. Vahdat also argues for open standards such as PyTorch support on TPUs rather than requiring Google’s JAX framework.

Vahdat treats orbital data centers as an active Google “moon shot,” driven largely by energy availability. In space, solar power could be about 1.4× stronger without atmospheric attenuation and available 90–100% of the time in a sun-synchronous orbit, versus roughly 28–35% on land, potentially reducing the need for batteries. Cooling, repair, reliability, and networking remain difficult, with free-space optical links replacing fiber. Looking toward 2036, he expects much tighter rack integration, perhaps 576 or 1,152 GPUs or equivalent accelerators, multiple megawatts per rack, and only a few connections for water, power, and fiber. He does not rule out racks being launched directly into space.