Invest Like the Best with Patrick O'Shaughnessy
Neil Movva - Making AI 10x Cheaper - [Invest Like the Best, EP.488]
· Summarized
Key takeaways
- The fundamental cost of AI inference—tokens—can be significantly reduced by optimizing hardware, software, and power infrastructure through innovative architectures and strategies.
- Building a full stack approach—from kernels to chips to energy sources—is essential to creating a sustainable, low-cost AI inference ecosystem.
- Distributed, small-scale data centers powered by renewable energy and managed flexibly can be more efficient and resilient than traditional large data centers.
- The hardware landscape is evolving with alternative memory technologies and scavenger strategies to address supply shortages and improve performance per dollar.
- The future of AI involves democratization through abundant, customizable tokens, with a continued strong role for open-source models despite proprietary advances.
- Optimizing for throughput rather than latency will enable long-running background AI agents that are proactive rather than reactive, opening new possibilities for AI applications.
- The AI hardware market may see contradictions, with supply shortages creating opportunities for scavengers and niche chip architectures gaining prominence.
Summary
Revolutionizing AI Cost and Software Infrastructure
Neil Movva discusses his company Sail, which aims to drastically reduce the cost of AI tokens by creating a 'token factory' focused on inference. He emphasizes building a full stack ecosystem—from software kernels to chips and power—to optimize inference cost and efficiency. Neil highlights the importance of long-running AI agents in background operations and how focusing on throughput over latency, along with innovative hardware strategies, can reshape the economics of AI deployment.
The Full Stack of Intelligence and Hardware Optimization
Neil provides a detailed overview of the software and hardware layers that compose AI inference, including GPU kernels, chip architectures, and memory technologies. He advocates for more efficient use of GPUs by better orchestrating compute, exploring alternative memory solutions like SRAM and DRAM, and designing chips to skip traditional bottlenecks. He also discusses the importance of co-designing hardware and software for specific inference workloads, especially long horizon and background tasks.
Energy, Data Centers, and Market Dynamics in AI Infrastructure
The conversation explores the future of data centers and energy utilization. Neil suggests that smaller, distributed data centers powered by renewable sources can be more cost-effective and resilient, especially when combined with flexible power strategies. He discusses the ongoing chip supply shortages and how scavenging unused capacity, along with a focus on scaling supply and innovative buildouts, will be key. Additionally, Neil challenges conventional assumptions about manufacturing capacity, highlighting that efficiency and supply chain management are crucial to meet the demands of AI's growth.
Contrarian Views and Market Outlook for AI Chips
Neil offers contrarian insights on the AI hardware market, including skepticism about the long-term dominance of Nvidia and the current state of Moore's Law improvements. He emphasizes that the core of AI growth will rely on optimizing compute and memory efficiency through novel architectures, offloading, and better orchestrating the global compute supply. Neil advocates for a flexible, scavenger-style approach to sourcing chips and power, predicting that an era of abundant, diversified hardware infrastructure is on the horizon.
Vision for the Future of AI and Open vs Closed Models
Neil envisions a future where abundant, customizable tokens democratize AI usage, allowing diverse enterprises and individuals to harness tailored AI agents. While current models are driven by proprietary advancements, Neil stresses that open source will continue to play a vital role, as capabilities diffuse quickly and the race to scale persists. He believes that decentralized, custom AI ecosystems will become prevalent, supported by infrastructure that continuously lowers token costs and enhances accessibility.
Sponsors
- Ramp · Ramp is the only platform built to make your finance team leaner, faster and better, saving businesses 5% annually on average. Ramp customers grow revenue 3.2 times faster.
- Vanta · Vanta automates security and compliance for over 16,000 companies, helping them stay audit-ready and trustworthy.
- WorkOS · WorkOS provides APIs for enterprise capabilities like SSO, SCIM, RBAC, and Audit Logs, enabling AI and B2B companies to scale to enterprise level quickly.
- Rogo · Rogo is the AI platform designed specifically for Wall Street, building agents that understand business workflows for diligence, modeling, and delivering analysis.
- Ridgeline · Ridgeline offers a unified, real-time operating system for investment management, integrating trading, reporting, and compliance in one platform.
Show notes
My guest today is Neil Movva, founder of Sail. Sail is building what Neil calls a token factory, an inference company designed for a specific kind of future, one where AI agents run in the background for hours or days at a time rather than answering a human in real time.
In that world, latency matters less and cost matters more, and Neil has built the whole company around driving the cost of a token as low as it can possibly go.
What makes this conversation special is that it is one of the most detailed tours I have ever done through the full stack of intelligence, the software, the chips, and the power, and how all three connect.
Along the way we cover the trade-off between speed and cost that lives inside every GPU, his scavenger strategy for buying the chips and power nobody else wants, his contrarian view on Nvidia, and why the premium the frontier labs charge for being three to six months ahead may not last.
Please enjoy my conversation with Neil Movva.
For the full show notes, transcript, and links to mentioned content, check out the episode page here.
-----
Become a Colossus member to get our quarterly print magazine and private audio experience, including exclusive profiles and early access to select episodes. Subscribe at colossus.com/subscribe.
-----
Ramp’s mission is to help companies manage their spend in a way that reduces expenses and frees up time for teams to work on more valuable projects. Go to ramp.com/invest to sign up for free and get a $250 welcome bonus.
-----
Trusted by thousands of businesses, Vanta continuously monitors your security posture and streamlines audits so you can win enterprise deals and build customer trust without the traditional overhead. Invest Like the Best listeners get a special offer of $1,000 off Vanta when you go to vanta.com/invest.
-----
WorkOS is the infrastructure B2B and AI-native companies use to sell to enterprise. It covers everything enterprise security requires: SSO, SCIM, RBAC, Audit Logs, AI governance, and more. Trusted by 2,000+ fast-growing companies, including OpenAI, Anthropic, Cursor, and Vercel.
-----
Rogo is the AI platform for finance. They're building agents for Wall Street that are trained to understand how bankers and investors actually do work: from diligence and modeling, to turning analysis into deliverables. To learn more, visit rogo.ai/invest.
-----
Ridgeline has built a complete, real-time, modern operating system for investment managers. It handles trading, portfolio management, compliance, customer reporting, and much more through an all-in-one real-time cloud platform. Visit ridgeline.ai.
-----
Editing and post-production work for this episode was provided by The Podcast Consultant.
Timestamps:
(00:00:00) Welcome to Invest Like The Best
(00:02:20) Neil Movva
(00:03:22) Building a Token Factory
(00:05:32) The Rise of Long-Running Agents
(00:08:47) Deep Research and Cybersecurity
(00:15:12) The Full Stack of Intelligence
(00:20:03) Throughput Versus Latency
(00:24:58) The Future of AI Chips
(00:33:19) Why Transformers Work
(00:36:43) The Future of Data
(00:44:05) The Market for AI Chips
(00:47:56) Is the AI Boom Different?
(00:51:08) Reinventing the Data Center
(00:56:43) Scavenging Power
(01:01:04) Where Compute Is Most Inefficient
(01:07:02) Open Versus Closed Models
(01:10:37) A Trillion Tokens a Day
(01:12:42) The Contrarian Case on NVIDIA
(01:14:38) Advice for AI Hardware Founders
Follow this show on AudioBrevity and new episodes get summarized for you. Create a free account