🔬 Scaling Past Informal AI - Carina Hong, Axiom Math

The podcast discusses the paradigm shift in AI verification, emphasizing its importance beyond regulatory compliance to be a tool for scaling and enhancing human intelligence. The analogy of Ramanujan's mathematical brilliance illustrates how formal verification transforms intuitive insight into solid, shareable knowledge, benefitting the community by enabling others to build upon verified results. The emphasis is on verified AI as a means to exponentially increase the capabilities of AI systems through rigorous proof and formal methods, ultimately leading to more reliable and powerful AI.

The episode explores how formal verification is integrated into AI training and inference, especially through tools like Lean, a formal proof language. Axiom's achievements in generating proofs with 99% accuracy on benchmarks surpass existing models, reducing reliance on informal proofs and boosting robustness and performance. The discussion touches on the importance of high-quality data, recursive reasoning, and the potential of AI to automate proof discovery, leading to more efficient and trustworthy AI systems. The concept extends to real-world applications like hardware and software verification, emphasizing that verified AI can enhance safety, reliability, and performance across industries.

The podcast highlights the current fragmentation in AI research, with many players working independently, which hampers progress. Achieving category leadership in areas like AI for math can foster collaboration and reduce fragmentation. The discussion also covers technical bottlenecks such as knowledge graph building, formalizing informal proofs, and managing large proof sizes. It stresses the importance of long-term focus, talented teams, and alignment around shared goals. The conversation underscores that formal verification, while costly to produce, is inexpensive to verify, making it a promising path toward achieving trusted and scalable AI. It concludes with reflections on societal and scientific impacts, like reliable AI in critical systems and the pursuit of AGI.