The Pragmatic Engineer
Formal methods with Hillel Wayne
· Summarized
Key takeaways
- Formal verification, such as using tools like TLA+ and Alloy, is a powerful method for proving correctness of critical systems, especially in distributed and concurrent environments.
- Most software engineering practices are still not using formal methods widely due to their complexity and modeling challenges, but they hold significant promise for high-assurance applications.
- AI tools are helpful in generating code snippets, fixing syntax errors, and understanding error traces, but they are currently weak at creating meaningful, high-level properties and proofs.
- A strong foundation in logic and discrete mathematics makes software engineers better at reasoning about systems, debugging, and designing reliable software.
- The practice of formal methods can be improved by starting small, focusing on high-stakes systems, and gradually integrating more rigorous verification as needed.
- The differences between software and traditional engineering include materials, sharing culture, and iteration practices. Software benefits from open source and version control, which are not as prevalent in other engineering fields.
- Practicing design, testing, and migration repeatedly helps engineers become proficient at complex, error-prone tasks, like concurrency and database migrations, just as repeated practice improves skills in other engineering disciplines.
Summary
The Role of Formal Methods in Software Engineering
This section introduces formal methods as a potential solution to improve software correctness, especially in complex distributed systems. Hillel explains how formal specification languages like TLA+ model state machines and how model checking can reveal subtle bugs that are difficult to find through traditional testing. The discussion emphasizes that formal methods are currently niche due to their complexity and the challenge of modeling real-world systems, but have significant potential for critical systems in cloud infrastructure, banking, and aerospace.
Comparison Between Software and Traditional Engineering
Hillel explores how software engineering relates to traditional engineering disciplines. He highlights similarities, such as the iterative approach and the high cost of mistakes, but also differences, notably in the materials used—software being intangible and highly flexible. He discusses how different engineering fields handle iteration, planning, and sharing information, noting that software benefits from open-source culture and comprehensive version control, which are not prevalent in other engineering areas. The conversation underscores how these differences influence the perception and practice of engineering in software.
Tools and Techniques for Formal Verification and Modeling
This section covers specific tools like TLA+, Alloy, and others. Hillel demonstrates how TLA+ models distributed systems and checks properties via model checking, while Alloy offers a more accessible declarative language and visualization for data and domain models. The discussion touches on the practical use of these tools in industry, such as at Amazon, and their limitations, like difficulty modeling probabilistic behaviors. He shares insights on how these tools help visualize, verify, and debug complex systems, especially where concurrency and interleavings are involved.
AI, Formal Methods, and the Future of Verification
A key focus is the impact of AI on formal verification. Hillel discusses how AI helps generate specifications, fix syntax errors, and understand error traces, but currently struggles with creating meaningful properties or comprehensive proofs. He emphasizes that AI is a force multiplier but cannot replace the expertise needed to define correct properties. The conversation points out the ongoing challenges and the cautious optimism about AI making formal methods more approachable but not fully replacing the need for human judgment and mathematical understanding.
Mathematics, Logic, and the Practical Skills for Software Engineers
Hillel advocates for a solid foundation in logic and discrete mathematics for software engineers, as these skills underpin formal reasoning and system modeling. He reasons that understanding formal logic, graph theory, and set theory can make engineers better at debugging, reasoning about concurrency, and designing reliable systems. The podcast also touches on how mathematical representation improves understanding, especially in distributed and stateful systems, and how AI tools currently lack the ability to autonomously generate useful properties for models.
Practical Advice for Implementing Formal Methods
Finally, Hillel offers practical guidance: start small by experimenting with formal tools on critical components like distributed databases or network protocols. He suggests reading influential books on safety, systems thinking, and debugging, to build familiarity and confidence. The discussion underlines that formal methods are most effective when integrated into a broader engineering practice focused on resilience, correctness, and rigorous reasoning, complemented by AI where appropriate.
Sponsors
- Antithesis · verify your system’s correctness without human review or traditional integration tests – and avoid bugs or outages.
- TurboPuffer · a vector and full-text search engine built on object storage. It’s fast, cheap, and extremely scalable.
- WorkOS · everything you need to make your app enterprise ready.
Show notes
Brought to You By:
• Antithesis – verify your system’s correctness without human review or traditional integration tests – and avoid bugs or outages.
• turbopuffer– a vector and full-text search engine built on object storage. It’s fast, cheap, and extremely scalable.
• WorkOS – everything you need to make your app enterprise ready.
—
There’s a popular theory that AI will finally make formal verification mainstream because mathematical proof of correctness will be needed when machines write most or all of the code. But will this happen? Today, I’m talking with one of the best people to tackle the prediction. Hillel Wayne is a formal methods consultant, educator, and author, who’s deeply interested in software history.
In this episode of Pragmatic Engineer podcast, I sit down with Hillel to compare software engineering with traditional engineering, discuss where formal methods fit into modern software development, and we explore why they are essential for some of the world's most complex systems. We cover the formal specification language, TLA+, walk through several formal verification tools, examine why distributed systems are so difficult to reason about, and look into whether AI will make formal methods accessible to more engineering teams.
—
Timestamps
00:00 Intro
03:21 The Crossover Project
10:26 What software engineering does better
14:19 What traditional engineering does better
17:06 Formal methods
28:21 TLA+: what it is and demo
35:47 TLA+ at Amazon
36:59 Ways distributed systems break
39:52 Formal methods and systems thinking
45:09 The value of learning math
49:12 What TLA+ is good for and isn’t
51:39 Alloy: a declarative language for software modeling
57:42 Other formal methods tools
1:00:13 Property-based testing
1:04:20 AI and the need for formal verification
1:11:18 Logic for programmers
1:13:24 Hillel’s 2025 prediction on AI’s impact
1:20:19 Book recommendation
—
The Pragmatic Engineer deepdives relevant for this episode:
• How to debug large, distributed systems: Antithesis
• How Big Tech does quality assurance (QA)
• Resiliency in distributed systems
—
Production and marketing by https://penname.co/. For inquiries about sponsoring the podcast, email podcast@pragmaticengineer.com.
Get full access to The Pragmatic Engineer at newsletter.pragmaticengineer.com/subscribe
Follow this show on AudioBrevity and new episodes get summarized for you. Create a free account