🔬 ESM: The Bitter Lesson is Coming for Proteins - Alex Rives, BioHub

The podcast opens with a discussion about innovative AI methodologies applied to protein science, highlighting approaches like world modeling and predictive modeling for designing proteins, including antibodies and other therapeutic molecules. Alex Rives from Biohub introduces the concept of using large-scale, unsupervised language models trained on vast biological data to uncover biological structures and functions that emerge predictably with increased computational resources.

The conversation details the evolution of the ESM series—from initial models trained on protein sequences to the latest models like ESM3 that incorporate massive datasets including metagenomic sequences. The models use simple next-token prediction tasks but have surprisingly learned complex biological properties, supporting a 'world model' approach that captures the diversity across all of evolution. The significance of scaling data and parameters to unlock new capabilities is emphasized.

A detailed explanation of the 'world model' concept is provided—an unsupervised learning framework that abstracts meaningful patterns from massive biological data, supporting generalization and compositionality. This model can be adapted with different 'heads' for tasks like structure prediction or functional annotation. It enables the generation and prediction of protein features, facilitating protein design, with specific attention to therapeutically valuable molecules like antibodies.

The episode discusses how AI models like ESMFold2 and the broader world model have achieved success in predicting protein interactions and designing complex molecules such as single-chain antibodies (SCFVs). These advances could revolutionize therapeutic development by enabling rapid, high-affinity antibody design, even for binding interfaces generally challenging to engineer through traditional methods.

A recurring theme is the importance of data diversity and quantity. While current models benefit from large, diverse datasets, there is a recognition of the need for deliberate data collection strategies to target underrepresented biological systems. The conversation covers the potential of scaling both data and compute to overcome biological complexity, including cellular interactions and dynamic processes, moving towards more comprehensive virtual cell models.

The final sections explore the vision of integrating AI models with experimental biology—creating a feedback loop where models generate hypotheses, which are tested in the lab, refining the models further. The goal is to develop scalable, accurate digital representations of biological systems from molecules up to physiology, ultimately transforming understanding, diagnosis, and treatment of disease. The role of open science and community collaboration in accelerating this progress is also emphasized.