⚡ An Eval for AI Scientists That Knows the Answer Key - Ian Quigley, Leash Bio

Brandon and RJ introduce Ian Quigley from Leash Bio for a follow-up to an earlier episode about DNA-encoded libraries. The company’s screens can test millions of compounds against thousands of proteins at high throughput. Ian says the team realized that these measurements could be turned into a “virtual sandbox”: an agent can make a decision, receive immediate feedback from the already measured data, and perform a large number of exploratory rollouts as if it were acting like a medicinal chemist. The main problem they have developed is structure–activity relationship, or SAR, optimization. Given molecules that bind to a target, an agent is asked to find molecules that bind more strongly. The current environment focuses only on binding—not metabolism or other drug-like properties. An agent receives a catalog, a target, and a budget of 100 molecules, but no additional information. It proposes readable-language hypotheses about why certain molecules work or fail, tests those hypotheses within its budget, and uses the results to choose subsequent molecules.

Ian says Leash has now run roughly 90,000 agent rollouts. Different language-model agents use different search policies: some spend their budgets exploring broadly, while others repeatedly investigate local modifications. Because Leash has measured the full environment rather than only the compounds an agent happens to test, the researchers can compare an agent’s result with the best molecule available anywhere in that landscape. That makes the benchmark more than a simple score. It allows the team to study scientific decision-making itself: whether an agent gives up too early, retests the same molecule despite its limited budget, or chooses exploration over exploitation. One notable result is that stronger models abandon unproductive chemical “neighborhoods” more quickly. Weaker models keep making small changes to low-scoring molecules because they remain attached to a plausible hypothesis, whereas stronger models recognize that the whole line of investigation is unlikely to work and move elsewhere. Ian says the researchers do not yet know exactly what internal reasoning produces that behavior.

The agents do not merely output molecules. Their transcripts contain explanations such as “this ring is good” or “this particular nitrogen is painful,” along with plausible medicinal-chemistry stories about why a modification should improve binding. Ian says the team plans to release many thousands of these transcripts so people can inspect the agents’ reasoning and failure modes, potentially by asking other language models to analyze large collections of rollouts. The hosts ask whether humans have been given the same task. Ian says only a small number of medicinal chemists have reviewed the outputs so far, but their reaction was that the agents’ proposed strategies were plausible. The group then considers turning the task into a game—a “medicinal chemistry captain” experience in which a person spends a limited experimental budget. Ian cautions that the pre-measured environment restricts players to compounds that were actually tested. A useful game would therefore need a smaller, understandable set of building blocks or modifications, and it would have to explain the available chemical options meaningfully.

Asked which models perform best, Ian says the current champion is “after six,” with Sonnet 5.5 and “Saul 6.1” also performing well, as those names appear in the conversation. He says the version five model was both expensive and not very performant, while the team has also tested smaller open models, including Qwen-like systems, GLMs, Kitties, and DeepSeek models. One DeepSeek version performed “not that bad.” The performance plot resembles an ARC-AGI-style chart, with a range from poor models to stronger ones. Ian explains that Leash began supervised fine-tuning work the previous summer because frontier models were initially poor at handling SMILES, the string notation used to describe chemical structures. A SMILES string represents a molecular graph as a one-dimensional traversal, and the same molecule can have multiple valid SMILES strings depending on where the traversal starts. At the time, models struggled even to convert a molecule into a non-canonical SMILES representation. Around February, the team found that models had begun working with chemistry meaningfully enough to support the iterative agent experiments.

The hosts point out that successful SMILES manipulation could indicate that a model is internalizing something about the physical structure of molecules. Ian is cautious about the wording: the agents are not changing their weights during these rollouts, so he cannot claim that they are learning in the training sense. However, they are clearly using information about previously tested molecules to choose better molecules later, and removing that molecular information substantially harms their decisions. The hosts ask whether Leash has compared the language-model agents with a Bayesian hyperparameter-optimization algorithm as a deterministic baseline. Ian says they have not, largely because they expect the chemical landscape to be too spiky and high-dimensional for that approach, while acknowledging that this is currently their “party line” rather than a demonstrated result. The central capability they are trying to measure is not merely optimization performance, but how an agent spends a fixed experimental budget and updates its strategy from feedback.

Ian describes the broader framework as a lab in the loop with “amortized latency.” In a conventional lab-in-the-loop system, each rollout requires a new physical experiment, making the process slow, expensive, and limited in counterfactual density. Leash instead records an environment once through multiplexed experiments, then allows effectively unlimited virtual rollouts with immediate feedback. The tradeoff is that agents can choose only among the options Leash measured in advance, rather than freely inventing any experiment. Leash has screened about 1,000 proteins against several chemical libraries since its launch. The team is currently producing roughly two new environments per day and hopes to reach around 20 per day, potentially by the beginning of 2026 or 2027 as stated in the conversation. Ian says the company is looking for other biological and scientific questions that can be expressed in the same format, especially problems involving limited-budget decisions such as when to stay in a region and when to move. They do not yet know whether reinforcement-learning progress in these environments will transfer to new biology, but they intend to test that, potentially using a covering set of environments and adding more structural tool calls.