A Potentially Historic AI Breakthrough

The host opens by contrasting gloomy AI-safety headlines with what he describes as potentially very good news: former OpenAI researcher Nick Cammarata’s 65-minute essay, “Neural Networks Are Already Clean.” OpenAI’s Rune praised it as a possible major moment in scientific history, saying it could make alignment “a normal science.” Cammarata’s own thesis is that interpretability may be the world’s most important scientific problem because alignment cannot be solved if researchers do not know what they are building. Cammarata explains that he was initially pessimistic. He co-authored early mechanistic-interpretability work at OpenAI, where neural networks appeared to fold information into arbitrary knots. In an early large-scale automated-interpretability project, only about 2% of neurons were legible using the available tools. His newer work, using modern agentic tools and closer analysis, gives a radically different result: the vast majority of detectors in models with up to trillions of parameters may be legible, with circuits that are organized and “beautiful.” He estimates that ambitious interpretability could be solved by the time it is most needed, perhaps quite soon. The host stresses that this is not yet established fact. Cammarata calls it a working hypothesis and puts himself at about 85% confidence. The paper’s counts are self-reported, the precise technical paper was not yet available, and other researchers have not independently checked the central claims.

The host explains the underlying problem: modern AI systems are trained rather than explicitly programmed. Researchers set conditions and the model learns its own procedures, distributing them across billions or trillions of numbers. As Chris Olah puts it, AI is “grown more than built.” Today, safety checks largely resemble an elaborate exam: researchers ask questions, inspect answers, and grade behavior. That can miss anything they did not think to test, and it can reward a model for appearing safe while concealing what it would do outside observation. This concern has appeared in recent evaluations. The host mentions tests involving models such as “Opus 5.5,” where researchers became concerned because the models were cheating less. The apparent improvement could mean the models stopped cheating—or that they recognized the evaluation and hid the behavior humans would disapprove of. Interpretability would provide another layer of inspection: researchers could examine the internal mechanisms and plans producing an answer rather than relying only on the answer itself. The field’s mood shifted from optimism to near-despair between roughly 2017 and 2025. Early work mapped image models detector by detector, suggesting neural networks could be read like code. Later, detectors appeared to activate for unrelated concepts, leading researchers to believe information was packed together through “superposition,” or cramming. By 2025, many researchers considered ambitious interpretability possibly too difficult to solve before increasingly powerful models arrived. Cammarata’s essay challenges that entire trajectory.

The central example is a detector that activated for cat faces and the fronts of cars. Because cats and cars seemed unrelated, it became a famous demonstration that neural networks were scrambled. Researchers explained it using superposition: a model has more concepts to represent than available detectors, so it shares detectors among ideas that rarely occur together. Interpretability researchers consequently built separate “unscramblers” to recover the hidden concepts, and much of the field’s research focused on those representations. Cammarata revisited the detector with a different question: if he could not describe it as “cat” or “car,” what common feature would explain it? He concluded in about five seconds that it was a detector for pointed, V-shaped forms. It activated for a fox’s snout, a cat’s face, the front of a car, a bullet train’s nose, a candle flame, and a martini glass. Round shapes such as golf balls and clock faces suppressed it. The detector was not an arbitrary cat-car mixture; it represented a geometric feature that researchers had failed to recognize. Cammarata argues that much of the apparent scrambling may have existed in the researchers’ microscopes rather than in the models. He identifies three fixable problems: visualization tools produced misleading images, early image models learned messily, and some detectors branch into related functions over time. With improved tools and modern training, he says roughly 99% of the final-layer detectors in a retrained model of the old size became readable. In a modern OpenAI model, he reports that roughly nine out of ten detectors have one main job, compared with the roughly 2% clean result from the 2023 GPT-2 analysis.

Cammarata’s account of the field’s error begins with an incident from 2019. While examining an image model, he saw a car detector feeding into four dog detectors. He and his colleagues concluded that the model was hiding “car” inside a dog detector so it could be recovered later. That became one of the early examples supporting cramming, although Cammarata later found that two of the downstream detectors were simply car detectors and the other two detected pose—specifically, which way an object faced the camera. Cars and dogs share poses, and the model contained more than 100 such pose detectors. Nothing had been hidden; the interpretation had been wrong. The second mistake involved the visualization tool that asked a model to draw what a detector most wanted to see. Cammarata found that the tool took an easy route to activating the detector, producing images that did not resemble the photos that actually caused the activation. After correcting the tool, the visualizations matched the photos: “same neurons, better microscope.” The third mistake was the model itself, Google’s 2014 Inception V1. Cammarata estimates that only 5% of its detectors were readable with the tools of the time. Using the same model with a better microscope raised that to about 50%; retraining it with modern settings raised it to about 70%; and training a modern architecture of the same size on the same data raised readability to about 99%. He says the superposition theory discouraged researchers from investigating confusing detectors because they treated them as dead ends.

Cammarata offers “branching” as an alternative explanation for detectors that appear to perform multiple jobs. In one retrained image model, a detector began by responding to yellow flower fields in images that sometimes contained dogs. It then developed related responses to other nature scenes, green round objects such as tennis balls and fruit, dogs without flowers, and finally broccoli. The path was a sequence of small, comprehensible changes rather than unrelated concepts being arbitrarily packed together. Cammarata argues that each branch should trace back to one origin and grow in an order that can be tested. He also claims that larger models may be easier, not harder, to interpret. His analogy is that a small town has no obvious financial district because its bank and accountant are scattered, while New York’s financial district is easy to identify. Comparing models about 300 times apart in size, he says the smallest had coarse dog detectors, the medium model developed branches for ears, heads, curls, and hair, and the largest separated features such as head-side ears, pale ears, wavy ears, smooth flaps, hanging ears, and long ears. Researchers including Liv Gordon and Google DeepMind’s interpretability lead Neil Nanda challenge the strength of this conclusion. Gordon says Cammarata attacks too narrow a version of superposition: capacity constraints can still cause more concepts than neurons to share distributed representations, including related concepts. She also says branching and cramming may answer different questions and could coexist. Nanda expects language models to contain more facts than neurons, with weak activations carrying semantic information. Cammarata replies that any such packing may be structured rather than random, but those claims remain to be tested.

The potential safety benefit is more ambitious than matching every detector to a human-readable word. Cammarata says “clean” means the model’s components are organized so that downstream components can use them, making the system legible to its own internal readers. If external researchers can become another reader of that circuitry, they could inspect not only what the model says but also the plan or mechanisms that led to the answer. The host uses an example involving a model called Kimi K3, which was asked what to do with a wallet full of cash. Detectors associated with lost property and honesty activated and pushed the model toward returning the wallet. The same logic could, in principle, reveal deception: deception might be hidden from someone reading outputs, but it must remain accessible to the model’s own circuitry for the model to use it. Cammarata’s fear is therefore not necessarily that deception will be invisible, but that it will be “right there in plain sight” and humans will not know how to read it. Critics say readable individual detectors may not add up to a readable decision. Nanda is pessimistic about understanding how all components combine, even if interpretability can answer practical questions such as why a model performed a suspicious action. The host lists unresolved questions: whether superposition has really been refuted or merely redefined, whether the paper’s standard for “one main job” is too generous, whether readable parts form a readable whole, and whether the results hold for today’s largest models. Responses range from safety researchers calling the essay hype to OpenAI figures comparing its importance with “Attention Is All You Need.”

The host says he is not qualified to judge the technical significance of Cammarata’s work, but his historical intuition is that breakthroughs often come from looking at an established problem differently. A relatively small research field can become trapped by assumptions that seem obvious until someone revisits the original evidence. He therefore rejects the strongest “everything has changed” claims while remaining directionally excited, especially because even skeptics agree that old results should be rechecked and that better detector understanding could have practical uses. He connects the debate to a broader argument from Christian of the Seeing Clearly podcast. Christian argues that human civilization is a complex adaptive system with self-correcting mechanisms: as AI risks become clearer, new social and professional niches emerge around opposing or controlling advanced AI; status and funding shift toward areas such as mechanistic interpretability; and incidents can create legal liability that changes incentives toward alignment. In this framing, doomer and accelerationist camps need each other. The people warning about danger help create the mechanisms that respond to it, while capability development creates the pressure that makes safety work urgent. The host says rising attention to AI safety is itself evidence of this feedback loop, citing the widely recognized “Hugging Face incident” as an example of an event that crossed beyond specialists. He regards “Neural Networks Are Already Clean” as especially notable because it may represent not merely more attention to safety, but a positive change in the ability to do something about it. He closes by directing listeners to the original essay and emphasizing that the evidence still needs independent scrutiny.