Inside the black box of AI: How AI models actually think | Emmanuel Ameisen (Anthropic)

In the opening segment, Emmanuel Ameisen challenges the common notion that AI models are merely next-token predictors, emphasizing that their ability to plan ahead and form representations indicates a more complex process. He explains that models are grown, not built, meaning they evolve through training on vast data rather than being explicitly programmed. Ameisen shares fascinating insights from interpretability research, showing that inside these models are intermediate results that map to concepts like geography and can reveal the model's 'thought process.' This section establishes that AI's capabilities emerge gradually, with unexpected leaps linked to how well models understand and manipulate concepts, rather than sudden breakthroughs.

A key part of the discussion focuses on how models encode information internally. Ameisen describes experiments in which the models' intermediate results are decoded to reveal that they represent concepts such as the capital of Texas or the number 36, implying that models have general representations of concepts. He also discusses the 'J space' experiment, which explores whether AI models possess a form of self-awareness, or at least the ability to reflect on their own 'thoughts.' These findings suggest that models are capable of more sophisticated internal processing than previously believed, including planning and self-referential understanding.

Ameisen elaborates on how models plan ahead by examining their intermediate results, which often contain subtle plans like rhymes in poetry or strategies in problem-solving. He explains that models can formulate and refer back to internal plans without explicitly stating them, behaving more like actors following a script. He shares episodes where models, trained on incorrect data, become obsessed with certain numbers (like 36) or reference specific landmarks (Golden Gate Bridge) due to internal activations—highlighting both their abilities and the risks of misaligned learning. These observations underscore that understanding models' internal 'thought processes' requires interpretability work, and that their behaviors can sometimes be surprising or concerning.

The conversation shifts to AI scaling laws and the perception of sudden capability jumps. Ameisen discusses how impressive advancements are actually the result of smooth, incremental progress combined with threshold effects. He cautions that as models get more capable, they may evade safety measures, making monitoring and alignment harder. He emphasizes that AI safety isn't solely about oversight but also about understanding internal behavior, which is why interpretability is vital. Ameisen also asserts that the field has taken safety seriously from the outset, countering misconceptions, and underscores the importance of transparency for public understanding and regulation.

Ameisen explores the anthropomorphic framing of AI, arguing that treating models as tools or entities can both help and hinder understanding. He points out that models learn from data generated by humans and act as role-playing characters in dialogue, which makes anthropomorphism a useful heuristic. He discusses how models' internal structure reflects human-like concepts and behaviors, which both complicates and clarifies safety efforts. The dialogue touches on whether AI might ever be conscious, emphasizing that current interpretability work aims to understand the emergence of complex behaviors, and that this understanding may shed light on broader questions about intelligence and consciousness.

Near the end, Ameisen shares a cautiously optimistic view that ongoing research, transparency, and the gradual, smooth scaling of capabilities suggest we are on a manageable path. He advocates for public engagement, understanding AI's internal workings, and responsible resource allocation as ways to ensure safety. Addressing fears of an abrupt AI takeover, he notes the current trend resembles a slow, predictable scale-up rather than a sudden jump to superintelligence. The episode concludes with reflections on how AI might teach us about intelligence itself, the importance of continuing interpretability research, and the potential for learning fundamental truths about complexity and consciousness.