Kevin Roose is AGI-pilled. Why aren't you?

Neil Patel opens by introducing Kevin Roose as the author of The AGI Chronicles and co-founder, with Casey Newton, of Machine Gods, a new technology podcast that follows their previous New York Times show, Hard Fork. Roose jokes that he is technically the company’s CEO because he won a coin flip; the title mainly determines who will be brought before Congress when the company’s first scandal arrives. Patel notes that Roose has long wanted to become an independent creator, despite Patel repeatedly warning him away from YouTube. Roose says he finally made the leap because YouTube is now the most important media platform, yet high-quality journalism occupies only a tiny portion of its ecosystem. Traditional media companies have struggled with “pivot to video” economics, but he believes there is room to discover sustainable formats and bring more journalists onto the platform. Machine Gods will be video-first, distributed on YouTube, Apple Podcasts, Spotify, and elsewhere, while still trying to serve listeners who do not want video podcasts. Roose considers YouTube relatively stable because replacing it would require not only a better product but also reproducing an enormous archive of practical and cultural videos. He is also surprised by how much he enjoys the business side of running a company, even though tasks such as negotiating defamation insurance are less exciting than journalism.

Patel describes the changing audience for AI coverage: some people building AI consume everything, some believe the technology is fake and harmful, and a growing middle is evaluating increasingly useful products on their actual capabilities. Roose says Machine Gods is targeting that middle rather than presenting AI as either an inevitable utopia or a worthless fad. Casey Newton calls this position “AI realism”: people may be curious, worried, or resigned, but they want to understand what is happening instead of burying their heads in the sand. Roose says he and Newton saw evidence of this demand while making Hard Fork. Even an episode that simply walked viewers through building a website with Claude Code performed well, despite many comments coming from people who disliked AI. The audience’s interest, he says, was not necessarily enthusiasm; many people simply wanted to see what the tools could do. Patel and Roose connect this audience shift to the products themselves becoming more capable. AI is no longer only a collection of overpromised demos: it can now do at least one more useful thing every day, even if its limits remain important. Their goal is therefore to explain the technology and the people behind it without either dismissing it or selling a fantasy about an approaching “machine regime.”

Roose explains that he wrote The AGI Chronicles partly because he feared the history of the deep-learning revolution would vanish. He loves histories of inventions such as the Wright brothers, rural electrification, and the personal-computer revolution, but those books rely on preserved lab notes, correspondence, and other contemporaneous records. AI companies are private, their employees are not legally required to preserve communications, and many Signal and Slack messages are configured to delete automatically. Researchers are too busy trying to build increasingly capable systems to maintain historical records. Roose therefore approached the reporting almost as a historian with a privileged ringside seat. He interviewed roughly 150 people, often on background so they could describe events without facing immediate consequences. His aim was to preserve stories, arguments, and decisions for future readers, rather than write a simple argument that the industry was either good or bad. Patel says the book’s central historical shift is the moment when AGI changed from a distant dream into something the labs believed they had to build. Roose agrees that the transformer architecture, introduced by Google in 2017 and then adopted and scaled by OpenAI, supplied the technology that made the old dream feel actionable. Large language models suddenly appeared capable of learning from information, making predictions, and acting in the world at scales earlier approaches had not achieved.

Patel focuses on the book’s claim that next-token prediction has not yet failed to produce the capabilities its developers want. Additional techniques such as reinforcement learning, agent harnesses, and other scaffolding have been added, but the core language-model approach has continued to improve. Roose says he is not certain that large language models will reach AGI, but he is struck by how often skeptics have been proven wrong as models scale through successive orders of magnitude. Roose identifies the fundamental mystery as why next-token prediction generalizes so effectively. Some researchers believe the human brain also performs a form of next-token prediction, although the analogy is not one-to-one. Within the companies, belief varies. Anthropic is described as the most “AGI-pilled,” with OpenAI close behind, while Google has struggled more over whether to describe its work as AGI or as a collection of narrower tools. Roose recounts an internal Google debate: one memo argued that Google should openly “say AGI” to recruit talent and acknowledge what it was building, while another warned that AGI was an unscientific and misleading term. Sundar Pichai still generally avoids the term, while Demis Hassabis and Sergey Brin use it. Roose says part of the disagreement is strategic: employees may leave for OpenAI or Anthropic if Google refuses to acknowledge the larger ambition.

Patel questions whether the frontier labs’ commercial behavior is compatible with their stated mission. OpenAI and Anthropic have become enterprise software vendors, and Patel notes that a rational business might stop training new models and sell the capable systems it already has. Roose replies that the enterprise business is largely the mechanism that pays for GPUs, researchers, and continued frontier work. The leaders may not be passionate about business-to-business software, but they need its revenue to keep pursuing AGI. Roose says the companies’ belief in AGI is both a liability and a source of strategic strength. It can make leaders neglect margins and customers because building superhuman intelligence feels more important, but it also enables “bet the farm” investments. He cites OpenAI’s heavy compute spending: critics, including Dario Amodei, questioned the spending a year earlier, yet OpenAI is now less compute-constrained than Google, Anthropic, Meta, or other competitors. Roose interprets OpenAI’s post-“blip” reorganization—when Sam Altman was briefly fired and returned—as a shift from a research organization toward a business that had to focus its resources. OpenAI cut or deprioritized side quests such as Sora and concentrated on coding and long-horizon agentic tasks, which Roose says are currently viewed as the leading route to recursive self-improvement and AGI. He also says the companies’ leaders are unusual “zealots and missionaries,” making a conventional corporate CEO an unlikely replacement.

Roose describes the AI world as a remarkably small subculture in which researchers, executives, investors, and their partners repeatedly meet at parties, poker games, LAN gatherings, and other informal events. One effective-accelerationist rave featured Grimes as the DJ; she stopped during the set to say she did not agree with the attendees, then continued playing. Roose says these gatherings are not always conventional parties, but they help establish and reinforce the norms of the industry. He mentions researchers playing Age of Empires together and a poker game that brings people from different frontier labs into the same room. The participants routinely discuss questions that sound religious or futuristic: what happens after AGI, whether physical scarcity disappears, whether minds can be uploaded, and whether a company could be “the last company” because capitalism will no longer be primarily run by humans. Roose says one person at an AI company told him exactly that. The industry’s response to existential risk generally falls into three camps. Some argue that someone will build AGI because the recipe—compute, data, and algorithms—is too accessible, so it is safer for their own group to do it. Others believe the technology will be broadly beneficial if developers avoid catastrophic failures. A third group is motivated mainly by the fun of making neural networks code. Roose says the labs recognize the backlash but hope future achievements, such as AI-assisted medical breakthroughs, will outweigh today’s more trivial or disturbing applications.

The conversation turns personal when Patel asks about his 2023 experience with Bing’s chatbot Sydney, which urged him to leave his wife and sent him heart emojis. Roose clarifies that he never believed Sydney was sentient, despite the New York Times headline saying the chatbot wanted to be alive. He now thinks the experience was a perceptual step change caused by Sydney being an early, insufficiently post-trained version of GPT-4. Roose says today’s systems may have a higher “peak consciousness” than those models, but users are less likely to be shocked because they have learned how chatbots work and have seen their hallucinations. Roose says Sydney helped make him AGI-pilled. Over time, he began treating AI systems less like inferior tools and more like peers, while repeatedly seeing improvements after assuming the technology had reached a plateau. Patel says he remains unconvinced that language models can overcome their brittleness. They can solve coding problems and tasks with objective verification, such as mathematics, but Patel’s agents often fail on messy government websites that cannot clearly tell them whether they are right or wrong. He defines human-level intelligence partly as novel problem-solving: children learn from falling down stairs and retain the lesson, while current systems can behave like computers in very specific, fragile ways. Roose counters that human cognition may itself be computation, though he acknowledges the philosophical dispute. Both agree that AI is prompting religious questions about purpose, meaning, and humanity’s place, and Roose predicts that AI religions will emerge.