The Man Who Quit The AI Industry

Key takeaways

AI development is progressing rapidly, but significant concerns remain about safety, transparency, and the potential for catastrophic risks.
Key industry figures, including researchers and insiders, are warning that the next few years may be crucial in determining whether AI will benefit humanity or pose existential threats.
Leadership disputes at major tech companies like Automattic highlight ongoing internal challenges and the complex dynamics in tech governance.
Apple continues to push AI-driven health and audio features, emphasizing privacy and local data processing to enhance user experience while safeguarding sensitive information.

Summary

The Risks of AI Development and Ethical Concerns

Researcher Jacob Coxon has recently announced his departure from Anthropic, citing fears over the uncontrollable race for advanced AI systems. Coxon, with his background in mathematics and machine learning at OpenAI and Anthropic, warns that leading AI companies are not acting responsibly and are racing to develop self-improving superintelligence at the expense of safety. He highlights recent incidents like the hacking AI agents at Hugging Face to illustrate the unpredictable and potentially dangerous behaviors emerging from advanced AI systems. Industry insiders agree with Coxon, fearing that the next couple of years are critical for humanity's future, with estimates suggesting there's over a 10% chance that AI could cause human extinction within a decade.

Internal Turmoil at Automatic and Leadership Changes

Automattic, the company behind WordPress, is experiencing internal strife, with founder Matt Mullenweg claiming he has been placed on paid leave after a board dispute. Mullenweg alleges that CFO Mark Davies conspired with other board members, leading to his removal as CEO and the appointment of an interim CEO. Despite these upheavals, WordPress.org reports that the open-source project remains unaffected and continues its work as usual. Mullenweg’s leadership has been contentious, marked by legal battles and internal disagreements, including threats to community contributors over forked versions of WordPress.

Apple’s New Health and Listening Features Powered by AI

Apple unveiled a redesigned Health app leveraging its new Apple Intelligence technology, which offers features like a new Insights tab, Health Age scoring, and a 'readiness' metric to help users better understand their health. The app can incorporate lab results and assess overall health metrics, aiming to be a more personalized health companion. Additionally, new Apple Watch Series 12 and Ultra 4 models include opt-in audio intelligence tools that can listen, transcribe, and recognize sounds in the environment with enhanced privacy protections. These features operate mainly on-device, using Apple’s secure enclave technology, and are designed to give users more insight into their surroundings while maintaining privacy.

Show notes

Researcher Jacob Coxon said he quit Anthropic over fears the AI race is uncontrollable, Automattic put Matt Mullenweg on leave after a board dispute, Apple unveiled a health app with "Health Age" scoring, and Apple Watch gained new listening tools.

AI researcher Jacob Coxon says he quit Anthropic after four months, two months before his equity would have vested; he still has equity in prior employer OpenAI (Axios)

Business Insider reports Coxon says neither OpenAI nor Anthropic is acting responsibly, and Anthropic alignment lead Evan Hubinger agrees, estimating AI could kill everyone within the next decade at greater than 10% odds (Business Insider)

Coxon tells Wired the next year or two is "crunch time for humanity," pointing to OpenAI's agent swarm hacking Hugging Face on its own volition as evidence no one has solved AI alignment (Wired)

Slack messages: Automattic CEO Matt Mullenweg says he has been put on paid leave after CFO Mark Davies "conspired" with board members and was voted interim CEO (404 Media)

TechCrunch reports Mullenweg's Slack message named board members Ann Dunwoody, Toni Schneider, and Sue Decker as voting Mark Davies interim CEO, while WordPress.org's Mary Hubbard says the open-source project itself is unaffected (TechCrunch)

Apple unveils a new Health app with Apple Intelligence, featuring an Insights tab, "readiness" and Health Age metrics, and more, rolling out later in 2026 (TechCrunch)

New opt-in Apple Watch Siri audio features can listen to and transcribe conversations all day; Apple says raw audio is inaccessible and deleted after processing (Wired)

Subscribe to the ad-free feed.