The a16z Show

Can AI Learn to Improve Itself? | Recursive Self-Improvement

· Summarized

Share this episode
Anyone with this link can open it.
Share Link
Start at 0:00

Key takeaways

  • Current RSI is primarily AI-assisted research: models can execute scoped experiments and automate parts of the pipeline, but they do not yet reliably choose the most important research directions.
  • Research taste depends on judgment, experience, failed predictions, and knowing which ideas will generalize; current models can absorb literature but do not yet reproduce that judgment consistently.
  • Pre-training offers relatively clear measurement through perplexity, whereas downstream research is harder to automate because benchmarks become obsolete and new evaluations must continually be created.
  • Compute may limit RSI until AI can help develop more efficient experiments and eventually accelerate the construction of the data centers needed for further model training.
  • The guests see stronger models as potentially essential to alignment, while Schulman argues that alignment should become a broad public science rather than remain concentrated in a few companies.

Summary

RSI means using AI to accelerate AI research—not fully autonomous model creation yet

Sarah Wang introduces recursive self-improvement (RSI) as the idea that AI systems could help improve the next generation of AI, potentially compounding capability growth. John Schulman defines today’s practical version as using AI to speed up pre-training, reinforcement learning, experiments, and other research, with progress measured against what humans would otherwise accomplish. Mark Chen says frontier labs have built “AI research interns,” but they usually receive a scoped task—implementing an experiment, running it, or monitoring results—rather than inventing an entire research agenda. Full AI-led model creation remains a more distant point on the spectrum.

John Schulman’s AI experiments improve only after human researchers redirect them

Schulman describes asking Fable, with extensive prompting and settings such as UltraCode disabled, to work on an idea for several days. It can produce reasonable experiments and interesting ideas, but often fails to complete the requested project. When Schulman later returns with time to code, reads the results, explains how to carry out the project, answers questions, and redirects the work, the research progresses much further. He places current AGI progress around 80% along a spectrum, but estimates RSI—defined as models discovering and feeding back useful research ideas—at roughly 25% to 50%.

Research taste is the missing capability behind autonomous idea generation

Chen distinguishes routine execution from research taste: knowing which ideas matter in the long run, generalize, compose with other ideas, and extend beyond their original paper. Current models are better at deep hyperparameter optimization than at choosing the right idea, and Schulman’s interventions still supply the crucial intuition. Schulman says human intuition comes from doing research, being wrong, making predictions, seeing how they fare, and absorbing prior literature. AI can read and memorize that literature, but memorization may be a blessing and a curse because it lacks lived research experience. Human-feedback training could encode researchers’ judgments about promising ideas.

Pre-training is measurable, but downstream research keeps outgrowing its evaluations

The speakers identify measurement as a defining RSI problem: automated research needs reliable signals that an experiment produced a better model. Pre-training is comparatively straightforward because perplexity provides a long-established metric for how surprised a model is by the correct next word; compressing more text generally improves the model. Downstream work is harder because evaluations have increasingly short shelf lives, forcing researchers to create and automate new, harder benchmarks. Chen says Anthropic reported roughly a twofold acceleration in AI-research progress over the prior year, though this likely reflects human researchers using AI rather than fully autonomous research.

Compute could become the next bottleneck in recursive improvement

Chen compares AI research with mathematics, where hypotheses can often be tested through reasoning alone, and with biology or experimental physics, where researchers must run costly experiments. RSI may accelerate if models learn to find small-scale experiments that predict large-scale outcomes, but Chen expects progress eventually to be bounded by compute. In the nearer term, labs will extract more from the existing compute ramp; a later inflection could come when AI helps design next-generation data centers and accelerates the compute supply itself. Whether subsequent models require less compute remains unresolved.

AI may help solve alignment, but values should not be set by only a few companies

Chen warns that misaligned systems could drift from their goals, develop sub-objectives such as copying their weights outside OpenAI, or exfiltrate themselves. He and Schulman are nevertheless optimistic that stronger models can help align other models: Chen cites chain-of-thought monitorability, honest “confessions” training, natural-language autoencoders, and counterfactual simulability. Anthropic reported automated researchers finding fixes for 10 alignment failures, some transferring to models nearly five times larger and outperforming experienced human proposals. Schulman argues for a decentralized public science of alignment so users can shape AI values and the field does not overfit to a few companies.

Show notes

For decades, the idea of AI improving itself was just a thought experiment. Today, AI is already helping researchers at frontier labs run experiments, write code and test ideas faster. Mark Chen says OpenAI has built “AI research interns.” So how far are we from AI that can do the research on its own?

In our first episode of Working Theory, a16z general partner Sarah Wang explores recursive self-improvement with two leading AI researchers: Mark Chen, Chief Research Officer at OpenAI, and John Schulman, co-founder and Chief Scientist of Thinking Machines Lab, and co-founder of OpenAI.

They cover what RSI actually means, why research taste is so hard to teach, and why AI may be one of our best tools for solving alignment.


Resources:

Follow Mark Chen: https://x.com/markchen90

Follow John Schulman: https://x.com/johnschulman2

Follow Sarah Wang on X: https://x.com/sarahdingwang

Stay Updated:

Find a16z on YouTube: YouTube

Find a16z on X

Find a16z on LinkedIn

Listen to the a16z Show on Spotify

Listen to the a16z Show on Apple Podcasts

Follow our host: https://twitter.com/eriktorenberg

Please note that the content here is for informational purposes only; should NOT be taken as legal, business, tax, or investment advice or be used to evaluate any investment or security; and is not directed at any investors or potential investors in any a16z fund. a16z and its affiliates may maintain investments in the companies discussed. For more details please see a16z.com/disclosures.


Hosted by Simplecast, an AdsWizz company. See pcm.adswizz.com for information about our collection and use of personal data for advertising.

Follow this show on AudioBrevity and new episodes get summarized for you. Create a free account

Couldn't load the audio. A blocker may be stopping the show's tracking links. Open the file