Academia is for Ambition — Alex Zhang, MIT

Alex Zhang explains that GPU Mode began as “CUDA mode,” a Discord started around 2023 by Mark, Andreas, and Jeremy Howard. Its original purpose was to teach people how to write GPU kernels through lectures and competitions. Alex joined while interning at Snapchat, where he was bored with his work and happened to be exploring whether specialized kernels could accelerate Infinite Attention, a Google paper. Although that Snapchat project went nowhere, the community introduced him to Mark, Matei, and others who were building the ecosystem. Mark’s “Popcorn” idea eventually became the GPU Mode leaderboard. The premise was that GPU optimization resembles competitive programming: there are many possible problems, but a relatively small set of recurring kernels and optimization techniques. With enough examples, the community hoped to scale and automate kernel development. This led to KernelBench, which asks whether language models can generate GPU kernel code. Alex says GPU Mode differs from more serious ML-performance competitions because beginners can follow its lectures and participate mainly to learn, whereas established benchmarks often attract only major labs and companies. He now occasionally helps with lectures but is less involved than before.

The hosts ask about recent claims that AI-generated kernels can make systems such as Terra and Luna roughly 80% cheaper. Alex says that on the GPU Mode leaderboard, most recent solutions are AI-generated. However, a member known as Gounderst illustrates the remaining value of expertise: he used AI too, but guided it with strong GPU intuition, and his kernel was essentially the only top-ten solution that Alex found stable in real end-to-end systems. This exposes what Alex calls GPU kernels’ long-standing verification problem. KernelBench has had reward hacking from the beginning: a submission can score well under the benchmark while failing to behave reliably in a larger system. The shorter AI-generated programs can look impressive, but leaderboard speed alone does not establish practical usefulness. Alex compares this to mathematical proof systems and other AI applications: models may produce plausible outputs, but people who understand the domain remain valuable as verifiers, guides, and problem decomposers. For a kernel, there is often a theoretical “speed of light” based on hardware limits, data location, and whether data begins in CPU memory, GPU DRAM, or HBM. Matrix multiplication makes this estimate relatively easy, but real kernels may have unavoidable bottlenecks and rarely approach the theoretical limit. Alex also distinguishes isolated-kernel speed from system-level speed: slowing one operation can be worthwhile if it keeps data cached for a later operation, a fusion or “mega-kernel” issue.

The conversation turns from kernels to research judgment. Alex says research taste is partly developed through opportunity: he was fortunate to work at Princeton and later with Omar at MIT. But he argues that the strongest academic work often starts with problems industry is not prioritizing. PhD students have a rare advantage because they can pursue ideas without the bureaucracy and product constraints of a major lab, so they should take large, risky bets even though many will fail. He uses Recursive Language Models (RLMs), SWE-bench, chain-of-thought, ReAct, and related work as examples. These ideas can initially look trivial or confusing—people asked why RLMs were not merely sub-agents—but that reaction can signal that the work is opening a new design space. SWE-bench was initially dismissed as impossible and irrelevant until systems such as Devin made it an obvious target for improvement. Alex similarly cites Eric Zelikman’s work on ideas such as “star” and “quite star” as simple concepts that nonetheless changed what researchers considered possible. He makes the same point about “Jev.” Rather than dismissing it as an ordinary classifier, Alex sees it as a challenge to the assumption that a language model must be an autoregressive text-to-text decoder. A language-model backbone could produce a different output space for a narrower task, such as binary classification, and avoid paying the roughly 400-times cost of asking a full language model to do something simple.

Alex argues that a smaller lab cannot sensibly compete with frontier companies by copying their autoregressive decoder strategy. Frontier labs control much more compute and data, so a new entrant needs a different advantage. He believes this is why “neo labs” and side projects can be valuable: a constrained team may be forced to explore a design the largest labs would not risk funding while their current approach is still working. He is excited by models that change the usual language-model tradeoffs, including Jev and loop transformers. Jev appears to offer very fast inference for tasks such as classification, although Alex does not speculate confidently about its exact implementation. He has seen suggestions that it might involve diffusion or parallel decoding, but says the more important achievement is whatever training procedure made it work. An open-source replication may use a model such as Qwen, but Alex expects it to perform worse because the original training recipe is the difficult, non-obvious part. Calibration is another underappreciated issue. A model asked how confident it is may output “43%,” but that number can simply be another likely next-token prediction rather than a grounded probability. Alex thinks calibration could be useful when treating these systems as fast classifiers, and that synthetic data might make it relatively straightforward to compare predicted confidence with known ground truth. He also likes the use of fast models in games: models can retain language understanding while meeting low-latency constraints that ordinary language models handle poorly.

Alex defines an RLM as a harness in which code is the central interface. The model can call itself or other models as sub-agents through code, while the context it is manipulating is stored in a persistent environment such as files or a Python or Bash runtime. The key difference from ordinary coding harnesses is not merely a longer context window; it is that the model can programmatically inspect, transform, delegate, and revisit a shared context rather than issuing a fixed sequence of tool calls. He argues that existing harnesses such as Claude Code, Codex, Pi, and similar systems are mostly variations on the same loop: append the trajectory to the prompt, make a tool call, append the result, and repeat. This works for familiar coding workflows but becomes awkward for long-context or multi-agent tasks. An RLM can instead offload context and ask a sub-agent to solve a local problem. That makes each individual model call more likely to be in distribution, even when the overall task is unfamiliar. Alex describes this as “local in-distribution” behavior. A retrieval task and an aggregation task may look entirely different at the top level, but if both are decomposed into similar local programs, an RLM can learn the same strategy for each. In experiments described in his blog, models trained on short tasks generalized to tasks 8–30 times longer, and strategies learned on one type of task transferred to another. He says the same inductive argument could connect competitive programming and GPU optimization: the sub-agent’s domain knowledge differs, but the high-level plan—generate candidates, delegate promising ones, test them with a verifier, and evolve them—can be shared.

Alex became involved with Prime after its researchers published a post arguing that RLM-style harnesses could be important. He says Prime understood that the point of the RLM work was not just solving long-context problems, but exploring more opinionated harness designs. Prime Agent is built on Pi Mono but restricts the model’s direct tool to an IPython environment. Other tools are loaded as Python modules or Bash scripts, so the model writes code to invoke and combine them. Prime Agent also incorporates a “continual harness” concept developed by another PhD student who worked on language models playing games. The harness can let the system modify parts of its own setup, including its available skills, sub-agents, or system prompt. Prime Agent includes an agent-to-agent communication framework for systems that spawn many sub-agents, with design choices about which agents can communicate and how. It also supports persistent sub-agents that can survive beyond the original agent’s runtime, be prompted again, and remain inspectable. Alex contrasts this with Codex-style sub-agents, which he finds too ephemeral for long-running work. His practical solution is to externalize state to the filesystem and trust the model to write code that manages that state. Prime is also training a model for this kind of workflow, although Alex is not involved in that training and is pursuing other PhD projects. He says that better training around a harness could eventually make RLM workflows much more efficient, rather than merely spending more tokens on a weakly structured swarm.

The hosts discuss outside work influenced by RLMs. Harvey reportedly post-trained an RLM for legal work involving large document collections, and Alex calls the results exciting because legal questions often require sifting and reasoning beyond straightforward retrieval. Headlong, associated with Andy Kowinsky, is another example: its system is designed to “think persistently,” continuing to work on problems even when it is not actively being queried, while controlling token costs. Alex also mentions other RLM-inspired harnesses and Arc AGI 3 systems that use shared context, code, or agent composition. Alex is cautious about large swarms. OpenAI reportedly used 10,000 agents over 88 hours on a difficult problem, consuming around 130 billion tokens—roughly $40 million at public pricing, according to the hosts. Alex says it is remarkable that the industry can even spend that amount on an unsolved problem, but a swarm does not automatically make its search efficient. Much of the exploration may be useless, and the field still needs to determine when to use compaction, a single agent, a structured harness, or a swarm. He distinguishes OpenAI’s work from the public presentation of Kimi’s agent swarm and from Gemini’s dynamic workflows. Alex thinks OpenAI clearly trained its system to converge toward a goal; the fact that it can justify such a large spend suggests a real engineering accomplishment. By contrast, he found Kimi’s description interesting but was not convinced it demonstrated the ability to solve novel problems. The core research question is how to make agents collaborate rather than simply generate a large amount of loosely coordinated output.

Alex says current models are still not good enough to perform a relatively simple job consistently over a month. He believes this is a “skill issue” that a better harness could address without requiring a new frontier model. The goal would be a general system that can be treated like an intern: given a loosely specified task, it can explore, maintain state, use its strong coding and mathematical abilities, and keep working reliably. He sees automated data-science research as an early example, even if the immediate results are mainly optimization of easy parts rather than major discoveries. This connects to open-endedness, which Alex describes as related to unsolved mathematics and evolutionary search. The challenge is not only generating a huge amount of candidate work but deciding which results are genuinely interesting when there is no fixed objective function. A system might generate possible goals as well as solutions, but researchers still need a way to sift through the resulting “slop” and find the valuable ideas. He views this as a more fundamental alternative to machine learning’s usual setup, in which the objective is defined in advance. Alex closes by saying that academia gives him room to pursue these bets. He may have 10 or 15 ideas at once, most of which are bad, but when one becomes compelling he likes to drop other work and pursue it intensely. He welcomes collaborators who are opinionated, have thought through their position, and bring a concrete idea—not just a request to “pick his brain.” He also says his movement from GPU kernels toward harnesses reflects where he sees the most unexploited innovation, though he remains interested in model-level work and applications to science.