Roblox’s approach to tripling AI code review acceptance rates

Host Brian Houck begins by asking Andrew Swerdlow, Roblox’s VP of Engineering for AI Acceleration and Core Platforms, about the gap between AI that can generate code and AI that can safely act on its own. Swerdlow says Roblox is seeing many AI-generated pull requests, but overall lead time to production has not changed much. The generated code gets bottlenecked by the rest of the software development lifecycle, especially code review, deployment, testing, and production debugging. Swerdlow argues that AI must be integrated across the entire lifecycle for organizations to realize its full promise. Before an agent can receive permission to deploy, the organization needs confidence that it will not introduce regressions, outages, or “slop” into production. The required guardrails are largely familiar engineering investments: strong unit-test coverage, integration testing, production canaries, issue detection, and automated rollback. In his view, companies that are moving fastest with AI are often companies that had already invested for years in continuous deployment and testing infrastructure, because those systems provide the safety boundary needed for autonomy. Discussing humans in the loop, Swerdlow says people must remain accountable for the products and code they create, but reviewing every line of AI-generated code may be the wrong place to put human attention. Instead, humans should teach systems what good and bad production code looks like. The goal is to move from reviewing every output to establishing rules that let agents make more decisions safely.

Houck asks what an AI agent needs to know about an organization that a frontier model does not already know. Swerdlow gives the example of a company deliberately remaining on an old Python version or library even though the broader industry has moved on. A general model might recommend upgrading, but it would not know that Roblox had consciously decided the migration was not yet worth the work. Agents therefore need to understand the organization’s actual conventions, constraints, and historical decisions. Roblox spent roughly a year and a half building a system that extracts this knowledge from about 20 years of company history. The system examines commit histories, code-review comments, Slack data, and other sources, then converts useful signals into guidance for agents. Swerdlow says code-review comments turned out to be the highest-fidelity source because they contain engineers explicitly telling colleagues how code should be written inside Roblox. They are also noisy: some are mere “nits,” use inconsistent language, or concern temporary details. The company processes the comments with an LLM, normalizes them into rule-like statements, discards material that cannot be reliably parsed, clusters similar comments in a vector database, and turns the clusters into rules. Roblox found that historical guidance decays: comments from 20 years ago are usually not relevant, while the most useful window is roughly the last 18 months to two years. Each repository receives a UI showing automatically extracted rules, called “exemplars,” along with citations. Repository owners can see how often a rule was cited, who cited it, and add approved rules to the repository as a YAML file for use in code review and code generation.

Houck asks whether automating code review could eventually dry up the very source of engineering knowledge Roblox is mining. Swerdlow says that if an AI reviewer became so accurate that people accepted all of its feedback, he would view that as a successful endpoint rather than a problem. Roblox is not there yet: in the episode’s current data, roughly 60%—nearly 70% that week—of the code-review agent’s comments were accepted by humans. The remaining comments may be wrong, unnecessary, or incomplete, leaving room for improvement. For comparison, Swerdlow says human-written code-review comments are accepted about 50% to 55% of the time, consistent with published research. Roblox’s comparison includes before-and-after observations and holdout repositories that have not adopted the AI reviewer. Based on those results, he says the AI reviewer now produces more acceptable comments than humans on average. That performance came through iteration rather than a single prompt. Roblox’s first approach simply ran an LLM on a pull request and asked for feedback; only about 20% of those comments were accepted. The company improved the system over the following year and a half by extracting, normalizing, clustering, ranking, and human-approving organization-specific exemplars. Swerdlow says Roblox currently relies mainly on repeated citation frequency rather than a formal seniority score. The company did build an expertise classifier using signals such as an engineer’s tenure and experience in a repository, but repeated citation was a stronger signal. People can still inspect who cited a rule and use their judgment about whether those people are authoritative.

Houck points out that code-review comments mostly capture decisions about artifacts that already exist, while the reasons behind those decisions often remain in engineers’ heads or in hallway conversations. Swerdlow agrees that documentation is difficult at Roblox and at other companies he has worked for. He says code and the systems surrounding code are usually more accurate sources of truth than formal documentation. Slack can provide useful context, but code-review comments are more concrete, attributable, and directly connected to implementation behavior. Roblox supplements its mined rules with explicitly authored ones. Engineers can use a playground to write a rule in natural language, provide one-shot examples, and test it against existing pull requests or purpose-built cases. This is especially useful after an incident: rather than merely adding a test, alert, runbook entry, or piece of tribal knowledge, the team can encode the lesson as an exemplar that will be injected into future code reviews. Swerdlow describes this as a new capability because the rule remains active even after the people involved leave the company. The team is also experimenting with incident reports and postmortem follow-up actions, while deliberately trying to avoid noisy guidance. Swerdlow says Roblox wants rules that prevent meaningful defects from escaping into production, not rules that generate endless stylistic feedback. The result is a combination of implicit rules extracted from recent engineering behavior and explicit rules created to prevent known failures from recurring.

Houck argues that code review is valuable not only for finding defects, but also for transferring knowledge, making work visible, and creating shared understanding among engineers. If AI removes much of that interaction, some of that value disappears. Swerdlow responds that the most important information is likely to be the intent of a change, the architecture used to achieve it, and the specification for what is being built. The details of function names and syntax may soon sit below the agentic abstraction layer. The conversation turns to design documents. Swerdlow credits Malte Ubl, formerly associated with Google’s design-document culture and now at Vercel, with the idea that a design document may be unnecessary when an AI-assisted prototype can be built faster. Swerdlow agrees that if the purpose of a design review is to get feedback on a system’s contours, rapidly creating and evaluating a prototype may be more useful than writing a long document. He distinguishes a specification from a traditional design document: a spec is tied more closely to the intended implementation, architecture, component boundaries, and separation of concerns. Roblox has not settled on one canonical specification format, but Swerdlow says specs are cheaper to revise than even fast AI-generated prototypes. He also says code reviews are increasingly being replaced by specification reviews, with engineers commenting on the intended system before the agent spends significant tokens implementing it. A good spec should explain what the system is trying to achieve, how its components fit together, and where responsibilities are separated.

Swerdlow says experienced engineers currently get more leverage from AI tools because they have built large systems, encountered operational failures, and developed “scar tissue.” Less experienced engineers may trust an agent too readily, accept a superficially working design, and miss scalability or maintainability problems. Senior engineers are more likely to question an agent’s first answer, probe boundary conditions, and iterate toward a durable design. He expects skepticism, pushback, and the ability to judge long-term quality to become skills that junior engineers must learn. He predicts that coding interviews will move away from isolated whiteboard problems. Some companies are already testing candidates on how effectively they use AI to build a system from zero to one. The evaluation may focus on process and efficiency: one engineer might “vibe code” through many prompts and consume a large number of tokens, while another plans, iterates on a specification, and produces an elegant implementation in one pass. Converging on a working answer will become more commoditized; how the candidate reaches a maintainable, product-appropriate answer will matter more. Swerdlow says “AI-native” may have a limited shelf life because eventually every engineering organization will claim the label. At minimum, it means using AI deeply throughout engineering work, but the stronger version involves autonomy supported by trustworthy systems. Roblox measures AI-written code and has found that most of its code is generated by AI. However, Swerdlow says the more useful question is not simply how much code AI wrote, but which generated code was accepted, survived in production, avoided rollback, and required no human intervention.

Roblox built a fingerprinting system that records generated code across tools including Cursor, Copilot, Codex, and Claude Code. It hashes each generated line and compares those fingerprints with lines that reach the GitHub repository and production codebase. If a human changes even the capitalization of a line, Roblox counts it as human-authored, giving people the benefit of the doubt. Swerdlow says the raw percentage of AI-written code may eventually lose importance, but the underlying data enables richer analysis: which model version generated a line, how long the line survives, whether it is rolled back after a bug or incident, how many turns it took, and how much the generation cost. Roblox’s prompt and session data showed that about 60% of token usage goes toward understanding code, asking questions, or related knowledge-management work, rather than generating code. Only about 40% is used to generate code, and an even smaller portion survives into the main codebase. The company also found waste from bloated context windows, tool failures, cache misses, and repeated compaction. It instruments agents through an observability system based on OpenTelemetry and Grafana, storing data in an efficient database so the company can analyze both system efficiency and individual workflows. For autonomy, Roblox tracks how long sessions run and whether they converge on useful work. A long session alone is not enough, so AI evaluates whether the resulting change reached production, how complex the task was, and what kind of task it represented. Roblox also classifies pull requests as features, bug fixes, and other categories, creating a feature-velocity metric per engineer. Swerdlow calls feature velocity a close proxy for innovation because shipping new customer capabilities is more meaningful than counting raw lines or pull requests.

Swerdlow says model routing is popular, but Roblox learned not to build infrastructure that specialized vendors or frontier labs are already improving faster. The company instead emphasizes flexibility: engineers can use multiple harnesses, frontier models, and open-weight models, including Codex, Claude Code, Cursor, and Gemini. Roblox tries to avoid one-way dependencies on a particular vendor, model, or metadata format so that engineers can move when a tool performs poorly. Swerdlow says engineers “vote with their feet,” sometimes migrating to another model or harness within days. Roblox also optimizes the cost of its AI reviewer. Its code-review system may evaluate many exemplars per pull request with multiple frontier models and an alignment loop, which can become expensive at Roblox’s scale. An engineer on the team, Kushan Mehta, developed a caching strategy that recognizes that exemplars often concern a specific part of a diff—for example, tests. Roblox pre-warms caches with relevant pull-request metadata and reuses that context across agent runs, reducing the cost of evaluating each exemplar by about 90%. Session analysis is run asynchronously, often at night, using cheaper models or open-weight models because the task does not always require the highest reasoning capability. For autonomous deployment, Roblox takes a conservative risk-based approach. A human must explicitly review and approve each exemplar, establishing the boundary for how the code-review agent should behave. A pull request can then be deployed without a human review only if programmatic checks find no issues, the repository has opted into autonomous deployment, and other safety conditions pass. A final LLM assessment looks for hidden risk. Initially, Roblox is limiting this to low-blast-radius changes such as documentation and test updates, while using evaluations and injected faults to build confidence among engineers, compliance teams, and leadership.

At the end of the conversation, Swerdlow describes a cultural change accompanying the technical one. Roblox calls its people “builders,” a term that reflects how AI tools blur the boundaries among engineers, designers, and product managers. Product managers are increasingly able to fix bugs and ship features without depending entirely on software engineers, while engineers work at higher levels of abstraction. Swerdlow does not think software engineering as a discipline will disappear: senior engineers who understand scale, failure modes, specifications, and architecture remain essential because they can help build the right system once rather than rebuild it repeatedly. He compares the change with earlier shifts in abstraction, such as the move from punch cards and assembly language to higher-level programming languages. During a transition, people may still inspect what is beneath the abstraction, but eventually they trust the abstraction enough to work above it. AI-assisted development is currently in the phase of checking beneath the abstraction and building confidence that generated systems are correct. Swerdlow returns to the uncomfortable loop in which AI writes code, AI reviews it, and AI may deploy it. Roblox keeps human accountability by having people approve the reusable exemplars that define acceptable behavior, rather than requiring them to approve every individual line. The company’s target metrics are therefore not token counts alone, but accepted code, rollbacks, production survival, feature velocity, developer sentiment, and code shipped without human intervention. His closing recommendation is to stop treating token consumption as the main measure of progress and instead measure whether the system produces reliable, useful software autonomously.