Runway’s WorldPrompt and the Engineering of Real-Time Worlds

Runway co-founder Anastasis described the company’s original thesis as a matter of “when, not if”: after seeing early generative models in 2016–2017, he expected content generation to improve predictably until most content was generated. Runway would therefore need to rethink creative tools rather than merely build another model. He had previously split his time between an art practice involving interactive installations and engineering work at startups, where he worked as an ML and backend engineer. The hosts reviewed an installation he made in 2015 that gave gallery visitors generated identities—such as “you’re an architect, you’re 30 years old, and you like sports”—then matched them with other visitors for simulated small-talk conversations. Because language models were not capable enough, it used templates and Markov-chain-generated text. Another early project with co-founder Chris used NVIDIA’s 2016–2017 Pix2PixHD model, trained on self-driving data, to turn semantic maps containing pedestrians, traffic signs, bicycles, and stoplights into images. Artists deliberately pushed the model out of distribution, creating scenes with gigantic people or millions of traffic signs. Anastasis said this established a Runway principle: models trained on ordinary data can become artistically surprising when placed in tools that let artists use them differently.

The first version of Runway made contemporary open-source models usable by artists who were not ML engineers. Within its first year, however, the company concluded that the available image and video models were not yet good enough to be integrated into professional creative workflows, so it created an internal research organization to push the technology forward. For several years that research happened mostly in the background. Anastasis identified 2022 as the major inflection point, when latent diffusion and DALL·E 2 demonstrated a step change in quality and feasibility. He recalled being especially struck that Stable Diffusion could run on consumer hardware. Before generative video, Runway’s main product was Green Screen, an image- and video-segmentation tool that automated the rotoscoping process used extensively in visual effects. It was used in films including Everything Everywhere All at Once and other high-profile productions. In mid-2022, Runway committed to a cluster of 1,000 A100 GPUs, an unusually aggressive decision for a Series B company. Anastasis said the bet followed the belief that scaling laws would apply to image and video generation as they had to language models. The team initially wanted text-to-video, but found video-to-video easier because an existing video provides stronger conditioning. That led to Gen-1, released in January 2023, which converted an input video into a depth map and then generated a restyled RGB video.

Gen-1 was not simply a text-to-video system: it used depth-conditioned generation to restyle existing footage. Runway saw users turn rough phone footage, primitive city scenes, Blender renders, and Unity captures into realistic or stylized outputs. Anastasis especially liked the storyboard use case, where someone could build a scene out of boxes or crude objects, film it, and translate the result into a more realistic version. He argued that video-to-video remains powerful because real footage supplies “ground truth” structure while the model changes its style or appearance. Gen-2 followed only about two months later, partly because the team found a practical workaround for text-to-video. A model converted text into a depth map, and Gen-1 then converted that depth map into RGB video. Anastasis said this two-stage pipeline was imperfect—the generated structure could look off—but it allowed Runway to ship quickly. He noted that later systems also adopted multi-stage planning, including models that generate bounding boxes before rendering and systems that rewrite a user’s prompt into a more detailed description. Runway’s early experience showed that text alone was not enough. Gen-2 was a major quality improvement, but users wanted control over camera movement, object movement, input frames, and references. During 2023, Runway added features including Motion Brush and camera control. Anastasis said camera control was particularly important because it made the model feel less like a content generator and more like a way to navigate inside a world.

Asked about the Stable Diffusion controversy, Anastasis said the story began with the latent-diffusion paper from late 2021. Patrick Esser, then at Runway, had worked on latent diffusion with Robin Rombach and others. Stable Diffusion was essentially a continuation of that work, trained with more compute, improved with techniques such as classifier-free guidance, and trained on a more aesthetic subset of LAION data. Stability AI financed the large training run, which made the model broadly successful. Anastasis characterized the disagreement as partly a normal dispute over people moving between companies and partly a question of how an open research project should be continued. Runway believed it was important to preserve the original open-source intent, which led to the release of Stable Diffusion 1.5. He described a short period of miscommunication that was resolved within hours and said that, in retrospect, the episode was in the past and both Stability AI and Runway had followed their own paths. The hosts also recalled how small the generative-art research community was from roughly 2018 to 2022. Early projects could become viral immediately because there were so few people working in the area. Anastasis cited Jean Kogan’s video of riding the New York subway and crossing the Williamsburg Bridge, then stylizing the footage in the style of a painter such as Van Gogh. He said that work felt revolutionary at the time, illustrating how quickly the field went from experimental, difficult-to-use diffusion tools to everyday creative products.

When Sora appeared in February 2024, Anastasis said Runway briefly experienced an “existential crisis.” Sora’s quality was far beyond Gen-2, and public commentary suggested that Runway might be finished. The team had already spent much of late 2023 building infrastructure for diffusion transformers and distributed training, but it had little prior expertise in model parallelism and repeatedly failed while trying to scale video models. Runway responded by scaling the model and training compute by roughly 10×, learning the necessary distributed-training techniques, and releasing Gen-3 that summer. Anastasis described the push as a defining period for the company: it had to absorb several years of progress in language-model infrastructure within months. The first Gen-3 model was expensive to serve, so Runway subsequently released a Turbo version. He argued that distillation and real-time generation had not disappeared; Runway’s real-time character model is a step-distilled autoregressive video model that runs at 24 frames per second. He distinguished two forms of distillation: making a smaller model, analogous to language-model distillation, and reducing the number of diffusion steps—for example, converting a 50-step model into a four-step model. Runway can sometimes distill the large frontier model directly rather than training a separate small model. Anastasis predicted that real-time video generation will become dominant because it offers a better user experience, costs less to serve, and should gradually close its quality gap with slower models.

Runway’s world-model work began in late 2023, although Anastasis resisted calling it a pivot. He framed it as an expansion of the original video-model mission: generating convincing video requires increasingly capable simulation of physics, human actions, and world dynamics. The company’s thesis is that scaling video prediction should eventually produce general-purpose world simulators, much as scaling language models moved from barely coherent text to systems capable of long autonomous tasks. He argued that text descriptions omit enormous amounts of information. People may be unable to explain precisely how a guitar is played or how a physical interaction unfolds, even though they can demonstrate it. Training on observational video therefore exposes models to complexity that is not captured in human-written descriptions. In response to Yann LeCun’s criticism that understanding physical dynamics is different from generating “cute videos,” Anastasis said the two are fundamentally related, while acknowledging that current models can hide their deficiencies through shot changes and other artifacts. Runway evaluates physical understanding with benchmarks such as Physics IQ, which tests mechanics, fluid dynamics, and optics. One example gives a model an initial frame, such as a ball hanging from a ceiling, and asks it to predict what happens next. Anastasis said the score improves predictably as compute and model scale increase, although better data and other techniques can improve it further. He called counterfactual generation—the ability to make both success and failure equally realistic—the major distinction between ordinary video generation and a true world model.

Runway showed an Interface World Model that renders a software interface directly as pixels from a real-time video-generation model. It does not use HTML, CSS, or React for the visible interface. Instead, the model predicts what should happen after clicks, drags, and scrolling, while prompts describe behaviors such as what a button should do. Anastasis presented this as an “universal interface to software”: rather than writing code that renders pixels, the system generates the pixels directly. The proposed applications include rapid prototyping, creative tools, educational experiences, open-ended visualizations, and industrial-design software. The hosts noted that this approach could be expensive compared with conventional rendering, particularly when the interface contains ambient motion such as people walking in the background. Anastasis agreed that serving real-time video is computationally costly, but said the first priority is to identify interactions that are compelling enough to justify the expense; efficiency can improve later through optimization, smaller models, or looping mechanisms. He also described a longer-term vision of a “fully neural operating system.” Current language models can discuss many tasks but are accessed through rigid interfaces. A learned renderer could instead generate a personalized interface for each user, changing text size for accessibility or adapting to study preferences. The same world model could serve humans and computer-use agents: it could generate synthetic interfaces for training, act as a live reinforcement-learning environment, and provide diverse interaction outcomes without manually coding every application.

Runway’s GWM-1 world model adapted the Gen-4.5 video backbone into an autoregressive model that accepts actions. Instead of merely watching generated video, a user or policy can take an action, observe a rollout, and explore counterfactual outcomes. Runway initially saw robotics applications in generating synthetic data, then concluded that world models could also serve as simulators for evaluating robotic policies. The company tested real-to-simulation correlation using the RoboArena benchmark. It compared actions and scenarios inside the world model with the same actions in physical reality and reported strong correlation. Anastasis said this could let robotics teams evaluate policies at much greater scale without repeatedly using physical hardware. The advantage over traditional simulators is particularly clear for cloth, slippery surfaces, and other complex interactions that are difficult to model manually. A world model can start from a picture of an environment and roll out possible actions, rather than requiring a 3D scan and a carefully authored digital version of every object. Runway’s broader robotics thesis is that third-person video is the most abundant source of training data. Teleoperation and egocentric datasets are valuable but much smaller and more expensive to collect. Humans also learn many skills by observing other people, so a large video model can learn physics, human dynamics, and task structure before being fine-tuned on hundreds of hours of robot data. Current models are adapted for particular robot embodiments, but Anastasis expects navigation, manipulation, and human-behavior models eventually to unify into one real-time world model.

Anastasis identified long context as a major remaining challenge. Autoregressive models feed generated frames back into themselves, so small errors accumulate and eventually degrade a scene. Runway’s character models can generate long interactions, while its more open-ended world models currently operate on the order of minutes. The ideal system would generate indefinitely without losing consistency, but some applications can work around the limitation by cutting to a new scene. Runway’s creative workflow has moved from prioritizing quality to prioritizing controllability and now increasingly latency. Anastasis said users first wanted better-looking output, then more control through references, storyboards, camera directions, and other inputs. As quality rises, they want to generate many options almost instantly so they can explore ideas as quickly as they could in tools such as Photoshop. Runway uses a large creative team alongside researchers to evaluate what counts as a usable video and to produce detailed captions describing cinematography, aesthetics, and camera direction. The conversation ended with Runway’s open research initiative, the Cosmos Coalition, which aims to release world-model weights, benchmarks, and infrastructure. Anastasis said Western video models currently lag behind Chinese models on leaderboards, making open research and stronger community investment important. He also argued for eventually combining modalities—RGB, audio, depth, heat, text, and other signals—because learning in one modality may transfer to another. His example was Refusion, which fine-tuned an image diffusion model on spectrograms to generate music. He expects future models to combine language reasoning, video generation, audio, and action prediction rather than treating them as separate systems.