Why PMs are suddenly valuable again | Tulsee Doshi (Head of Product, Gemini)

Tulsee Doshi, Google’s head of product for frontier models, says model development differs from conventional product management because outcomes cannot be fully predicted. Teams can set goals for capabilities, user outcomes, and evaluation benchmarks, then carefully monitor training runs and intervene as metrics change. But the finished model may develop unexpected strengths or behaviors—“a little bit of magic,” in her Snape’s potions class analogy. A PM must therefore study what comes out, identify where the model uniquely shines, understand its “vibes,” and turn those discoveries into useful products rather than only controlling what goes in.

Doshi rejects the idea that Google has opted out of the frontier-model race. She says Gemini 4 Argon represents Google’s push to build a frontier lab and hopes listeners will be using it by the time the episode airs. Google’s advantage, in her view, is its full stack: hardware, models, and products used by billions—she tentatively cites 14 products with more than a billion users. The strategy is to offer models across the cost-quality curve for search users, engineers, and cyber defenders, while using feedback from those users to improve the portfolio.

Doshi, who previously led Google’s responsible-AI product work for six years, says safety must be embedded from initial training through deployment. She highlights monitors that inspect a model’s chain of thought for alignment with its directed actions, allowing teams to intervene and improve the model when misalignment appears. She is cautiously optimistic about safety, but only because people are being intentional, worried, and willing to interrogate how models are built. She also emphasizes AI’s defensive potential: Google is working with cyber defenders, including Wiz, to find and patch vulnerabilities, such as one identified in a hospital system.

Doshi agrees that AI products can feel overloaded with tabs, dropdowns, agents, and experimental features, but says labs must keep testing new interaction patterns. Google is exploring different form factors through products such as Flow, Genie, Veo, NotebookLM, and Omni. The right complexity depends on the audience: AI Mode must be accessible to billions of search users, while Antigravity can target power users comfortable with slash commands, terminals, and IDEs. She adds that the model itself has a UX: its transparency, personality, explanations, questions, and communication style all shape the experience.

The host argues that AI has made building easy enough for teams to produce large amounts of nonsense, renewing the value of PM judgment. Doshi agrees that when anything can be built, taste—separating signal from noise and recognizing a genuinely useful experience—becomes a differentiator. PMs also create order in organizations such as Google, where faster building can otherwise create chaos. For research PMs, she increasingly wants people who use models daily, have opinions about their behavior, stress-test them, inspect data, build evaluations, and remain agile as the market changes every two weeks.

Doshi says PMs do not need to begin with a massive benchmark. They can start with five prompts, or a small set of roughly 20, designed to stress-test a capability and reveal differences between models. Initially, the PM can act as the evaluator and define what a good response looks like, especially for long, multi-step, multi-turn tasks. She recommends choosing tasks the evaluator genuinely cares about—creative writing, visual generation, financial analysis, or a work problem—because personal interest makes it easier to notice how a model’s capabilities and behavior evolve.

Doshi describes replacing reliance on the six-tone Fitzpatrick scale, which was created for healthcare rather than global representation and skewed toward lighter skin tones. Google worked with Harvard professor Monk, whose research produced a ten-tone scale based on years of interviews and study. The team incorporated it into model and product evaluations, then used the results to improve skin-tone representation in products including Pixel camera experiences and image generation. She says evaluation must cover the entire user experience: the judge’s quality, production filters and mitigations, serving behavior, and—in audio—the content, naturalness, tone, and inflection.

Doshi’s 60-20-20 framework adapts the usual MVP advice to rapidly improving models. About 60% of a product should work now; another 20% may work imperfectly but plausibly soon; and the final 20% should represent the magical experience the team can already glimpse if the model improves. The percentages are flexible, but the principle is to ship something useful without designing only for today’s capabilities. Teams should also retest their products as models improve, because a rigid harness built around an older model may constrain a newer one; sometimes removing skills or instructions reveals that the model now handles the task itself.

Doshi defines an AI harness as the wrapper around a model: system instructions, tools, skills, context, and other data that determine the environment in which the model operates. She connects this idea to her earlier research on “flash teams,” where crowdsourced workers with different strengths were assembled temporarily to complete a task efficiently. Agent systems can now reproduce some of that pattern through sub-agents that divide work and coordinate dynamically. To help Google PMs adapt, her team shares internally developed skills, runs workshops on inspecting data and long trajectories, and plans a two-day session where PMs can step away from daily work to experiment.

Doshi says PMs should build the habit of pausing between a task and its execution to ask, “Can AI do this for me?” One successful example often builds enough confidence to encourage further experimentation. She sees AI-generated documents as most useful for gathering data—usage, metrics, feedback, and external discussion—while the PM supplies the insight and interpretation. AI has also changed PM work through rapid prototyping and easier data analysis: the role is moving beyond writing requirements toward adding concrete heft through prototypes, evaluations, data, or other evidence. Doshi says Google still has “more work” to do on a personal assistant that can use Gmail, Calendar, and Docs, though she asks listeners to stay tuned.