⏭️ Forward Deployed: Voice AI on what works in 2026

The host introduces Basil, whose work spans dinners, gatherings, panels, podcasts, and the forward-deployed engineering (FDE) community. Basil explains that he started as a product manager at Intuit, working on Credit Karma for a couple of years, then joined a small venture studio. He later founded XFLAB Labs, a consulting business that worked with retailers, insurance companies, and similar customers to build agents. Basil says FDE felt like a natural extension of that product work: customers came with a specific use case, and he had to understand what they wanted, make trade-offs, and help build the solution. In January, he began working with several private-equity firms and noticed that people following AI discussions on Twitter and LinkedIn could not tell “what’s marketing BS and what’s not.” His podcast and events therefore bring in people working on interesting products at interesting companies, with the goal of doing deep technical dives that listeners can apply to their own businesses. He describes episodes and panels on agentic engineering, voice agents, computer-use agents, and enterprise agents. The featured voice-agent panel included participants from Decagon, Vapi, Retell, Daily, and Smallest AI, and focused on how voice agents are actually built and what engineers still have to solve.

Basil previews the panel’s central technical point: the current state of the art for many production voice agents is a cascaded pipeline rather than a single voice-to-voice model. Audio enters through WebRTC, a phone call, or WebSockets; speech-to-text transcribes it; an LLM interprets the request and may make tool calls; and text-to-speech generates the response. Additional components can handle background-noise removal, voice isolation, and turn detection. The speakers emphasize that turn-taking is not trivial. A voice system must distinguish between a pause in the middle of a sentence and a pause indicating that the speaker has finished. Systems commonly use voice-activity detection and “smart turn” models because, unlike text, speech has no equivalent of pressing Enter to indicate completion. Basil also identifies the main trade-off: more intelligent responses generally require more latency. Production systems must also handle model outages by maintaining a waterfall of fallback models so that a voice agent does not stop working when a provider such as Opus becomes unavailable. The discussion highlights customer support as the dominant voice-agent use case, involving enormous call-center spending and systems that are often poor today. Outbound calls add another problem: people frequently hang up as soon as they realize they are speaking to a bot.

Bruin, discussing Pipecat, explains the full cascade in more detail. After transcription and turn detection, the LLM may infer an answer, call tools, or perform other work. The result is sent to text-to-speech, which is usually faster than real time, and the generated audio is streamed back over the same transport. For an outbound collections call, the system may simply tell someone about an unpaid bill and offer payment options. If the person goes off-topic, the bot can end the call because it knows the narrow set of inputs it is expected to handle. Inbound calls are harder because the system starts with little context about why the person is calling. A small business, such as a flower shop, may have a limited set of possible requests, but a company such as Amazon has billions of products and different policies for physical goods, electronics, expensive items, and other categories. A single giant prompt cannot reliably contain all of that information: the speakers say LLMs tend to remember the first and last portions of a long context while losing material in the middle. Bruin compares this to coding agents, which compact their context before it becomes unmanageable. Voice systems have been doing similar work for longer: after several turns, they summarize or otherwise preserve what the human said, what has happened, and where the conversation stands. This is especially important because the bot often speaks more than the human, causing context to grow quickly.

The panel compares cascaded systems with speech-to-speech models. One speaker gives a failure case for an impressive voice-to-voice demo: the model may sound fast, listen to laughter, and respond to tone, but still misunderstand a date. If asked to schedule an appointment for “next week,” it might confidently convert that into June 10, 2030, even when that is wrong. The cascade makes it easier to insert supervisors that detect prompt injection or social engineering, select the right intent, condition the context, generate a response, and then check whether the answer is grounded in truth. Smallest AI’s representative explains that speech-to-speech began partly to preserve emotional information lost when audio is converted into text. Their deeper argument is that human conversation is asynchronous: people think while listening, interrupt, take notes, and perform work before the other person has finished speaking. Smallest’s Hydra is described as a multimodal speech-to-speech model that accepts both speech and text and produces both speech and text, allowing tool calls and guardrails rather than treating the model as a completely opaque audio-in/audio-out system. The speaker acknowledges that speech-to-speech may sacrifice some accuracy and interpretability compared with specialized speech-to-text models. Vapi’s perspective is that the medium-term answer will probably be hybrid: speech-to-speech can handle the active conversational loop, while a more deliberate cascaded process handles complex lookups or workflows. The group expects speech-to-speech to become more important, but says enterprises still deploy cascaded systems more often.

Asked which frontier models are used in cascaded pipelines, the speakers resist naming one universal choice. Basil asks about models such as GPT, Opus, and Sonnet, including whether GPT-4 is still useful because it balances speed and intelligence. One participant says the right choice depends on whether the model needs to reason deeply or simply respond quickly, and specifically says Gemini 2.5 has worked well because it is fast, while newer or more capable models can be slower. The group describes increasingly elaborate ways to parallelize the pipeline. Audio can be sent directly to a speech-to-speech model while also being forked through speech-to-text. Text can then be sent to several LLMs at once, with a downstream “gate” or evaluator deciding which answer is best. One speaker jokes that engineers can now revisit the kinds of computer-architecture ideas they had nothing to do with for 20 years: waterfalls, forks, gates, and parallel work become practical parts of a voice system. The panel also discusses specialized models for common cases. If a company knows that 50% or 70% of requests follow a predictable pattern, it can optimize a smaller model or a narrow workflow for that portion and reserve the expensive, complicated cascade for unusual requests. Smallest AI says customers are pairing its models with its smaller Electron language model for real-time workloads because it lowers cost and latency and gives customers more control than depending entirely on variable hosted API performance.

The speakers distinguish between the challenge of making speech-to-speech excellent in English and the practical demands of multilingual deployments. Smallest AI says its speech-to-speech work is currently focused on making English highly intelligent rather than introducing additional variables through other languages. The company has nevertheless seen substantial demand from India and Latin America, while the United States is described as mostly English with some Spanish and a range of accents. Vapi’s representative gives a concrete example from a deployment in Japan. A standard speech-to-text model such as OpenAI’s 4.1 or Deepgram could work adequately, but the available text-to-speech voices did not perform well for Japanese. The advantage of a cascade is that teams can replace only the component that fails. Vapi lets customers bring their own text-to-speech server, which can be important for pronunciation of local brands, addresses, and other market-specific terms. Arabic is mentioned as another language where pronunciation can be difficult. The conclusion is not that one provider can optimize every language, accent, brand, and use case itself. Instead, a modular cascade lets customers use specialized providers or local startups for the part of the system that needs regional expertise, while retaining the rest of the platform.

The panel asks whether the standard practice is one enormous system prompt containing every rule or a workflow made of separate nodes. The answer depends on the use case. For predictable inbound lines, a graph or node-based approach can route a caller through a known process. Outbound calls are less predictable because a person may become frustrated, angry, or annoyed, so one participant argues that a central prompt can give the model access to instructions that were not part of the original dedicated flow. Another speaker says their execution engine makes rules situational: it checks which instructions are relevant before adding them to the system prompt. This approach was particularly useful seven months earlier, when models often skipped a step such as “4a1.” But model quality has improved, and selectively deciding which instructions matter can itself be less reliable than simply giving the model the full context. Their current practice is therefore moving back toward providing more information and testing whether the model can reason over it. The panel repeatedly returns to the same operational answer: there is no universal prompting best practice. Teams must test each customer’s workflow, use an LLM as a judge to assess reliability and grounding, then add preprocessing, context optimization, or specialized agents if failures persist. A collections workflow, for example, can delegate payment-method work to a specialized agent and then return to a more flexible conversational agent once that task is complete.

The speakers describe latency as the hardest voice-deployment trade-off because improving reliability and control often adds processing steps. One participant says contextual fillers—phrases such as “give me a second” while a tool call is running—are not necessarily bad. In human conversation, people naturally make brief sounds while thinking, and a well-timed filler can feel better than a long silence. The problem is that fillers must sound natural, especially when a customer API takes several seconds to respond. Vapi’s representative recommends offloading long instructions into specialized agents and streaming work to smaller models that can classify intent or collect information in parallel. This can improve both responsiveness and cost. A large prompt is expensive even when the caller hangs up after ten seconds, which happens frequently. Smaller self-hosted models can also avoid unpredictable API spikes and can be scaled according to demand. Basil and the panel stress the importance of evaluations. Daily publishes turn-based benchmarks for speech-to-text, LLMs, and text-to-speech, with open turns that teams can run against new models or their own infrastructure. Forward-deployed engineers can turn conversations gathered during customer discussions into smoke tests before deployment. The practical loop is to build the relevant evals, test repeatedly, use the results to identify failures, modify the pipeline, and test again; “test and find out” is presented as the real answer for nondeterministic voice systems.