AI Agent Development Tradeoffs You NEED to Know

Key takeaways

AI agents like Alice and Julian are transforming sales outreach by automating complex tasks.
Using robust frameworks like LangGraph Cloud can enhance the reliability and functionality of AI agents.
Integration of observability tools is vital for monitoring AI agents and ensuring their accuracy.
The AI agent landscape is evolving, with potential simplification as models improve.

Summary

AI Agent Development and the Role of Digital Workers

Sherwood Callaway, from 11X, discusses the evolution of AI agents in sales, particularly focusing on Alice, an AI sales rep, and Julian, a voice agent. He reflects on how these digital workers automate intricate tasks in the sales process, yielding higher efficiency. Callaway shares insights into his previous experience with OpKit, a voice AI company, and contrasts the challenges of developing voice-based solutions versus text-based agents.

Agent Frameworks and the Use of LangGraph Cloud

The conversation delves into the controversies surrounding different agent frameworks, particularly favoring LangGraph Cloud for building AI agents. Sherwood discusses the flexibility and maturity of LangGraph compared to other frameworks like OpenAI's SDK, emphasizing the importance of a robust structure in designing reliable agents that can navigate complex tasks effectively.

Navigating the Complexities of AI Agents in Production

Sherwood shares the ongoing challenges faced while integrating observability tools like Langsmith and Arise into AI agent workflows. He outlines the various metrics and evaluation (eval) methodologies used to track performance and mitigate hallucinations in AI-generated outputs, noting how critical these evaluations are to maintain accuracy and reliability in communication.

The Future State of AI Agents and Market Dynamics

Concluding the discussion, Sherwood speculates on the future trajectory of AI agents, suggesting a potential simplification of existing agent frameworks as technology and capabilities improve. The conversation offers a thought-provoking look ahead at how the landscape of AI agents might evolve and become more streamlined.

Show notes

Sherwood Callaway, tech lead at 11X, joins us to talk about building digital workers—specifically Alice (an AI sales rep) and Julian (a voice agent)—that are shaking up sales outreach by automating complex, messy tasks.

He looks back on his YC days at OpKit, where he first got his hands dirty with voice AI, and compares the wild ride of building voice vs. text agents. We get into the use of Langgraph Cloud, integrating observability tools like Langsmith and Arize, and keeping hallucinations in check with regular Evals.

Sherwood and Demetrios wrap up with a look ahead: will today's sprawling AI agent stacks eventually simplify?

// Bio

Sherwood Callaway is an emerging leader in the world of AI startups and AI product development. He currently serves as the first engineering manager at 11x, a series B AI startup backed by Benchmark and Andreessen Horowitz, where he oversees technical work on "Alice", an AI sales rep that outperforms top human SDRs.

Alice is an advanced agentic AI working in production and at scale. Under Sherwood’s leadership, the system grew from an initial prototype to handling over 1 million prospect interactions per month across 300+ customers, leveraging partnerships with OpenAI, Anthropic, and LangChain while maintaining consistent performance and reliability. Alice is now generating eight figures in ARR.

Sherwood joined 11x in 2024 through the acquisition of his YC-backed startup, Opkit, where he built and commercialized one of the first-ever AI phone calling solutions for a specific industry vertical (healthcare). Prior to Opkit, he was the second infrastructure engineer at Brex, where he designed, built, and scaled the production infrastructure that supported Brex’s application and engineering org through hypergrowth. He currently lives in San Francisco, CA.

// Related Links

~~~~~~~~ ✌️Connect With Us ✌️ ~~~~~~~

Catch all episodes, blogs, newsletters, and more: https://go.mlops.community/TYExplore

MLOps Swag/Merch: [https://shop.mlops.community/]

Connect with Demetrios on LinkedIn: /dpbrinkm

Connect with Sherwood on LinkedIn: /sherwoodcallaway/

#aiengineering

Timestamps:

[00:00] AI Takes Over Health Calls

[05:05] What Can Agents Really Do?

[08:25] Who’s in Charge—User or Agent?

[11:20] Why Graphs Matter in Agents

[15:03] How Complex Should Agents Be?

[18:33] The Hidden Cost of Model Upgrades

[21:57] Inside the LLM Agent Loop

[25:08] Turning Agents into APIs

[29:06] Scaling Agents Without Meltdowns

[30:04] The Monorepo Tangle, Explained

[34:01] Building Agents the Open Source Way

[38:49] What Production-Ready Agents Look Like

[41:23] AI That Fixes Code on Its Own

[43:26] Tracking Agent Behavior with OpenTelemetry

[46:43] Running Agents Locally with Phoenix

[52:55] LangGraph Meets Arise for Agent Control

[53:29] Hunting Hallucinations in Agent Traces

[56:45] Off-Script Insights Worth Hearing