GenAI Traffic: Why API Infrastructure Must Evolve... Again // Erica Hughberg // #296

Key takeaways

The internet has evolved significantly, transitioning from dial-up to robust networking protocols designed to handle immense traffic and concurrency.
The shift from monolithic applications to microservices yields efficiency but introduces complex networking challenges that can prohibit optimal performance.
GenAI creates unique infrastructure challenges due to larger payloads and unpredictability in response times, necessitating smarter API management solutions.
Community advocacy plays a vital role in understanding and addressing developers' real-world challenges, ensuring that technological solutions align with user needs.
Future API and network design patterns must consider the changing landscape of tech demands, focusing on collaborative development to build lasting solutions.

Summary

The Evolution of the Internet and Networking

Erica Huberg discusses the evolution of the internet since the early 2000s, highlighting key shifts from dial-up to broadband and the rapid growth of online services. She explains the challenges arising from increased user connections, leading to innovations in networking such as event-driven architectures and proxies capable of handling concurrent connections more efficiently. Huberg utilizes relatable analogies, such as comparing proxies to waitstaff in a restaurant, to illustrate how networking has evolved to accommodate the demands of modern applications.

From Monoliths to Microservices

The conversation explores the transition from monolithic applications to microservices. As application demands grew, it became clear that resource efficiency was necessary, leading to the breakdown of monolithic structures into smaller, more manageable microservices. This paradigm shift allowed for better scaling and resource allocation, though it also presented new networking challenges. Huberg emphasizes the importance of enabling dynamic routing and virtualized resources in handling the complexities of microservices.

The Impacts of GenAI on API Infrastructure

Huberg addresses the unique challenges GenAI (Generative AI) brings to API infrastructure. The discussion highlights issues of network traffic management, significantly larger payloads, and unpredictable response times associated with GenAI models compared to traditional microservices. She discusses how existing infrastructure often lacks the capability to optimally manage these heavier workloads, emphasizing the need for smarter, more adaptable networking solutions to handle the evolving demands.

Community Advocacy and Collaborative Development

Huberg shares her role as a community advocate, reflecting on how her technical experience informs her interactions with the developer community. She emphasizes the importance of advocating for users' needs within the ever-changing landscape of technology and discusses the need for collaborative development to address common challenges, particularly around GenAI traffic management. The focus is on building community-driven solutions that can alleviate issues as they arise.

Future of API and Network Design Patterns

The podcast wraps up with a forward-looking perspective on how API design patterns must evolve alongside technological advancements in GenAI. Huberg warns against implementing quick fixes that may not stand the test of time while encouraging thoughtful, community-driven innovation that anticipates future needs. There is a strong emphasis on the interconnectedness of various challenges in networking and API infrastructure that must be addressed holistically.

Show notes

GenAI Traffic: Why API Infrastructure Must Evolve... Again // MLOps Podcast #296 with Erica Hughberg, Community Advocate at Tetrate.

Join the Community: https://go.mlops.community/YTJoinIn

Get the newsletter: https://go.mlops.community/YTNewsletter

// Abstract

The way we handle API traffic is broken for GenAI. We've spent years optimizing for microservices—fast, stateless, and lightweight API calls. But GenAI changes everything. Requests are slower, heavier, and more complex, requiring long-lived connections, massive payloads, and streaming responses. Suddenly, traditional API gateways are struggling—timeout limits are too short, rate limiting models don’t fit, and payload constraints are blocking innovation.

In this episode, we unpack the new challenges of GenAI traffic and why infrastructure must evolve—again. We look back at previous API shifts, from the C10K problem to the monolith-to-microservices revolution, and how they reshaped networking. Now, AI-driven workloads demand a new kind of API gateway—one that handles token-based rate limiting, cost-aware request shaping, and scalable AI inference traffic.

// Bio

Erica Hughberg is a technical leader and community advocate passionate about helping engineering teams build scalable, secure, and human-centric application platforms. With a background in software engineering and a deep understanding of cloud-native technologies, she specializes in driving the adoption of open-source projects like Envoy Gateway, Istio, and Kubernetes Gateway API, which enable organizations to simplify traffic management, security, and API distribution.

As a maintainer of Envoy AI Gateway, she plays a key role in shaping the future of API infrastructure. She focuses on features to ensure organizations can securely and efficiently integrate AI-powered services while simplifying traffic management, security, and API distribution. In the Envoy community, she drives collaboration, mentorship, and contributions that advance the project and its adoption.

Lastly, as a believer in the power of storytelling, Erica enjoys translating complex technical concepts into engaging, accessible narratives in the form of social media posts, conference talks, podcasts, and educational content.

// Related Links

Efficient Deployment of Models at the Edge // Krishna Sridhar // MLOps Podcast #284 - https://youtu.be/sFqm7GTeulg

~~~~~~~~ ✌️Connect With Us ✌️ ~~~~~~~

Catch all episodes, blogs, newsletters, and more: https://go.mlops.community/TYExplore

Join our Slack community [https://go.mlops.community/slack]

Follow us on X/Twitter [@mlopscommunity](https://x.com/mlopscommunity) or [LinkedIn](https://go.mlops.community/linkedin)]

Sign up for the next meetup: [https://go.mlops.community/register]

MLOps Swag/Merch: [https://shop.mlops.community/]

Connect with Demetrios on LinkedIn: /dpbrinkm

Connect with Erica on LinkedIn: /ericahughberg

Timestamps:

[00:00] Erica's preferred coffee

[00:30] Takeaways

[01:50] Evolving Web Gateways

[14:35] Microservices to LLM Shift

[17:42] Intelligence Privacy Model

[22:26] Infrastructure for AI Creativity

[25:25] AI Gateway Networking Challenges

[30:37] Streamlit MVP to Production

[43:03] AI Model Scaling Challenges

[47:48] Tech Advocacy and Skills

[53:17] Optimizing Edge AI Performance

[56:43] Product Management Insights

[1:00:02] Navigating Evolving Tech Challenges

[1:04:35] Wrap up