Lessons from Building AI Agents with Rafal Wilinski

Key takeaways

Zapier Agents provide a user-friendly platform for non-technical users to build custom AI agents.
Hands-on integration of AI tools into applications is critical, emphasizing the need for engineers to adopt a product-oriented approach.
Using evals is vital in assessing AI model performance and ensuring continuous improvement.
Agentic workflows allow for dynamic user input interactions but come with unpredictability risks.
The type of technology stack matters; choosing tools like TypeScript can enhance AI development workflows.
Open source models are evolving rapidly, making significant contributions to the AI landscape.

Summary

Introduction to Zapier Agents

Rafal Wilinski discusses the capabilities of Zapier Agents, a platform designed for non-technical users to build their own agents. The conversation explores how some users are leveraging this tool as a free alternative to chat-based AI like ChatGPT, utilizing it for various tasks such as debugging code, writing articles, and proofreading.

Transitioning to AI Development

Rafal provides insights into his career transition towards AI technology and product management. He shares his experiences at Zapier and Vendr, emphasizing the importance of hands-on integration of AI tools into software products beyond just using existing AI solutions.

Best Practices for Building with AI

The discussion highlights key challenges and best practices for integrating AI into products, particularly the significance of using evals to assess model performance. Rafal stresses the need for engineers to adopt a product-oriented mindset, focusing on user needs and continuously iterating based on user feedback.

Agentic Workflows vs. Traditional Workflows

Rafal explains the difference between agentic workflows and traditional predetermined workflows. He discusses the advantages and risks associated with agentic workflows enabled by AI, such as unpredictability and varying outcomes based on varying inputs.

The Importance of Evaluation in AI

A significant portion of the dialogue centers on the process of evaluating AI output through 'evals.' Rafal discusses how to measure success in AI models and the methodologies to ensure continuous improvement while balancing user expectations.

AI Infrastructure and Tech Stack

Rafal talks about the technological stack used at Zapier, highlighting their preference for TypeScript over Python, and detailing the tools they integrate into their AI ecosystem, including AI SDKs and the use of Light LLM for load balancing.

AI Development Mindset in Different Contexts

The conversation touches on how engineers must adopt a transitioning mindset due to AI's unpredictability and user variability. Rafal stresses that due to the broad capabilities of AI products, engineers must think critically about the broader user experience.

Future of AI and Open Source Models

The final segment discusses the rapid progress of AI technologies and the rise of open-source models. Rafal reflects on how the market dynamics and community innovation are changing the landscape of AI development and deployment.

Show notes

Today we're talking with one of our favorite engineers, Rafal Wilinski. Rafal has been on the cutting edge of AI development in the last few years as he has led AI teams at Zapier and Vendr.

Rafal walks us through the hard-won lessons about actually integrating AI tools into the applications you're building. One of the hardest things in integrating these AI tools is how to ensure you're getting better and not regressing as you improve your prompts and upgrade your models. He shows how using evals is one part of the story along with deeply investigating customer signals to see how they are or aren't succeeding with AI.

Along the way, we also talk about RAG, his favorite models, his AI development toolset, and why Poland has been killing it lately. Check it out and be sure to follow Rafal if you want to learn more on building with AI.