24/12/3 Deploying Private Open Source LLMs on Azure

Key takeaways

Understanding the architecture and deployment of private AI models using open source tools can enhance security and compliance.
Infrastructure-as-code patterns simplify resource management and scaling in Azure environments.
Using models like Olamma provides predictable cost control and flexibility compared to proprietary vendor models.
Open WebUI allows for effective interaction with multiple models, improving user experience by comparing responses.
Engagement through Q&A fosters community learning and the sharing of operational challenges and solutions.

Summary

Introduction to Private AI Infrastructure

In this episode, Alec Harrison introduces the concept of deploying private AI infrastructure on Azure, encouraging listeners to engage with local user groups and participate in events. The episode features Feynman Liang, the CTO of Blueteam AI, who discusses the importance of using open source tools like Ollama and Open WebUI alongside Azure's capabilities to create a secure and compliant AI infrastructure.

Deploying Large Language Models with Open WebUI and Ollama

Feynman outlines how to deploy large language models using Open WebUI paired with Ollama, a project that facilitates model management and serving. He explains the architecture of their implementation, clarifying how users can interact with models like the 3.21 billion parameter Llama, all while controlling costs and flexibility by leveraging open source solutions.

Infrastructure as Code with OpenTofu

The discussion progresses to the implementation of infrastructure-as-code patterns using OpenTofu, a fork of Terraform. Feynman emphasizes the significance of automating and reproducibly managing resources on Azure Kubernetes Service (AKS). He shares practical deployment steps, showcasing how users can set up the infrastructure without needing to navigate the Azure portal.

Best Practices and Real-Time Use Cases

Feynman illustrates best practices for scaling and managing open source AI infrastructure through a live demo. He covers multiple responses from different models for better decision-making and highlights the integration of document upload and retrieval capabilities in the Open WebUI, which enhances the user interaction experience.

Engagement and Discussion

Toward the end, an interactive Q&A session is facilitated, allowing participants to pose questions regarding operational challenges, model sizes, and resource management. This segment reinforces the practicality of the discussed solutions and encourages knowledge sharing among the attendees.

Show notes

Join Feynman Liang, CTO of Blueteam AI, for a practical demonstration of running open source AI models privately in Azure. This session will walk through a production-grade reference implementation that deploys Ollama and Open WebUI on AKS using infrastructure-as-code patterns. You'll learn how to set up a secure, compliant AI infrastructure using familiar tools like OpenTofu and Kubernetes, and understand the key architectural decisions that make this implementation suitable for enterprise use. Whether you're evaluating open source LLMs or looking to deploy them in production, this talk will provide you with actionable patterns and hands-on examples you can start using today.

Key Topics:
- Infrastructure-as-code patterns for AI workloads on Azure
- Security and compliance considerations for private AI deployments
- Practical deployment steps using OpenTofu and AKS
- Live demo of the reference implementation
- Best practices for scaling and managing open source AI infrastructure