How to Systematically Test and Evaluate Your LLMs Apps // Gideon Mendels // #269

Key takeaways

Developers must integrate ML and software engineering practices to effectively build and evaluate LLM applications.
Evaluation metrics for LLMs are evolving, requiring new approaches that are distinct from traditional ML metrics.
Keeping track of all parameters and outputs in LLM experiments is crucial for replication and improvement.
Opik offers a seamless way to track LLM outputs, support manual and automated labeling, and analyze results.
Collaboration and sharing of insights within teams are essential for effective experimentation and improving LLM applications.
Finding a balance in alert systems is vital to avoid false positives while ensuring quality outputs from LLMs.

Summary

Introduction to LLM Applications

Demetri introduces Gideon Mendels, the CEO of Comet, and emphasizes how developers must combine machine learning (ML) and software engineering practices for building LLM applications. They stress the importance of defining evaluation metrics and tracking experimentation to ensure successful deployment.

Evaluation Metrics in AI and LLMs

Gideon discusses traditional evaluation metrics used in machine learning and contrasts them with the newly emerging metrics necessary for LLMs, such as perplexity and heuristic distance metrics. He addresses how evaluation methodologies differ when evaluating LLM outputs and emphasizes the need for a more experimental and metric-driven approach.

Experiment Tracking and Methodologies

The conversation explores the major differences in experiment tracking between traditional ML and LLMs. They discuss how traditional methods do not fully apply to LLMs due to their nondeterministic nature. Gideon highlights the importance of tracking not just results but all input and output parameters in order to replicate results effectively.

The Role of Opik in Evaluation and Tracking

Gideon introduces Opik, Comet's tool for automatically tracking response outputs and evaluation parameters. He explains how Opik allows teams to bootstrap datasets and effectively analyze outputs through both manual and automated labeling, enhancing the exploratory phase of deploying LLM applications.

Collaborative Experiment Tracking

Demetri and Gideon discuss the importance of collaboration in the experimentation process. Gideon details how Opik facilitates teamwork by allowing data sharing, annotation, and observability of user interactions with LLMs, necessary for understanding how to improve outputs.

Future Directions and Challenges in AI Evaluation

Towards the end, the discussion veers into the future of alerts and feedback mechanisms. They highlight the challenge of alert fatigue in AI systems, discussing how identifying failed outputs without generating excessive alerts will be key to ensuring quality in LLM applications.

Show notes

Gideon Mendels is the Chief Executive Officer at Comet, the leading solution for managing machine learning workflows. How to Systematically Test and Evaluate Your LLMs Apps

// MLOps Podcast #269 with Gideon Mendels, CEO of Comet.

// Abstract

When building LLM Applications, Developers need to take a hybrid approach from both ML and SW Engineering best practices. They need to define eval metrics and track their entire experimentation to see what is and is not working. They also need to define comprehensive unit tests for their particular use case so they can confidently check if their LLM App is ready to be deployed.

// Bio

Gideon Mendels is the CEO and co-founder of Comet, the leading solution for managing machine learning workflows from experimentation to production. He is a computer scientist, ML researcher and entrepreneur at his core. Before Comet, Gideon co-founded GroupWize, where they trained and deployed NLP models processing billions of chats. His journey with NLP and Speech Recognition models began at Columbia University and Google, where he worked on hate speech and deception detection.

// MLOps Swag/Merch

https://mlops-community.myshopify.com/

// Related Links

Website: https://www.comet.com/site/

All the Hard Stuff with LLMs in Product Development // Phillip Carter // MLOps Podcast #170: https://youtu.be/DZgXln3v85s

Opik by Comet: https://www.comet.com/site/products/opik/

--------------- ✌️Connect With Us ✌️ -------------

Join our Slack community: https://go.mlops.community/slack

Follow us on Twitter: @mlopscommunity

Sign up for the next meetup: https://go.mlops.community/register

Catch all episodes, blogs, newsletters, and more: https://mlops.community/

Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/

Connect with Gideon on LinkedIn: https://www.linkedin.com/in/gideon-mendels/

Timestamps:

[00:00] Gideon's preferred coffee

[00:17] Takeaways

[01:50] A huge shout-out to Comet ML for sponsoring this episode!

[02:09] Please like, share, leave a review, and subscribe to our MLOps channels!

[03:30] Evaluation metrics in AI

[06:55] LLM Evaluation in Practice

[10:57] LLM testing methodologies

[16:56] LLM as a judge

[18:53] OPIC track function overview

[20:33] Tracking user response value

[26:32] Exploring AI metrics integration

[29:05] Experiment tracking and LLMs

[34:27] Micro Macro collaboration in AI

[38:20] RAG Pipeline Reproducibility Snapshot

[40:15] Collaborative experiment tracking

[45:29] Feature flags in CI/CD

[48:55] Labeling challenges and solutions

[54:31] LLM output quality alerts

[56:32] Anomaly detection in model outputs

[1:01:07] Wrap up