We're All Finetuning Incorrectly // Tanmay Chopra // #304

Key takeaways

AI systems possess significant potential, yet their practical implementations often fail to realize these capabilities.
The shift to AI-driven agents won't entirely eliminate traditional software applications; instead, both will likely coexist.
Effective evaluation methods are crucial for understanding AI performance and driving improvement.
Organizations need to embrace AI as a dynamic and ever-evolving journey instead of a static tool.
Finding balance in AI development between flexibility and complexity is vital, along with maintaining robust feedback loops.

Summary

Understanding AI Limitation vs Potential

The podcast opens with a discussion about the gap between the incredible potential of AI systems and their actual output in production. Both Dan Mitchell and Tanmay Chopra emphasize that while systems like ChatGPT show great promise, they often fail to meet practical requirements when deployed, leading to frustration among users.

The Future of Software Applications in AI

A significant portion of the conversation revolves around the future of traditional software applications versus AI-driven agents. Tanmay and Dan debate whether we will see the end of traditional apps, with Tanmay expressing skepticism. They agree that AI's role might best serve as a complement to existing tools rather than a complete replacement.

Evaluations and Model Confidence in AI

The dialogue flows into the importance of evaluations in AI systems. Early discussions highlight the distinction between evaluating AI systems based on subjective feel versus quantifiable success metrics. Tanmay emphasizes the need for better evaluations (e-valls) to assess AI performance and prioritize continuous improvement.

Transitioning from ML to AI: A New Paradigm

Tanmay reflects on the challenges and navigations within organizations transitioning from traditional machine learning approaches to new AI-centered frameworks. The conversation underscores that AI isn’t merely a tool but an evolving system that requires organization, strategy, and consistent revision, unlike static ML models.

Finding the Right Balance in Development

In the latter part of the podcast, both speakers address the balance needed in developing AI systems - weighing flexibility against complexity, and personalized models versus generalized models. They emphasize the inherent uncertainty within AI development and the necessity for companies to focus on optimizing their feedback loops.

Show notes

We're All Finetuning Incorrectly // MLOps Podcast #304 with Tanmay Chopra, Founder & CEO of Emissary.

Join the Community: https://go.mlops.community/YTJoinIn

Get the newsletter: https://go.mlops.community/YTNewsletter

// Abstract

Finetuning is dead. Finetuning is only for style. We've all heard these claims. But the truth is, we feel this way because all we've been doing is extended pretraining. I'm excited to chat about what real finetuning looks like - modifying output heads, loss functions, and model layers, and its implications on quality and latency. Happy to dive deeper into how DeepSeek leveraged this real version of finetuning through GRPO and how this is nothing more than a rediscovery of our old finetuning ways. I'm sure we'll naturally also dive into when developing and deploying your specialized models makes sense and the challenges you face when doing so.

// Bio

Tanmay is a machine learning engineer at Neeva, where he's currently engaged in reimagining the search experience through AI - wrangling with LLMs and building cold-start recommendation systems. Previously, Tanmay worked on TikTok's Global Trust&Safety Algorithms team - spearheading the development of AI technologies to counter violent extremism and graphic violence on the platform across 160+ countries. Tanmay has a bachelor's and master's in Computer Science from Columbia University, with a specialization in machine learning.

Tanmay is deeply passionate about communicating science and technology to those outside its realm. He's previously written about LLMs for TechCrunch, held workshops across India on the art of science communication for high school and college students, and is the author of Black Holes, Big Bang and a Load of Salt - a labor of love that elucidated the oft-overlooked contributions of Indian scientists to modern science and helped everyday people understand some of the most complex scientific developments of the past century without breaking into a sweat!

// Related Links

~~~~~~~~ ✌️Connect With Us ✌️ ~~~~~~~

Catch all episodes, blogs, newsletters, and more: https://go.mlops.community/TYExplore

Join our Slack community [https://go.mlops.community/slack]

Follow us on X/Twitter [@mlopscommunity](https://x.com/mlopscommunity) or [LinkedIn](https://go.mlops.community/linkedin)]

Sign up for the next meetup: [https://go.mlops.community/register]

MLOps Swag/Merch: [https://shop.mlops.community/]

Connect with Demetrios on LinkedIn: /dpbrinkm

Connect with Tanmay on LinkedIn: /tanmayc98

Timestamps:

[00:00] Tanmay's preferred coffee

[00:17] Takeaways

[00:55] LLM Potential vs Reality

[06:41] Prompting and Workflow Challenges

[13:39] LLM Fine-Tuning vs Prompt Engineering

[16:53] Foundational Models vs ML

[23:32] Vertical vs Horizontal Workflows

[28:25] AI CoE Concerns

[32:09] 500 Examples vs API

[36:38] LLM as DAG Node

[39:26] Success with AI Tools

[43:45] AI for regional ads

[48:13] AI Systems and Infrastructure

[51:13] Prompt Experimentation and Evaluation

[56:38] Python vs IML Tools

[59:32] Wrap up