Agentic Conversations (formally mlops.community)
MLOps with Databricks // Maria Vechtomova // #314
· Summarized
Key takeaways
- Databricks has emerged as a leading platform for MLOps due to its comprehensive features and ease of integration.
- Users should be aware of both the advantages and limitations of Databricks and approach its features with best practices in mind.
- Data quality challenges persist, especially in collaboration between data producers and consumers, highlighting the need for data contracts.
- Staying informed about the latest advancements in MLOps and related technologies is crucial for practitioners.
- Practical education in MLOps can empower data scientists to effectively utilize platforms like Databricks for real-world applications.
Summary
Understanding Databricks and MLOps
Maria Vechtomova, an MLOps tech lead and co-founder of Marvelous MLOps, delves into the complexities of the MLOps landscape and why platforms like Databricks have become essential. With an extensive background in machine learning tools, Maria focuses on the advantages of using integrated platforms over piecemeal toolsets. She discusses how Databricks has evolved to offer a robust environment for MLOps, helping users efficiently manage machine learning workflows.
The Good, the Bad, and the Ugly of Databricks
The conversation transitions into a candid evaluation of Databricks. Maria shares the pros, such as the platform's user-friendly interface and comprehensive MLOps features, while also shedding light on areas needing improvement, particularly around its feature store and development processes. She emphasizes how many practices in the community don't promote best practices, advocating for better educational resources for data scientists.
Challenges in Data Quality and Team Collaboration
Maria highlights that while her team uses Unity Catalog for data, they face challenges regarding the quality checks conducted by the data-producing team. Although there is an acknowledgment of the need for quality monitoring, discrepancies in priorities make it a complex issue. This ultimately leads to discussions about the necessity of data contracts.
Future Trends in Databricks and MLOps
As the discussion wraps up, Maria reflects on potential advancements and trends in MLOps, particularly in the realms of LLMOps (Large Language Model Operations) and how they can enhance the capabilities of existing ML workflows. She stresses the importance of keeping up with new developments in platforms like Databricks to stay relevant in the fast-evolving tech landscape.
Course and Book on Databricks MLOps
Towards the end of the podcast, Maria highlights her upcoming course on Maven focused on MLOps with Databricks, aiming to provide a highly practical learning experience where participants can build real-world projects. She also discusses her forthcoming book that encapsulates her extensive knowledge and best practices in Databricks MLOps, set to be released in the coming year.
Sponsors
- Databricks · Databricks is a leading platform for MLOps and has become the tool of choice for ML in many companies due to its ease of use and integration with data engineering.
- Marvelous MLOps · Offering a course on MLOps with a focus on practical applications and building projects that can be reused in various companies.
Show notes
MLOps with Databricks // MLOps Podcast #314 with Maria Vechtomova, MLOps Tech Lead | Founder at Ahold Delhaize | Marvelous MLOps.
Join the Community: https://go.mlops.community/YTJoinIn
Get the newsletter: https://go.mlops.community/YTNewsletter
// Abstract
The world of MLOps is very complex as there is an endless amount of tools serving its purpose, and it is very hard to get your head around it. Instead of combining various tools and managing them, it may make sense to opt for a platform instead. Databricks is a leading platform for MLOps. In this discussion, I will explain why it is the case and walk you through Databricks MLOps features.
// Bio
Maria is an MLOps Tech lead with over 10 years of experience in Data and AI.
For the last 8 years, Maria has focused on MLOps and helped to establish MLOps best practices at large corporations.
Together with her colleague, she co-founded Marvelous MLOps to share knowledge on MLOps via training, social media posts, and blogs.
// Related Links
Website: marvelousmlops.io
MLOps Course discount code: MLOPS100 for the podcast listeners - https://maven.com/marvelousmlops/mlops-with-databricks?promoCode=MLOPS100
~~~~~~~~ ✌️Connect With Us ✌️ ~~~~~~~
Catch all episodes, blogs, newsletters, and more: https://go.mlops.community/TYExplore
Join our Slack community [https://go.mlops.community/slack]
Follow us on X/Twitter [@mlopscommunity](https://x.com/mlopscommunity) or [LinkedIn](https://go.mlops.community/linkedin)]
Sign up for the next meetup: [https://go.mlops.community/register]
MLOps Swag/Merch: [https://shop.mlops.community/]
Connect with Demetrios on LinkedIn: /dpbrinkm
Connect with Maria on LinkedIn: /maria-vechtomova
Timestamps:
[00:00] Maria's preferred coffee
[00:42] Takeaways
[02:48] Why Databricks for MLOps
[09:56] Platform Adoption vs Procurement Pain
[12:56] Databricks Best Practices
[16:57] Feature Store Overview
[22:00] Managed system trade-offs
[29:15] Databricks Developments and Trends
[44:31] Insider Info and Summit
[45:47] Data Ownership Pros and Cons
[48:08] Data Contracts and Challenges
[51:25] MLOps Databricks Book Guide
[52:19] Wrap up
Follow this show on AudioBrevity and new episodes get summarized for you. Create a free account