Episode 52: Automating the Automators – How AI and ML are Transforming Data Teams · Value Driven Data Science summary

Key takeaways

AI and ML can significantly enhance the efficiency of data engineering teams, helping them focus on more meaningful tasks.
Kibo automates cloud warehouse optimizations, potentially reducing costs and freeing up resources for data professionals.
The complexity of modern data pipelines requires technology that can adapt and optimize without manual intervention.
Data professionals should embrace AI tools rather than resist them to stay relevant and enhance their value within organizations.
Focusing on innovation and continuous learning is crucial for data scientists to thrive in an increasingly automated landscape.

Summary

Introduction to AI and Data Engineering

In this episode, Dr. Genevieve Hayes welcomes Bazan Mosephari, co-founder and CEO of Keebo, to discuss how AI and machine learning are revolutionizing data professionals’ roles, particularly for data engineers. They explore how these technologies can enhance job performance not just for the broader organization but for the data teams themselves.

The Rise of Kibo and Modern Data Pipelines

Bazan shares insights about his work at the intersection of AI and data systems, which led to the founding of Keebo, a platform designed to streamline data analytics. They discuss how modern data pipelines have become more complex and costly, highlighting the challenges faced by data engineers in optimizing these intricate systems.

Cost Optimization and Automation with Kibo

Kibo focuses on automating the optimization process for cloud data warehouses, specifically targeting operational costs. The platform employs reinforcement learning to make real-time adjustments, allowing companies to potentially reduce their bills by up to 40% while freeing up engineering resources.

Innovation with AI: Techniques and Applications

The episode delves into the various AI and machine learning techniques Kibo utilizes. Bazan emphasizes the importance of not assuming uniformity among customer workloads, thus employing diverse models that adapt to specific customers' needs without compromising performance.

Navigating the Build vs. Buy Dilemma

Genevieve and Bazan discuss the hesitance of data professionals to adopt third-party tools versus building in-house solutions. Bazan asserts that leveraging external tools can lead to faster implementation and optimized resources, which ultimately allows teams to focus on value-adding activities.

Future Trends in Data Science

Bazan predicts the evolution of AI technology within data science, suggesting a shift towards more automated solutions for mundane tasks and an emphasis on the value of data scientists embracing these innovations. He encourages data professionals to grow their knowledge of AI and seek ways to integrate these technologies into their workflows.

Show notes

In many organisations, data scientists and data engineers exist as support staff. Data engineers are there to make data accessible to data scientists and data analysts, and data scientists are there to make use of that data to support the rest of the business.

But in helping everyone else in the business, data professionals can often forget to help themselves.

However, just as AI and machine learning can be used to help others in the organisation perform their jobs more effectively, there’s no reason why they can’t also be used to help data professionals excel in their own jobs. And as experts in applying these techniques, data scientists are perfectly placed to leverage them.

In this episode, Prof Barzan Mozafari joins Dr Genevieve Hayes to discuss how AI and machine learning are helping data professionals do their jobs more effectively.

Guest Bio

Prof. Barzan Mozafari is the co-founder and CEO of Keebo, a turn-key data learning platform for automating and accelerating enterprise analytics. He is also an Associate Professor of Computer Science at the University of Michigan and Prof. Barzan Mozafari is the co-founder and CEO of Keebo, a turn-key data learning platform for automating and accelerating enterprise analytics. He is also an Associate Professor of Computer Science at the University of Michigan and has won several awards for his research at the intersection of machine learning and database systems.

Highlights

(00:05) Meet Barzan Mozafari
(00:50) The role of AI in data engineering
(01:36) The birth of Keebo
(02:34) Challenges in modern data pipelines
(05:41) How Keebo optimizes data warehousing
(07:35) AI and ML techniques behind Keebo
(08:47) Reinforcement learning in practice
(16:23) Guardrails and safeguards in AI systems
(26:29) The build vs. buy dilemma
(36:03) Future trends in data science and AI
(39:36) Final advice for data scientists
(40:50) Closing remarks and contact information

Links

Keebo website
Connect with Barzan on LinkedIn
Connect with Genevieve on LinkedIn
Be among the first to hear about the release of each new podcast episode by signing up HERE

Episode 52: Automating the Automators – How AI and ML are Transforming Data Teams