You’ve got 99 problems but data shouldn’t be one

Key takeaways

Data engineering faces significant challenges due to a lack of comprehensive tools and scalable practices.
SQL remains an important and foundational tool for data engineers, and there is a need for its integration in more accessible ways.
Unit testing for SQL is often neglected, but innovative solutions like SQL Mesh can revolutionize testing practices.
Understanding the data transformation process is crucial for effective data analysis and reporting.
The evolution of AI continues to shape data practices, requiring careful considerations around data integrity and processing.

Summary

Introduction to Data Engineering

Ryan introduces the podcast and the guests, Toby Mao and Iaroslav Ziegerman, co-founders of Tobiko Data. The discussion begins with their unique journeys into the realm of data engineering, covering their experiences at companies like Netflix and Apple, and how these shaped their perspectives on the industry.

Need for Better Data Tools

The conversation shifts to the challenges data engineers face due to a lack of efficient tools. Both guests reflect on their experiences at Netflix and how despite the talent and best practices, the tooling has not kept pace, making data engineering practices hard to scale.

The Role of SQL in Data Engineering

The guests talk about their philosophy of embracing SQL as a primary interface for data engineers, arguing that it remains vital despite its challenges. They discuss how their company aims to reduce the entry barrier for data engineers by integrating SQL into their new tooling solutions.

Unit Testing in Data Engineering

A significant part of the dialogue focuses on the difficulty of unit testing SQL queries. The discussion encompasses their innovative approach with SQL Mesh, which allows users to transpile SQL code for easier local testing, thereby introducing better practices into data engineering.

SQL Mesh and SQL Glot Overview

Toby and Iaroslav explain the two main products of Tobiko: SQL Mesh and SQL Glot. SQL Glot serves as a SQL parser and transpiler, while SQL Mesh functions as a data transformation and orchestration framework, emphasizing streamlined pipeline management.

Transformations and Pipelines

The guests elaborate on the complexities of maintaining ELT pipelines, mentioning the cultivation of raw data into analytical information and the pitfalls of data modifications within these infrastructures. They emphasize the need for a well-organized workflow to effectively manage evolving data needs.

The Future of Data Engineering and AI

As the episode concludes, the discussion turns toward the implications of AI in data engineering. The guests express thoughts on the balance between leveraging AI in data practices while maintaining rigorous standards to ensure trustworthy results.

Show notes

Tobiko Data is creating a new standard in data transformation with their Cloud and SQL integrations. You can keep up with their work by joining their Slack community.

Connect with Toby on LinkedIn.

Connect with Iaroslav on LinkedIn.

Congrats to Stellar Answer badge winner Christian C. Salvadó, whose answer to What's a quick way to comment/uncomment lines in Vim? was saved by over 100 users.

See Privacy Policy at https://art19.com/privacy and California Privacy Notice at https://art19.com/privacy#do-not-sell-my-info.