Streaming Ecosystem Complexities and Cost Management // Rohit Agrawal // #302

Key takeaways

The streaming data ecosystem is complex and often fragmented, making effective data use a significant challenge for many organizations.
Cost management in streaming solutions requires careful consideration of operational expenses, checkpointing frequencies, and data retention strategies.
Emerging technologies and managed services in streaming can help reduce costs and operational burdens, enabling organizations to focus on product velocity.
The trend towards BYOC solutions reflects a growing demand for data governance and control by organizations, allowing them to keep sensitive data securely within their own cloud accounts.
Data architecture is evolving, with platforms like Iceberg positioned to become a central repository for data, enhancing accessibility and reducing the complexity of vendor dependencies.

Summary

Introduction to Rohit Agarwal and Tecton

Rohit Agarwal, Director of Engineering at Tecton, talks about his background and journey in the tech industry, particularly his focus on real-time data engineering. He describes his role in managing teams that work on streaming and batch data solutions, emphasizing the importance of efficiency in data handling.

Understanding the Streaming Data Ecosystem

In this segment, Rohit explains the complexities of the streaming ecosystem, noting that many companies invest in low-latency data streaming solutions like Kafka or Kinesis but struggle with using that data effectively. He discusses the fragmented nature of tools and the challenges in personnel skills required to manage the entire pipeline.

Cost Challenges Associated with Streaming Data

Rohit delves into the cost implications of maintaining complex data systems. He discusses the hidden expenses of operational maintenance, the costs associated with AWS and data processing, and the need for smarter configurations to minimize those costs.

Best Practices for Optimizing Streaming Systems

Key strategies for managing costs include tweaking the frequency of data checkpointing, determining data retention periods for streaming versus batch data, and leveraging newer technologies that can streamline operations. Rohit also mentions up-and-coming technologies that are improving the landscape of streaming solutions.

The Road to Managed Services and BYOC Trends

The conversation shifts to increasing managed services in the streaming industry, like the benefits of 'Bring Your Own Cloud' (BYOC). Rohit discusses how these trends reflect a shift towards offering users more control over their data while also simplifying the management of complex data workflows.

Future of Data Architecture

Rohit posits that data architecture will increasingly rely on systems like Iceberg as the universal standard, making it easier for organizations to manage resources and switch between service providers, reducing vendor lock-in.

Show notes

Streaming Ecosystem Complexities and Cost Management // MLOps Podcast #302 with Rohit Agrawal, Director of Engineering at Tecton.

Join the Community: https://go.mlops.community/YTJoinIn

Get the newsletter: https://go.mlops.community/YTNewsletter

// Abstract

Demetrios talks with Rohit Agrawal, Director of Engineering at Tecton, about the challenges and future of streaming data in ML. Rohit shares his path at Tecton and insights on managing real-time and batch systems. They cover tool fragmentation (Kafka, Flink, etc.), infrastructure costs, managed services, and trends like using S3 for storage and Iceberg as the GitHub for data. The episode wraps with thoughts on BYOC solutions and evolving data architectures.

// Bio

Rohit Agrawal is an Engineering Manager at Tecton, leading the Real-Time Execution team. Before Tecton, Rohit was a Lead Software Engineer at Salesforce, where he focused on transaction processing and storage in OLTP relational databases. He holds a Master’s Degree in Computer Systems from Carnegie Mellon University and a Bachelor’s Degree in Electrical Engineering from the Biria Institute of Technology and Science in Pilani, India.

// Related Links

~~~~~~~~ ✌️Connect With Us ✌️ ~~~~~~~

Catch all episodes, blogs, newsletters, and more: https://go.mlops.community/TYExplore

Join our Slack community [https://go.mlops.community/slack]

Follow us on X/Twitter [@mlopscommunity](https://x.com/mlopscommunity) or [LinkedIn](https://go.mlops.community/linkedin)]

Sign up for the next meetup: [https://go.mlops.community/register]

MLOps Swag/Merch: [https://shop.mlops.community/]

Connect with Demetrios on LinkedIn: /dpbrinkm

Connect with Rohit on LinkedIn: /agrawalrohit10

Timestamps:

[00:00] Rohit's preferred coffee

[00:34] Takeaways

[01:25] Data Streaming

[06:50] Optimizing for Speed

[09:17] DuckDB vs Spark Flink

[13:09] Optimizing Feature Engineering Best Practices

[16:42] Checkpointing Frequency Guide

[23:08] Streaming Tips and Tricks

[30:00] Cloud Costs vs Human Costs

[32:09] Race to Learn

[39:24] Right-Sized Engineering Practices

[42:06] Streaming Simplicity vs Complexity

[47:02] Wrap up