AI Reliability, Spark, Observability, SLAs and Starting an AI Infra Company

Key takeaways

Data infrastructure is evolving, and existing tools are no longer sufficient to meet modern day demands.
AI is transforming how businesses leverage data, enhancing its role as a core product rather than a mere backend component.
There are significant challenges in scaling AI solutions, particularly concerning the handling of unstructured data.
Cross-disciplinary collaboration is key in a rapidly evolving tech landscape, and leveraging community knowledge can enhance innovation.
Robust evaluation methodologies are necessary for AI systems to ensure their outputs are effective and reliable.

Summary

The Evolution of Data Infrastructure

In this section, the hosts discuss how the landscape of data infrastructure has evolved over the past decade. They highlight the obsolete nature of existing tools in light of emerging workloads, particularly with the rise of AI. The conversation emphasizes the need for new tooling that caters to modern demands, especially in the context of data becoming integral to business strategy.

AI as a Catalyst for Change

AI is described as a transformative catalyst reshaping how businesses view data. The hosts explain that AI's capabilities lead to data being viewed not just as a backend element, but as a core product driving business revenue. The rise of AI tools and their impact on traditional data roles is discussed, emphasizing the necessity for teams to adapt and embrace these changes.

Challenges in Implementing AI at Scale

The podcast dives into the complexities of implementing AI solutions and the issues surrounding reliability and scalability. The hosts discuss how many companies face significant hurdles when attempting to integrate AI into their workflows, particularly regarding the nuances of unstructured data and the non-deterministic nature of AI models.

The Importance of Community and Collaboration

Both guests emphasize the importance of community in driving innovation. They discuss how diverse engineering backgrounds contribute to solving complex problems in AI, pointing out that cross-disciplinary collaboration accelerates technological advancements. The episode underscores the need for shared knowledge among industry participants to navigate the evolving landscape.

Evaluating AI Effectiveness

The discussion shifts to the need for robust evaluation methodologies for AI systems. The importance of developing evaluation tools that ensure the reliability and effectiveness of AI outputs is stressed. The hosts propose a need for new frameworks that can trace data lineage through multi-stage AI processing pipelines, ensuring accountability and transparency.

Show notes

LLMs are reshaping the future of data and AI—and ignoring them might just be career malpractice. Yoni Michael and Kostas Pardalis unpack what’s breaking, what’s emerging, and why inference is becoming the new heartbeat of the data pipeline.

// Bio

Kostas Pardalis

Kostas is an engineer-turned-entrepreneur with a passion for building products and companies in the data space. He’s currently the co-founder of Typedef. Before that, he worked closely with the creators of Trino at Starburst Data on some exciting projects. Earlier in his career, he was part of the leadership team at Rudderstack, helping the company grow from zero to a successful Series B in under two years. He also founded Blendo in 2014, one of the first cloud-based ELT solutions.

Yoni Michael

Yoni is the Co-Founder of typedef, a serverless data platform purpose-built to help teams process unstructured text and run LLM inference pipelines at scale. With a deep background in data infrastructure, Yoni has spent over a decade building systems at the intersection of data and AI — including leading infrastructure at Tecton and engineering teams at Salesforce.

Yoni is passionate about rethinking how teams extract insight from massive troves of text, transcripts, and documents — and believes the future of analytics depends on bridging traditional data pipelines with modern AI workflows. At Typedef, he’s working to make that future accessible to every team, without the complexity of managing infrastructure.

// Related Links

Website: https://www.typedef.ai

https://techontherocks.show

https://www.cpard.xyz

~~~~~~~~ ✌️Connect With Us ✌️ ~~~~~~~

Catch all episodes, blogs, newsletters, and more: https://go.mlops.community/TYExplore

MLOps Swag/Merch: [https://shop.mlops.community/]

Connect with Demetrios on LinkedIn: /dpbrinkm

Connect with Kostas on LinkedIn: /kostaspardalis/

Connect with Yoni on LinkedIn: /yonimichael/

Timestamps:

[00:00] Breaking Tools, Evolving Data Workloads

[06:35] Building Truly Great Data Teams

[10:49] Making Data Platforms Actually Useful

[18:54] Scaling AI with Native Integration

[24:04] Empowering Employees to Build Agents

[28:17] Rise of the AI Sherpa

[36:09] Real AI Infrastructure Pain Points

[38:05] Fixing Gaps Between Data, AI

[46:04] Smarter Decisions Through Better Data

[50:18] LLMs as Human-Machine Interfaces

[53:40] Why Summarization Still Falls Short

[01:01:15] Smarter Chunking, Fixing Text Issues

[01:09:08] Evaluating AI with Canary Pipelines

[01:11:46] Finding Use Cases That Matter

[01:17:38] Cutting Costs, Keeping AI Quality

[01:25:15] Aligning MLOps to Business Outcomes

[01:29:44] Communities Thrive on Cross-Pollination

[01:34:56] Evaluation Tools Quietly Consolidating