Talk to Your Data: The SQL Data Analyst

Key takeaways

The Token Data Analyst is designed to assist employees in querying data efficiently, complementing rather than replacing human data analysts.
Effective deployment of such technology requires careful integration with existing data environments and processes.
User feedback is essential for the ongoing improvement of the agent to maintain trust and utility.
New reasoning models will likely advance the capabilities of data analysis tools, enabling more complex queries to be handled accurately.
In-house collaboration with data analysts is critical for refining the agent's outputs and ensuring the tool meets user needs.

Summary

Introduction to the Token Data Analyst

In this episode, Paul and Demetrios introduce the Token Data Analyst, an internal agent that has successfully made it into production within the Prosus group. They share insights on the motivation behind the agent's development, emphasizing the need for accessible data querying tools for employees across various Prosus tech companies. The Token Data Analyst aims to assist users in retrieving information from databases in a manner that mimics the role of a human data analyst.

Integration with Human Data Analysts

The discussion highlights the collaborative nature of the Token Data Analyst. It's not designed to replace human data analysts but to complement their work by addressing common questions quickly and efficiently. Employees can first approach the digital analyst with their data inquiries, and if needed, escalate to human analysts who oversee and refine the outputs for accuracy.

Deployment and Onboarding Challenges

The onboarding process for integrating the Token Data Analyst involves logistical challenges, such as establishing connections with various data sources like Snowflake and BigQuery. The team discusses the need for seamless onboarding experiences while ensuring the data analyst functions within the correct company context and dataset.

Technical Insights and Design Choices

Donnè Stevenson shares the technical foundation behind the Token Data Analyst agent, including its use of machine learning models and the integration of a SQL execution tool. They discuss strategies to ensure that data queries are accurate, including the complexities of SQL generation and the issues associated with model overconfidence, also known as hallucination.

Feedback and Continuous Improvement

To ensure reliability, they emphasize the importance of analyst feedback in refining the agent's outputs. New use cases are developed collaboratively with analysts who familiarize themselves with the system and contribute to shaping its functionality and use cases, ensuring a robust feedback loop.

Future of the Token Data Analyst

Looking ahead, the hosts speculate that advancements in reasoning models will significantly enhance the capabilities of agents like the Token Data Analyst. They discuss the potential for these models to better handle contextual understanding and improve the accuracy of queries, benefiting users in their data-driven decision-making.

Show notes

In Agents in Production [Podcast Limited Series] - Episode Four, Donné Stevenson and Paul van der Boor break down the deployment of a Token Data Analyst agent at Prosus—why, how, and what worked. They discuss the challenges of productionizing the agent, from architecture to mitigating LLM overconfidence, key design choices, the role of pre-checks for clarity, and why they opted for simpler text-based processes over complex recursive methods.

Guest speakers:

Paul van der Boor - VP AI at Prosus Group

Donne Stevenson - Machine Learning Engineer at Prosus Group

Host: Demetrios Brinkmann - Founder of MLOps Community

~~~~~~~~ ✌️Connect With Us ✌️ ~~~~~~~

Catch all episodes, blogs, newsletters, and more: https://go.mlops.community/TYExplore

Join our slack community [https://go.mlops.community/slack]

Follow us on X/Twitter [@mlopscommunity][https://x.com/mlopscommunity] or LinkedIn [https://go.mlops.community/linkedin]

Sign up for the next meetup: [https://go.mlops.community/register]

MLOps Swag/Merch: [https://shop.mlops.community/]

Connect with Demetrios on LinkedIn: /dpbrinkm