Evolving Workflow Orchestration // Alex Milowski // #291

Key takeaways

Workflows have evolved from simple rule-based systems to complex graph-based orchestrations, allowing intricate task relationships.
Understanding the proper tools, such as Airflow and Dagster, is essential for effectively managing workflows in data engineering and ML Ops.
Resistance to change in how processes are documented and executed can be overcome by fostering collaboration and using visual representations of workflows.
Emerging trends indicate a shift toward Python-based workflow systems, integrating LLMs, and recognizing the importance of governance in AI deployments.
Awareness of data governance is key in organizations as multiple AI tools can lead to compliance risks and challenges in data leakage.

Summary

Understanding Workflows: A Deep Dive

The episode provides a thorough exploration of workflow systems, tracing their origins back to the 1920s and their evolution over time. Alex Balowski discusses how workflows have shifted from rule-based systems to more complex, graph-based orchestrations. This entails understanding workflows as intricate graphs rather than simple sequences, where each task can alter the flow of data and outcomes—essentially transforming the way tasks interrelate in both technical and business contexts.

The Role of Workflow Systems in Data Engineering and ML Ops

Balowski emphasizes the criticality of workflow systems in data engineering and machine learning operations, detailing how they help streamline processes and improve efficiency. He discusses tools such as Airflow and more recent innovations like Dagster and Metaflow, highlighting the importance of selecting the right tool for diverse technical situations, while also considering human involvement in looping jobs.

Challenges and Solutions in Process Engineering

The conversation addresses the difficulties of implementing process engineering in organizations, especially with varying user experiences. Balowski shares insights from his prior work with scientists, where he encountered resistance to representing operations as workflows rather than traditional procedural documents. He underlines the importance of visual representations in aligning team understanding of processes.

Future Trends and Predictions in Workflow Orchestration

The transcript discusses the current trends in workflow orchestration, including the increasing adoption of Python for workflow specifications and the rise of generative AI, which allows for more dynamic agent-based tasks. The feasibility of integrating LLMs (large language models) into workflows and the subsequent governance and compliance challenges posed by an explosion of AI tools across organizations is also highlighted. Balowski expresses optimism about the growth potential in workflow systems, noting an active and vibrant ecosystem.

The Importance of Governance and Compliance in AI Deployment

Alex discusses the significant data governance challenges that organizations face as they incorporate various AI tools across departments. Emphasizing the necessity for centralizing knowledge around workflows, effective data management can prevent data leaks and maintain compliance in environments that evolve quickly, especially in regulated industries.

Show notes

Alex Milowski is a researcher, developer, entrepreneur, mathematician, and computer scientist.

Evolving Workflow Orchestration // MLOps Podcast #291 with Alex Milowski, Entrepreneur and Computer Scientist.

// Abstract

There seems to be a shift from workflow languages to code, mostly annotation Python - happening and getting us. It is a symptom of how complex workflow orchestration has gotten. Is it a dominant trend or will we cycle back to “DAG specifications”? At Stitchfix, we had our own DSL that “compiled” into airflow DAGs, and at MicroByre, we used an external workflow language. Both had a batch task executor on K8S, but at MicroByre, we had human and robot in the loop workflows.

// Bio

Dr. Milowski is a serial entrepreneur and computer scientist with experience in a variety of data and machine learning technologies. He holds a PhD in Informatics (Computer Science) from the University of Edinburgh, where he researched large-scale computation over scientific data. Over the years, he's spent many years working on various aspects of workflow orchestration in industry, standardization, and research.

// Related Links

Website: https://www.milowski.com/

--------------- ✌️Connect With Us ✌️ -------------

Join our Slack community: https://go.mlops.community/slack

Follow us on Twitter: @mlopscommunity

Sign up for the next meetup: https://go.mlops.community/register

Catch all episodes, blogs, newsletters, and more: https://mlops.community/

// MLOps Swag/Merch

⁠https://shop.mlops.community/⁠

Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/

Connect with Alex on LinkedIn: https://www.linkedin.com/in/alexmilowski/

Timestamps:

[00:00] Alex's preferred coffee

[00:20] Takeaways

[01:27] Workflow Systems Overview

[05:54] Everything as a DAG

[09:25] Workflows, Layering, and Ownership

[12:46] Workflow Awareness for Success

[17:31] Process Engineering Challenges

[21:20] ML Workflow Trends

[27:57] Data Engineering and ML Tools

[33:18] Infrastructure as Code Pros and Cons

[36:20] DSLs in Organizations

[43:04] Agents vs DAGs

[47:48] Agent Workflow Selection

[50:31] RPA Business Automation Tools

[55:00] LLM Integration in BPA

[1:01:14] Diversity in Decision Workflows

[1:03:09] Shadow AI Governance Risks

[1:08:34] Centralized ML Workflow Standardization

[1:12:50] Wrap up