Efficient GPU infrastructure at LinkedIn // Animesh Singh // MLOps Podcast #299

Key takeaways

The rise of LLMs has transformed various industries by creating new use cases, especially in personalized communication and automation.
Rising costs of GPU infrastructure necessitate ongoing optimization and scaling strategies to ensure efficient use and return on investment.
Simplifying AI infrastructure and model deployment is crucial to keep pace with advancements in technology and attract talent.
Optimizing checkpointing processes is essential for maintaining efficiency during long training sessions while minimizing downtime.
Bridging traditional ML with LLM technologies will require ongoing efforts in standardization of processes and tools.

Summary

The Rise of LLMs and Their Impact on Industry

Animesh discusses the significant rise of Large Language Models (LLMs) over recent years, especially after the introduction of ChatGPT. He outlines how LLMs have proven effective across various industries, creating new use cases such as profile summarization on LinkedIn and enhancing recruitment processes through personalized communication. The discussion also emphasizes the importance of optimizing GPU infrastructure to support LLM capabilities.

Challenges of GPU Infrastructure and Cost Efficiency

The conversation highlights the challenges faced regarding the efficiency of GPU infrastructure, especially concerning the rising costs associated with training and inferencing. Animesh shares insights into LinkedIn's investments in scaling their GPU capacities and discusses the importance of improving the return on investment. The focus is on finding ways to optimize GPU usage, reduce costs, and increase the performance of inferencing operations.

The Evolution of AI Infrastructure and Model Deployment

Animesh elaborates on the evolving landscape of AI infrastructure, particularly the shift from traditional ML models to LLMs and agent-based systems. He discusses the need for a more streamlined architecture, which simplifies model deployment, improves scalability, and attracts relevant talent. The challenges of balancing infrastructure costs with advancements in AI technology are also addressed, along with the need for effective monitoring and observability.

Checkpointing and Resilience in Training Processes

The discussion covers the complexities surrounding checkpointing in model training, particularly for LLMs. Animesh explains the strategies LinkedIn utilizes to improve checkpointing efficiency, such as hierarchical checkpointing that allows for rapid data transfer and restoration. He emphasizes the importance of minimizing interruptions during long training jobs and ensuring that data is securely and swiftly saved.

Bridging Traditional ML and LLMs

Animesh touches on the need to bridge the gap between traditional ML applications and new LLM technologies. He outlines how LinkedIn is revamping its machine learning training pipelines to accommodate both paradigms. The ongoing efforts to standardize processes and tools are highlighted, looking forward to a future where LLM architectures can be integrated into traditional ML frameworks.

Show notes

Building Trust Through Technology: Responsible AI in Practice // MLOps Podcast #299 with Animesh Singh, Executive Director, AI Platform and Infrastructure of LinkedIn.

Join the Community: https://go.mlops.community/YTJoinIn

Get the newsletter: https://go.mlops.community/YTNewsletter

// Abstract

Animesh discusses LLMs at scale, GPU infrastructure, and optimization strategies. He highlights LinkedIn's use of LLMs for features like profile summarization and hiring assistants, the rising cost of GPUs, and the trade-offs in model deployment. Animesh also touches on real-time training, inference efficiency, and balancing infrastructure costs with AI advancements. The conversation explores the evolving AI landscape, compliance challenges, and simplifying architecture to enhance scalability and talent acquisition.

// Bio

Executive Director, AI and ML Platform at LinkedIn | Ex IBM Senior Director and Distinguished Engineer, Watson AI and Data | Founder at Kubeflow | Ex LFAI Trusted AI NA Chair

Animesh is the Executive Director leading the next-generation AI and ML Platform at LinkedIn, enabling the creation of the AI Foundation Models Platform, serving the needs of 930+ million members of LinkedIn. Building Distributed Training Platforms, Machine Learning Pipelines, Feature Pipelines, Metadata engines, etc. Leading the creation of the LinkedIn GAI platform for fine-tuning, experimentation, and inference needs. Animesh has more than 20 patents and 50+ publications.

Past IBM Watson AI and Data Open Tech CTO, Senior Director, and Distinguished Engineer, with 20+ years of experience in the Software industry, and 15+ years in AI, Data, and Cloud Platform. Led globally dispersed teams, managed globally distributed projects, and served as a trusted adviser to Fortune 500 firms. Played a leadership role in creating, designing, and implementing Data and AI engines for AI and ML platforms, led Trusted AI efforts, and drove the strategy and execution for Kubeflow, OpenDataHub, and execution in products like Watson OpenScale and Watson Machine Learning.

// Related Links

Composable Memory for GPU Optimization // Bernie Wu // Pod #270 - https://youtu.be/ccaDEFoKwko

~~~~~~~~ ✌️Connect With Us ✌️ ~~~~~~~

Catch all episodes, blogs, newsletters, and more: https://go.mlops.community/TYExplore

Join our Slack community [https://go.mlops.community/slack]

Follow us on X/Twitter [@mlopscommunity](https://x.com/mlopscommunity) or [LinkedIn](https://go.mlops.community/linkedin)]

Sign up for the next meetup: [https://go.mlops.community/register]

MLOps Swag/Merch: [https://shop.mlops.community/]

Connect with Demetrios on LinkedIn: /dpbrinkm

Connect with Animesh on LinkedIn: /animeshsingh1

Timestamps:

[00:00] Animesh's preferred coffee

[00:16] Takeaways

[02:12] What is working?

[07:00] What's not working?

[13:40] LLM vs Rexis Efficiency

[21:49] GPU Utilization and Architecture

[27:32] GPU reliability concerns

[36:50] Memory Bottleneck in AI

[41:06] Optimizing LLM Checkpointing

[46:51] Checkpoint Offloading and Platform Design

[54:55] Workflow Divergence Points

[58:41] Wrap up