LLM Distillation and Compression // Guanhua Wang // #278

Key takeaways

DeepSpeed and Domino represent significant advances in making large language model training more efficient by addressing communication overhead.
High-quality data is crucial for optimizing small language models like Phi, and the extent of preprocessing can greatly affect model performance.
Quantization is emerging as a preferred technique over distillation for improving model efficiency without significant accuracy loss.
Domino's innovative approach hides communication latency, thereby optimizing training time for LLMs with high bandwidth requirements.
The ongoing evolution of hardware will likely define the next steps for frameworks like Domino, making them more scalable and efficient.

Summary

Introduction to DeepSpeed and Domino

Demetrios introduces Guanhua Wang, a senior researcher at Microsoft, who discusses his involvement with the DeepSpeed team and the development of the Domino training engine. The conversation touches on the significance of optimizing large language models (LLMs) to reduce training times, emphasizing the challenges related to communication overhead in distributed training environments. Demetrios encourages listeners to share the podcast with a friend.

Understanding Phi and the Transition from 5-3 to Other Models

Guanhua explains the project known as Phi, highlighting the transition from Phi 2 to Phi 3, and the rationale behind discontinuing further iterations like 5-4 or 5-5. He discusses the importance of high-quality data in training small language models and reflects on the potential of these models versus larger ones. The conversation reveals how optimization requires significant effort and quality data.

DeepSpeed Overview

Guanhua elaborates on DeepSpeed and its essential features, such as the zero optimizer, which allows memory-efficient data parallelism. He explains the mechanics of how each GPU manages only a part of the model's parameters, leading to improved performance compared to traditional methods. The discussion includes innovations like data offloading to alleviate GPU memory pressure, which facilitates larger model training.

Quantization versus Distillation Techniques

The episode dives into the comparison between quantization and distillation in model optimization. Guanhua argues for the efficacy of quantization, especially in scenarios demanding high accuracy from smaller models, while distillation might compromise performance. He notes that while distillation can work for personalized applications, it may not generalize well to broader contexts.

Exploring Domino and Communication Strategies

The latter part of the podcast revolves around Domino, focusing on its potential to hide communication overhead during the training of LLMs. Guanhua explains how Domino organizes independent training pieces into smaller units that allow task overlapping, thus optimizing computation time. Comparisons are drawn with Megatron, revealing how Domino can achieve speed boosts in training without significant additional complexity.

Future Directions and Hardware Scalability

Guanhua discusses the future of Domino in light of hardware advancements, particularly how the increasing bandwidth capabilities of GPUs can further enhance training efficiency. The conversation hints at ongoing projects, such as integrating quantization techniques into Domino, envisioning how it could evolve with the trends in hardware development. Guanhua notes his excitement about collaborating with various stakeholders across industry and academia.

Show notes

Guanhua Wang is a Senior Researcher in DeepSpeed Team at Microsoft. Before Microsoft, Guanhua earned his Computer Science PhD from UC Berkeley. Domino: Communication-Free LLM Training Engine

// MLOps Podcast #278 with Guanhua "Alex" Wang, Senior Researcher at Microsoft.

// Abstract

Given the popularity of generative AI, Large Language Models (LLMs) often consume hundreds or thousands of GPUs to parallelize and accelerate the training process. Communication overhead becomes more pronounced when training LLMs at scale. To eliminate communication overhead in distributed LLM training, we propose Domino, which provides a generic scheme to hide communication behind computation. By breaking the data dependency of a single batch training into smaller independent pieces, Domino pipelines these independent pieces of training and provides a generic strategy of fine-grained communication and computation overlapping. Extensive results show that compared with Megatron-LM, Domino achieves up to 1.3x speedup for LLM training on Nvidia DGX-H100 GPUs.

// Bio

Guanhua Wang is a Senior Researcher in the DeepSpeed team at Microsoft. His research focuses on large-scale LLM training and serving. Previously, he led the ZeRO++ project at Microsoft, which helped reduce over half of model training time inside Microsoft and LinkedIn. He also led and was a major contributor to Microsoft Phi-3 model training. He holds a CS PhD from UC Berkeley, advised by Prof Ion Stoica.

// MLOps Swag/Merchhttps://shop.mlops.community/

// Related Links

Website: https://guanhuawang.github.io/

DeepSpeed hiring: https://www.microsoft.com/en-us/research/project/deepspeed/opportunities/

Large Model Training and Inference with DeepSpeed // Samyam Rajbhandari // LLMs in Prod Conference: https://youtu.be/cntxC3g22oU

--------------- ✌️Connect With Us ✌️ -------------

Join our Slack community: https://go.mlops.community/slack

Follow us on Twitter: @mlopscommunity

Sign up for the next meetup: https://go.mlops.community/register

Catch all episodes, blogs, newsletters, and more: https://mlops.community/

Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/

Connect with Guanhua on LinkedIn: https://www.linkedin.com/in/guanhua-wang/