Efficient Deployment of Models at the Edge // Krishna Sridhar // #284

Key takeaways

AI is rapidly evolving, with significant focus on deploying models at the edge rather than relying solely on cloud computing.
Memory and power efficiency are critical challenges when deploying AI models on edge devices.
Qualcomm's AI Hub drastically simplifies the deployment and profiling of AI models for developers.
Practical applications of AI can be found in innovative use cases, enabling real-time functionalities in various domains.
The future of AI on devices is promising, with growing computational capabilities and a movement towards larger-scale implementations.

Summary

Introduction to AI on the Edge

In this episode, host Demetrios Brinkmann welcomes Krishna Sridhar, VP of Engineering at Qualcomm, to discuss the advancements and applications of AI on the edge. Krishna explains the significance of deploying AI models efficiently on devices like smartphones, cars, and appliances. The discourse centers on how Qualcomm's AI Hub facilitates developers in bringing their machine learning models into practical applications, ensuring optimal performance.

The Evolution of AI Technologies

Krishna shares his journey into the AI field, which began with his PhD work on numerical optimization for oil companies in Africa. He transitioned into machine learning, eventually playing a pivotal role in developing Apple's Core ML. The discussion highlights the shifts in AI technology, particularly the growing importance of deploying models directly on devices rather than relying heavily on cloud computing.

Challenges in AI Model Deployment

A fundamental challenge discussed is the constraints regarding memory and power efficiency in deploying AI models on edge devices. Krishna elaborates on the need to optimize not just for compute power but also for battery life, especially in mobile devices. As AI models grow in complexity, the need for efficient memory usage escalates, making it crucial to manage deployment parameters to enhance performance without draining resources.

Innovative Use Cases for Qualcomm AI Hub

Krishna recounts fascinating use cases of Qualcomm AI Hub, including real-time ball tracking for recreational cricket and automatic video clipping functionality. He stresses how the AI Hub empowers developers to create applications that harness local compute power for enhanced user experience. Additionally, technologies for automatic music generation and enhanced productivity tools showcase the versatile capabilities of AI on the edge.

The Role of Qualcomm AI Hub in Developer Empowerment

The episode wraps up with Krishna detailing how Qualcomm AI Hub streamlines the deployment of AI models, enabling developers to profile, validate, and optimize their models with ease. It allows seamless transitions from training models in the cloud to deploying them on various Qualcomm devices, ensuring compatibility across generations of chips. The emphasis is on making AI innovations accessible, efficient, and readily deployable.

Show notes

Krishna Sridhar is an experienced engineering leader passionate about building wonderful products powered by machine learning.

Efficient Deployment of Models at the Edge // MLOps Podcast #284 with Krishna Sridhar, Vice President of Qualcomm.

Big shout-out to Qualcomm for sponsoring this episode!

// Abstract

Qualcomm® AI Hub helps to optimize, validate, and deploy machine learning models on-device for vision, audio, and speech use cases. With Qualcomm® AI Hub, you can: Convert trained models from frameworks like PyTorch and ONNX for optimized on-device performance on Qualcomm® devices.

Profile models on-device to obtain detailed metrics, including runtime, load time, and compute unit utilization. Verify numerical correctness by performing on-device inference. Easily deploy models using Qualcomm® AI Engine Direct, TensorFlow Lite, or ONNX Runtime.

The Qualcomm® AI Hub Models repository contains a collection of example models that use Qualcomm® AI Hub to optimize, validate, and deploy models on Qualcomm® devices. Qualcomm® AI Hub automatically handles model translation from source framework to device runtime, applying hardware-aware optimizations, and performs physical performance/numerical validation. The system automatically provisions devices in the cloud for on-device profiling and inference. The following image shows the steps taken to analyze a model using Qualcomm® AI Hub.

// Bio

Krishna Sridhar leads engineering for Qualcomm™ AI Hub, a system used by more than 10,000 AI developers spanning 1,000 companies to run more than 100,000 models on Qualcomm platforms. Prior to joining Qualcomm, he was Co-founder and CEO of Tetra AI, which made it easy to efficiently deploy ML models on mobile/edge hardware. Prior to Tetra AI, Krishna helped design Apple's CoreML, which was a software system mission-critical to running several experiences at Apple, including Camera, Photos, Siri, FaceTime, Watch, and many more across all major Apple device operating systems and all hardware and IP blocks. He has a Ph.D. in computer science from the University of Wisconsin-Madison and a bachelor’s degree in computer science from Birla Institute of Technology and Science, Pilani, India.

// MLOps Swag/Merch

https://shop.mlops.community/

// Related Links

Website: https://www.linkedin.com/in/srikris/

--------------- ✌️Connect With Us ✌️ -------------

Join our Slack community: https://go.mlops.community/slack

Follow us on Twitter: @mlopscommunity

Sign up for the next meetup: https://go.mlops.community/register

Catch all episodes, blogs, newsletters, and more: https://mlops.community/

Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/

Connect with Krishna on LinkedIn: https://www.linkedin.com/in/srikris/

Timestamps: