Software Huddle
Deep Dive into Inference Optimization for LLMs with Philip Kiely
This episode isn’t summarized yet
Create a free account and summarize it. In a few minutes you’ll have the key takeaways and a section-by-section summary.
Show notes
Today we have Philip Kiely from Baseten on the show. Baseten is a Series B startup focused on providing infrastructure for AI workloads.
We go deep on Inference Optimization. We cover choosing a model, discuss the hype around Compound AI, choosing an Inference Engine, Optimization Techniques like Quantization and Speculative Decoding all the way down to your GPU choice.