07/09/2026
This video explains the engineering techniques used to make Large Language Models (LLMs) run faster and handle more information at a lower cost.
Key Concepts:
Speculative Decoding (0:00 - 12:40) : A clever way to speed up generation. A small, fast "drafter" model guesses the next few words, and the large "target" model checks them all at once. This gets more work done in a single read of the model's memory.
Managing the KV Cache (12:50 - 20:20) : LLMs store a "cache" of past tokens to avoid re-processing context. As conversations grow long, this cache can become huge. Techniques like *Ring Attention*, *Sparse Attention*, and *Cache Eviction* help the system manage this memory more efficiently.
Production Metrics (22:17 - 24:30) : In real-world use, developers don't just look at speed; they focus on **TTFT** (Time to First Token) and **TPOT** (Time per Output Token) at the 99th percentile to ensure users get fast, reliable answers.
Cost Efficiency (24:30 - 26:30) : By increasing the number of requests that share the same memory read, companies can drastically lower the cost per million tokens, sometimes by a factor of 15.
The Big Picture : The field is shifting toward "goodput," which is about delivering useful answers within strict time limits. As models get better at "thinking" (reasoning chains), they require even more inference work, making these efficiency gains essential for the future of AI.
Episode eight of The Engineering Behind LLM Inference covers specul...
