11/12/2026
AI Inference: a mini crash course … let’s go! ✨
Whether you run a business or build as an engineer, it’s one of the most important things to understand right now.
Training is how a model gets built. Inference is every time anyone uses it: every prompt, every agent call, every line of code.
Deloitte forecasts inference will be about two-thirds of all AI compute by 2026. With traditional software, one more user costs almost nothing. With AI, every request runs compute. More users, more compute, a bigger bill.
For LLM inference specifically, 4 common ways to bring that cost down:
🔹 KV caching: save earlier work so it isn’t redone
🔹 Continuous batching: keep the GPU busy
🔹 Quantization: smaller numbers, less memory
🔹 Right model, right job: easy requests go to smaller models
There are more too, like prompt caching, speculative decoding and distillation.
Save this for the next time someone asks what inference actually means.
Happy building 🤍