Skip to main content
🚀 Inference optimization 40 articles · 9 chapters

Module 4: Inference optimization

LLM inference basics, inference engine internals, common frameworks, quantization, speculative decoding, prefill/decode disaggregation, on-device inference, profiling and end-to-end practice.

Start learning

Chapters

1 Chapter 1: LLM inference basics 2 articles
How LLMs generate text in two phases (prefill and decode), how the KV cache works, and which metrics matter for inference performance
2 Chapter 2: Core inference engine techniques 5 articles
PagedAttention, Continuous Batching, Prefix Cache, Chunked Prefill, Attention backends, and graph optimization—the five core techniques in modern inference engines
3 Chapter 3: Reading the nano-vllm source 6 articles
A ~1200-line minimal inference engine as the through-line: continuous batching, PagedAttention, FlashAttention, tensor parallelism, and CUDA Graph as they appear in real code
4 Chapter 4: Quantization 6 articles
From scale and granularity through SmoothQuant, GPTQ/AWQ, KV cache quantization, and FP8/NVFP4/MXFP4, then pick a scheme and run controlled experiments on vLLM v0.32.0
5 Chapter 5: Speculative Decoding 5 articles
Draft + Verify and distribution-preserving sampling, comparing independent drafts, N-gram/Suffix, Medusa/EAGLE/MTP, plus where the gains stop and how to measure them in vLLM
6 Chapter 6: Distributed inference and large-model deployment 6 articles
Storage and communication costs of TP, PP, DP, and EP for inference, plus single-node and multi-node vLLM deploys, MoE DP+EP, DCP, and controlled experiments to pick a layout
7 Chapter 7: Prefill/Decode disaggregation
Problems with mixed P/D batching, DistServe/Splitwise-style disaggregation, KV cache transfer, and goodput with SLO-aware scheduling
8 Chapter 8: Production deployment and operations
Containerizing and deploying vLLM inference services on Kubernetes, plus observability, autoscaling, load balancing, and capacity planning
9 Chapter 9: On-device inference 1 articles
Resource constraints, heterogeneous hardware, runtimes, and optimization methods for on-device inference, plus a reproducible performance evaluation workflow

Study tips

Prerequisites

The prerequisites module plus basic CUDA programming

  • The LLM inference basics chapter explains what makes inference different from training.
  • Quantization and speculative decoding are where most inference work is happening now.
  • Practice deploying with a real inference framework such as vLLM.
  • For on-device inference, measure latency, memory, power and accuracy on a real device.