Skip to main content
#Announcement #Update log

AIInfraGuide article update log

A record of every content update to the AIInfraGuide knowledge base, so readers can track what changed

This page keeps a running record of AIInfraGuide knowledge-base content updates, newest first, so you can see what changed.


2026-11-07

New content

Module 4 · Inference optimization Chapter 6 “Distributed inference and large-model deployment” is live. It goes from the four parallel cuts used in inference to single-node and multi-node vLLM deploy plus side-by-side experiments. Config is based on vLLM v0.32.0, with 10 new hand-drawn-style diagrams:

Changes


2026-11-01

Changes

Module 4 · Inference optimization Chapter 3 “Reading the nano-vllm source” — 6 articles revised: source walkthrough aligned to the nano-vllm main commit from 2026-04-26, 3 new diagrams, all 8 existing Mermaid diagrams redrawn as hand-drawn-style SVG, and some inaccurate wording fixed:

  • Article: 3.1 Project overview and four-layer architecture | Project info table updates line counts, versions and deps, Star count; four-layer architecture and request-lifecycle diagrams become SVG; notes that Prefill and Decode are not mixed in one batch

  • Article: 3.2 Config, entry points, and one full generate path | Adds constraints for tensor_parallel_size, ignore_eos, and similar params; notes nano-vllm does not support greedy sampling, and that the README example skips a chat template; notes step returns num_tokens with sign distinguishing phases

  • Article: 3.3 Model skeleton and core ops | Tensor-parallel split diagram becomes SVG; new Sampler section; explains where the 2 all_reduces per layer come from, and how KV head count limits TP degree

  • Article: 3.4 KV Cache and BlockManager | Prefix-cache section split into hash chain, registration, and match; new prefix-hit allocation diagram, match flowchart becomes SVG; notes the last block is not looked up, the difference between hit-in-use and free blocks, and reuse truncation from whole-block granularity

  • Article: 3.5 Scheduler and continuous batching | Sequence-state and scheduling-loop diagrams become SVG; explains the three breaks in the prefill branch one by one, head-of-line blocking from strict FCFS; notes preemption uses recompute, and when while ... else runs

  • Article: 3.6 ModelRunner, CUDA Graph, and performance benchmarks | New slot_mapping compute diagram and CUDA Graph tier diagram; run flowchart and full-chapter call-chain diagram become SVG; notes 6-D KV Cache layout, tier caps vs max_num_seqs; performance section adds test conditions and where conclusions apply

Module 4 · Inference optimization Chapter 4 “Quantization” — 6 articles revised: 7 new diagrams, 4 existing flat illustrations redrawn in hand-drawn style, existing Mermaid diagrams all become SVG, and some wording fixed against vLLM v0.32.0 source:

  • Article: 4.1 Quantization basics | New four-granularity diagram; redrawn symmetric/asymmetric number line; adds 4-bit symmetric encoding range; PyTorch quantization references switched to torchao docs

  • Article: 4.2 W8A8 and SmoothQuant | Redrawn paired-scale diagram; offline quantization flowchart becomes SVG

  • Article: 4.3 Weight-only INT4 | New W4A16 pack and on-chip dequant diagram, GPTQ/AWQ comparison diagram; Marlin pipeline diagram becomes SVG

  • Article: 4.4 KV Cache quantization | Redrawn KIVI grouping diagram; notes Value residual region is quantized out token-by-token under a sliding window

  • Article: 4.5 FP8 and NVFP4/MXFP4 | New FP8 vs INT8 scale-distribution comparison; redrawn FP4 microblock scale diagram; online FP8 path adds fp8_per_channel

  • Article: 4.6 Quantization selection and vLLM practice | Selection flowchart becomes SVG; fixes online FP8 load path (v0.32.0 builds on meta first and converts layer by layer; it does not load full 16-bit weights at once)

Module 4 · Inference optimization Chapter 5 “Speculative Decoding” revised: 1 new diagram, all 4 existing Mermaid diagrams redrawn as hand-drawn-style SVG, and some wording fixed against vLLM v0.32.0 source:

  • Article: 5.1 Speculative decoding core ideas | Draft + Verify one-round flowchart becomes SVG, with accept/correct/reject positions marked for Îł = 4; new residual-sampling distribution-preserving diagram, using section 7 numbers to show the two paths sum to p

  • Article: 5.2 Draft models and draft-free methods | N-gram lookup flowchart becomes SVG, marking vLLM’s longest-then-earliest match rule; fixes where ngram_gpu applies (both V1 and V2 Model Runner); notes default when Suffix has no candidate cap

  • Article: 5.3 Medusa, EAGLE, and MTP | Draft Tree diagram becomes SVG, same tree as section 8 experiments, with tree Mask and depth-numbered Position IDs; notes Block acceptance only in V2 Model Runner; fixes MTP conditions that avoid version mismatch

  • Article: 5.4 Benefit boundaries and limits | Scheduling-loop diagram becomes SVG; adds an example of different step lengths for requests in the same batch; fixes Adaptive Verification support range (not only DSpark) and the N-gram/Suffix benefit description under high QPS

  • Article: 5.5 vLLM speculative decoding in practice | Per-position accept metrics use the actual exposed ..._per_pos_total name and explain the counters; adds --speculative-config short form -sc


2026-10-29

Changes

Module 4 · Inference optimization Chapter 2 “Core inference-engine techniques” fully revised: 5 articles gain 9 new diagrams, default params and source walkthroughs aligned to vLLM v0.32.0, and some outdated or inaccurate wording fixed:


2026-10-26

Changes

Module 4 · Inference optimization Chapter 4 “Quantization” — 6 articles revised, framework content aligned to vLLM v0.32.0, each article gains an information-freshness note, and chapter structure reorganized:

  • Article: 4.1 Quantization basics: Scale, granularity, and PTQ/QAT | Rewrites quantize/dequantize with a “ruler with limited ticks” metaphor; bit-width, matrix axes, and granularity become separate sections; calibration gets its own chapter covering range choice and static vs dynamic calibration

  • Article: 4.2 W8A8 quantization: how SmoothQuant handles activation outliers | Equivalent scaling gets formulas and an intuitive single-dot-product derivation; the smoothing-coefficient section becomes how to choose migration strength α; adds offline flow, benefits and fit conditions, plus a hands-on dot-product example

  • Article: 4.3 Weight-only INT4: GPTQ, AWQ, and Marlin | Adds GPTQ cost and effect, AWQ’s idea of scaling instead of mixed precision; new vLLM kernel selection, quantization-tool choice, and debug order

  • Article: 4.4 KV Cache quantization: KIVI and FP8 cache | New KV footprint estimate; FP8 KV Cache section rewritten for v0.32.0 covering format choice, Scale calibration, skipping sensitive layers, and storage vs compute precision; notes the old calculate_kv_scales param was removed

  • Article: 4.5 FP8 and NVFP4/MXFP4: how low-bit floats work | Adds E2M1’s 8 magnitudes, microblock shape notation and Scale’s own format; compares NVFP4 two-level Scale vs MXFP4 one-level Scale; new usage in vLLM v0.32.0

  • Article: 4.6 Quantization selection and vLLM practice | Separates quantization method, checkpoint format, and Kernel as three layers; explains vLLM quantization entry points; experiment baseline updated to v0.32.0

Module 4 · Inference optimization Chapter 5 “Speculative Decoding” — 5 articles revised, config baseline upgraded from vLLM v0.31.0 to v0.32.0, with new MTP content:


2026-10-12

New content

Module 4 · Inference optimization Chapter 3 “Reading the nano-vllm source” is live. Using the ~1200-line minimal inference engine nano-vllm as the spine, it walks module by module through how continuous batching, PagedAttention, FlashAttention, tensor parallelism, and CUDA Graph are organized in real code:

  • Article: 3.1 Project overview and four-layer architecture | See nano-vllm’s role and directory layout; one four-layer architecture diagram covers the full data path from user request to token output

  • Article: 3.2 Config, entry points, and one full generate path | Starting from example.py, break down Config/SamplingParams/LLM/LLMEngine and see how the generate main loop drives add_request, step, and postprocess

  • Article: 3.3 Model skeleton and core ops | Layer-by-layer Qwen3: RMSNorm, RoPE, SwiGLU, parallel Linear and tensor-parallel splits, vocab-parallel Embedding/LMHead, and Attention’s KV write plus the two FlashAttention paths

  • Article: 3.4 KV Cache and BlockManager | Blocked KV management: Block structure, PagedAttention’s block idea, xxhash prefix cache, block alloc/free, and Sequence state machine plus block_table

  • Article: 3.5 Scheduler and continuous batching | Scheduling core: Waiting/Running dual queues, Prefill/Decode split scheduling, Token Budget, chunked prefill, and preemption plus recompute when memory is short

  • Article: 3.6 ModelRunner, CUDA Graph, and performance benchmarks | Sequence tensorization, global Context injection, KV Cache memory splits, CUDA Graph capture and replay, plus official benchmarks vs vLLM

Changes

  • The old vLLM quick start moves under Chapter 1 as 1.2 vLLM quick start, as a hands-on entry to inference basics.

2026-10-08

New content

Module 4 · Inference optimization Chapter 4 “Quantization” — 6 articles live:


2026-10-07

New content

Module 4 · Inference optimization Chapter 5 “Speculative Decoding” — 5 articles live:


2026-09-15

New content

Module 4 · Inference optimization adds Chapter 12 “On-device inference”:


2026-07-09

New content

Module 4 · Inference optimization Chapters 1 and 2 articles live:

  • Article: 1.1 LLM inference basics | One article on the underlying logic of LLM inference: Prefill/Decode phases, the KV Cache memory ledger, metrics such as TTFT/TPOT, and using Roofline to explain why Decode is Memory Bound

  • Article: 2.1 PagedAttention | Deep dive into vLLM’s core technique PagedAttention: borrow virtual-memory paging, use a Block Table to remove KV Cache fragmentation, and support prefix sharing plus Copy-on-Write

  • Article: 2.2 Continuous Batching | Deep dive into Continuous Batching: Iteration-level Scheduling lets requests join as they arrive and leave when done, pulling GPU utilization from about 30% to 80%+

  • Article: 2.3 Prefix Cache and RadixAttention | Deep dive into prefix cache: how vLLM’s hash-based automatic prefix cache reuses shared-prefix KV blocks, and how SGLang’s RadixAttention uses a Radix Tree for more efficient prefix sharing

  • Article: 2.4 Chunked Prefill and unified scheduling | Deep dive into Chunked Prefill: chunk long-prompt Prefill so it stops interfering with Decode; how vLLM V1’s unified Token-budget scheduler flattens the Prefill/Decode boundary

  • Article: 2.5 Attention backends and graph optimization | Deep dive into vLLM’s pluggable Attention backends (FlashAttention/FlashInfer/FlashMLA/Triton) and graph opts: CUDA Graph and torch.compile cut Decode-phase CPU launch overhead

Module 3 · Distributed training collective communication, optimizer, and data-parallel chapter articles live:


2026-07-07

New content


2026-07-06

New content

Updated the distributed-training tutorial outline:

ChapterMain content
Chapter 1 Distributed training overviewWhy distributed training, training memory ledger, five parallel strategies, environment setup and DDP launch
Chapter 2 Collective communication primitivesAllReduce/ReduceScatter/AllGather/All-to-All semantics, Ring algorithm, communication-volume analysis
Chapter 3 OptimizersSGD/Adam/AdamW evolution, optimizer-state memory cost, large-batch optimizers LAMB/LARS
Chapter 4 Data parallelismDataParallel, DistributedDataParallel, FSDP
Chapter 5 ZeRO seriesZeRO-1/2/3 sharding and communication cost, ZeRO-Offload/Infinity heterogeneous memory offload
Chapter 6 Tensor parallel and sequence parallelMegatron Column/Row Parallel Linear, where communication is inserted, sequence-parallel activation memory opts
Chapter 7 Pipeline parallelBubble problem, GPipe, 1F1B, Interleaved schedule
Chapter 8 Mixed precision and memory optsFP16/BF16/FP8 mixed precision, gradient accumulation, Activation Checkpointing
Chapter 9 Long-sequence training and context parallelAttention’s O(s²) problem, Ring Attention, DeepSpeed-Ulysses, Megatron Context Parallel
Chapter 10 MoE parallelRouter mechanism, Expert Parallelism All-to-All, EPĂ—DPĂ—TPĂ—PP mixes and load balance
Chapter 11 3D parallel and hybrid strategiesTPĂ—PPĂ—DPĂ—EPĂ—CP topology design, communication-domain mapping, rank layout, cluster selection

2026-06-25

New content

  • Article: 6.1 FlashAttention V1 in depth | Deep dive into FlashAttention V1 principles and implementation: how Tiling + Online Softmax cuts Attention’s extra memory from O(N2)O(N^2) to O(N)O(N), sharply reduces HBM traffic, and speeds up without accuracy loss

  • Article: 6.2 FlashAttention V2 in depth | This this article deep-dives FlashAttention V2’s core changes vs V1: swap inner/outer loop order, improve work distribution inside the thread block, and cut non-matmul ops


2026-06-06

New content


2026-06-05

New content


2026-06-02

New content

  • Article: 4.1 CUDA GEMM operator performance optimization | From a naive implementation through Block-Warp-Thread three-level tiling, vectorized loads, Bank Conflict removal, double buffering, and more — a full methodology for CUDA GEMM optimization

2026-05-18

New content

  • Article: 3.1 CUDA Reduce operator optimization | Reduce is one of the most basic GPU ops and one of the best for learning parallel thinking. From the simplest implementation, this article gradually adds Warp-level primitives, vectorized loads, multi-element processing, and more — each step with performance comparisons and explanations so you really understand how to write a fast Kernel.

2026-05-16

New content

  • Module 3 · Distributed training adds Chapter 2: Optimizers | Follow SGD → Momentum → Adam → AdamW evolution and internal state makeup; hand-compute optimizer memory cost (AdamW is 75% of training memory); know when large-batch optimizers LAMB/LARS fit; foundation for later ZeRO memory opts and mixed-precision training.

Structure changes

  • Module 3 · Distributed training chapters renumbered: old Chapters 2–7 become 3–8; new Chapter 2 “Optimizers” sits between the overview and data parallelism
  • Synced the AI Infra learning path and the Module 3 chapter list in the README

2026-05-13

New content

  • Article: 2.4 Synchronization and atomic operations | Correct sync is the base of bug-free parallel programs. This article covers CUDA block sync __syncthreads(), Warp-level sync, Memory Fence, and when to use atomics, their cost, and how to optimize — so you can write high-performance parallel code without giving up correctness.

2026-05-12

New content

  • Article: 2.2 Memory-access optimization | Memory-access efficiency is the biggest lever on CUDA Kernel performance. Coalesced Access principles and checks, shared-memory Bank Conflict causes and Padding fixes, and vectorized loads (float4/int4) to raise bandwidth use.

  • Article: 2.3 Occupancy and resource allocation | Occupancy measures active Warps on an SM vs the theoretical max — a core CUDA Kernel tuning metric. Definition, how to compute it, three limiters (registers/shared memory/Block size), and why higher Occupancy is not always better — the real goal is balance between latency hiding and resource use.


2026-05-11

New content


2026-05-09

New content

  • Article: CUDA memory model | Global memory, shared memory, registers, constant memory, and unified memory — traits, fit cases, and optimization tips

  • Article: First useful Kernel: vector add | A full vector-add case that walks the whole CUDA programming path


2026-05-08

New content


2026-05-07

New content


2026-04-28

New content


2026-04-27

New content


2026-04-26

New content


2026-04-25

New content


2026-04-24

New content


2026-04-23

New content


2026-04-21

New content


2026-04-20

New content


2026-04-18

New content


2026-04-17

New content

  • Interview section: 180+ real interview sessions covering 40+ companies, organized by tier. Browse online →
  • Companies include ByteDance, Alibaba, Tencent, Baidu, Kuaishou, Meituan, Ant Group, NVIDIA, MiniMax, NIO, XPeng, Li Auto, and more.
  • Community entry: new WeChat discussion group, welcome to join

Content improvements

  • Updated the README and added a project intro image

2026-04-16

Major update

New content


2026-04-15

Project launch

  • AIInfraGuide knowledge base officially live
  • Published [AI Infra learning path](/AIInfraGuide/guides/AI Infra学习路线) overview
  • Published Understanding AI Infra from scratch starter guide

2026-04-14

Project init

  • Website framework built (Astro + Starlight)
  • Repo initialized, first commit published