AIInfraGuide article update log
A record of every content update to the AIInfraGuide knowledge base, so readers can track what changed
This page keeps a running record of AIInfraGuide knowledge-base content updates, newest first, so you can see what changed.
2026-11-07
New content
Module 4 · Inference optimization Chapter 6 “Distributed inference and large-model deployment” is live. It goes from the four parallel cuts used in inference to single-node and multi-node vLLM deploy plus side-by-side experiments. Config is based on vLLM v0.32.0, with 10 new hand-drawn-style diagrams:
-
Article: 6.1 Inference parallelism overview: not fitting in memory and not being fast enough are two different problems | Use a memory ledger to separate “does not fit” from “not fast enough”, compare what TP, PP, DP, and EP cut and what they sync, give an order for choosing parallel degrees, and a runnable estimate script
-
Article: 6.2 Tensor-parallel inference: split one layer into pieces, sync twice per layer | Derive per-layer AllReduce volume and latency, analyze KV head replication under GQA, the NVLink vs cross-node bandwidth gap, how TP degree affects latency and throughput, and vLLM’s Custom AllReduce
-
Article: 6.3 Pipeline-parallel inference: cut by layer, fill the pipeline with multiple batches | Clarify what moves between stages and how much, why Decode needs multiple in-flight batches, how uneven layer splits are handled, and compare TPĂ—PP mixes with a cost model
-
Article: 6.4 Data parallel and expert parallel: more replicas, and sending tokens to experts | From multi-replica Dense models to MoE attention DP + expert EP, covering two All-to-Alls, idle sync, load-imbalance simulation and EPLB, DeepEP, and DeepSeek-V3 deploy plans
-
Article: 6.5 Multi-node deploy: spread one service across machines with Ray or multiprocessing | Ray cluster setup and rank layout, multi-process
--nnodesusage, MoE multi-node DP + EP launch, InfiniBand and GPUDirect RDMA config, plus a communication self-check script and troubleshooting table -
Article: 6.6 vLLM distributed deploy in practice: same 8 GPUs, three cuts measured together | On one 8-GPU machine compare Llama-3.1-70B TP8, TP4Ă—PP2, and TP4Ă—DP2: estimate KV capacity before launch, check logs and metrics after, sweep concurrency with
vllm bench serve; also cover using DCP to remove long-context KV replication
Changes
- Article: Chapter 6 Distributed inference and large-model deployment | Chapter intro rewritten to match the actual content, with an information-freshness note and links to each section
2026-11-01
Changes
Module 4 · Inference optimization Chapter 3 “Reading the nano-vllm source” — 6 articles revised: source walkthrough aligned to the nano-vllm main commit from 2026-04-26, 3 new diagrams, all 8 existing Mermaid diagrams redrawn as hand-drawn-style SVG, and some inaccurate wording fixed:
-
Article: 3.1 Project overview and four-layer architecture | Project info table updates line counts, versions and deps, Star count; four-layer architecture and request-lifecycle diagrams become SVG; notes that Prefill and Decode are not mixed in one batch
-
Article: 3.2 Config, entry points, and one full generate path | Adds constraints for
tensor_parallel_size,ignore_eos, and similar params; notes nano-vllm does not support greedy sampling, and that the README example skips a chat template; notesstepreturnsnum_tokenswith sign distinguishing phases -
Article: 3.3 Model skeleton and core ops | Tensor-parallel split diagram becomes SVG; new Sampler section; explains where the 2 all_reduces per layer come from, and how KV head count limits TP degree
-
Article: 3.4 KV Cache and BlockManager | Prefix-cache section split into hash chain, registration, and match; new prefix-hit allocation diagram, match flowchart becomes SVG; notes the last block is not looked up, the difference between hit-in-use and free blocks, and reuse truncation from whole-block granularity
-
Article: 3.5 Scheduler and continuous batching | Sequence-state and scheduling-loop diagrams become SVG; explains the three
breaks in the prefill branch one by one, head-of-line blocking from strict FCFS; notes preemption uses recompute, and whenwhile ... elseruns -
Article: 3.6 ModelRunner, CUDA Graph, and performance benchmarks | New
slot_mappingcompute diagram and CUDA Graph tier diagram;runflowchart and full-chapter call-chain diagram become SVG; notes 6-D KV Cache layout, tier caps vsmax_num_seqs; performance section adds test conditions and where conclusions apply
Module 4 · Inference optimization Chapter 4 “Quantization” — 6 articles revised: 7 new diagrams, 4 existing flat illustrations redrawn in hand-drawn style, existing Mermaid diagrams all become SVG, and some wording fixed against vLLM v0.32.0 source:
-
Article: 4.1 Quantization basics | New four-granularity diagram; redrawn symmetric/asymmetric number line; adds 4-bit symmetric encoding range; PyTorch quantization references switched to torchao docs
-
Article: 4.2 W8A8 and SmoothQuant | Redrawn paired-scale diagram; offline quantization flowchart becomes SVG
-
Article: 4.3 Weight-only INT4 | New W4A16 pack and on-chip dequant diagram, GPTQ/AWQ comparison diagram; Marlin pipeline diagram becomes SVG
-
Article: 4.4 KV Cache quantization | Redrawn KIVI grouping diagram; notes Value residual region is quantized out token-by-token under a sliding window
-
Article: 4.5 FP8 and NVFP4/MXFP4 | New FP8 vs INT8 scale-distribution comparison; redrawn FP4 microblock scale diagram; online FP8 path adds
fp8_per_channel -
Article: 4.6 Quantization selection and vLLM practice | Selection flowchart becomes SVG; fixes online FP8 load path (v0.32.0 builds on meta first and converts layer by layer; it does not load full 16-bit weights at once)
Module 4 · Inference optimization Chapter 5 “Speculative Decoding” revised: 1 new diagram, all 4 existing Mermaid diagrams redrawn as hand-drawn-style SVG, and some wording fixed against vLLM v0.32.0 source:
-
Article: 5.1 Speculative decoding core ideas | Draft + Verify one-round flowchart becomes SVG, with accept/correct/reject positions marked for Îł = 4; new residual-sampling distribution-preserving diagram, using section 7 numbers to show the two paths sum to p
-
Article: 5.2 Draft models and draft-free methods | N-gram lookup flowchart becomes SVG, marking vLLM’s longest-then-earliest match rule; fixes where
ngram_gpuapplies (both V1 and V2 Model Runner); notes default when Suffix has no candidate cap -
Article: 5.3 Medusa, EAGLE, and MTP | Draft Tree diagram becomes SVG, same tree as section 8 experiments, with tree Mask and depth-numbered Position IDs; notes Block acceptance only in V2 Model Runner; fixes MTP conditions that avoid version mismatch
-
Article: 5.4 Benefit boundaries and limits | Scheduling-loop diagram becomes SVG; adds an example of different step lengths for requests in the same batch; fixes Adaptive Verification support range (not only DSpark) and the N-gram/Suffix benefit description under high QPS
-
Article: 5.5 vLLM speculative decoding in practice | Per-position accept metrics use the actual exposed
..._per_pos_totalname and explain the counters; adds--speculative-configshort form-sc
2026-10-29
Changes
Module 4 · Inference optimization Chapter 2 “Core inference-engine techniques” fully revised: 5 articles gain 9 new diagrams, default params and source walkthroughs aligned to vLLM v0.32.0, and some outdated or inaccurate wording fixed:
-
Article: 2.1 PagedAttention: manage the KV Cache the way an OS manages memory | New Block Table mapping diagram and fragmentation comparison;
gpu_memory_utilizationdefault updated to 0.92; notes V1 preemption only recomputes and no longer uses Copy-on-Write; fixesfree_blocksfree-queue enqueue logic -
Article: 2.2 Continuous Batching: dynamic batching so the GPU stops idling | New Static vs Continuous Batching timeline comparison; replaces uncited utilization numbers with idle-slot examples and Orca paper data; preemption pseudocode becomes the retry loop from the source
-
Article: 2.3 Prefix Cache and RadixAttention: reuse shared-prefix KV | New chained block-hash diagram and Radix Tree example; adds
cache_salt, hash seeds, and RadixAttention refcounting, leaf-first LRU, and scheduling by cache hit -
Article: 2.4 Chunked Prefill and unified Token Budget scheduling | New chunked timeline and Token-budget bin-packing diagrams; notes vLLM chunk length is set by remaining budget; adds
long_prefill_token_thresholdand officialmax_num_batched_tokenstuning advice -
Article: 2.5 Attention backends and graph optimization | New Kernel launch vs CUDA Graph comparison and Piecewise CUDA Graph diagram; adds default backends by GPU architecture; fixes CUDA Graph padding rules and V1 default
FULL_AND_PIECEWISEmode
2026-10-26
Changes
Module 4 · Inference optimization Chapter 4 “Quantization” — 6 articles revised, framework content aligned to vLLM v0.32.0, each article gains an information-freshness note, and chapter structure reorganized:
-
Article: 4.1 Quantization basics: Scale, granularity, and PTQ/QAT | Rewrites quantize/dequantize with a “ruler with limited ticks” metaphor; bit-width, matrix axes, and granularity become separate sections; calibration gets its own chapter covering range choice and static vs dynamic calibration
-
Article: 4.2 W8A8 quantization: how SmoothQuant handles activation outliers | Equivalent scaling gets formulas and an intuitive single-dot-product derivation; the smoothing-coefficient section becomes how to choose migration strength α; adds offline flow, benefits and fit conditions, plus a hands-on dot-product example
-
Article: 4.3 Weight-only INT4: GPTQ, AWQ, and Marlin | Adds GPTQ cost and effect, AWQ’s idea of scaling instead of mixed precision; new vLLM kernel selection, quantization-tool choice, and debug order
-
Article: 4.4 KV Cache quantization: KIVI and FP8 cache | New KV footprint estimate; FP8 KV Cache section rewritten for v0.32.0 covering format choice, Scale calibration, skipping sensitive layers, and storage vs compute precision; notes the old
calculate_kv_scalesparam was removed -
Article: 4.5 FP8 and NVFP4/MXFP4: how low-bit floats work | Adds E2M1’s 8 magnitudes, microblock shape notation and Scale’s own format; compares NVFP4 two-level Scale vs MXFP4 one-level Scale; new usage in vLLM v0.32.0
-
Article: 4.6 Quantization selection and vLLM practice | Separates quantization method, checkpoint format, and Kernel as three layers; explains vLLM quantization entry points; experiment baseline updated to v0.32.0
Module 4 · Inference optimization Chapter 5 “Speculative Decoding” — 5 articles revised, config baseline upgraded from vLLM v0.31.0 to v0.32.0, with new MTP content:
-
Article: 5.1 Speculative decoding core ideas: draft a stretch, then let the big model verify | Rewrites one full Draft + Verify round; adds verification when “Draft gives tokens only, not probabilities”
-
Article: 5.2 Draft models and draft-free methods: cheaper ways to guess better | New sections “matching the Target matters more than a high own score” and parallel Draft; unifies definitions for accept rate, mean accept length, and candidate coverage
-
Article: 5.3 Medusa, EAGLE, and MTP: reuse big-model info for better drafts | New section 7 on MTP covering the shift from pretrain objective to inference speedup, real results, shape in vLLM, and exploring one forward for a whole draft block; EAGLE-2 notes confidence is only used to allocate budget
-
Article: 5.4 Benefit boundaries of speculative decoding: high accept rate can still be slower | Adds speedup conditions and diminishing returns of speculation length; new load-adaptive dynamic speculation length; structured output enters the stop-condition discussion
-
Article: 5.5 vLLM speculative decoding in practice: measure config, accept rate, and performance together | Baseline upgraded to vLLM v0.32.0; new fourth EAGLE-3 experiment group and MTP attach conditions; collection script switches to per-request accept metrics, plus common-symptom debug order and next-round tuning directions
2026-10-12
New content
Module 4 · Inference optimization Chapter 3 “Reading the nano-vllm source” is live. Using the ~1200-line minimal inference engine nano-vllm as the spine, it walks module by module through how continuous batching, PagedAttention, FlashAttention, tensor parallelism, and CUDA Graph are organized in real code:
-
Article: 3.1 Project overview and four-layer architecture | See nano-vllm’s role and directory layout; one four-layer architecture diagram covers the full data path from user request to token output
-
Article: 3.2 Config, entry points, and one full generate path | Starting from example.py, break down Config/SamplingParams/LLM/LLMEngine and see how the generate main loop drives add_request, step, and postprocess
-
Article: 3.3 Model skeleton and core ops | Layer-by-layer Qwen3: RMSNorm, RoPE, SwiGLU, parallel Linear and tensor-parallel splits, vocab-parallel Embedding/LMHead, and Attention’s KV write plus the two FlashAttention paths
-
Article: 3.4 KV Cache and BlockManager | Blocked KV management: Block structure, PagedAttention’s block idea, xxhash prefix cache, block alloc/free, and Sequence state machine plus block_table
-
Article: 3.5 Scheduler and continuous batching | Scheduling core: Waiting/Running dual queues, Prefill/Decode split scheduling, Token Budget, chunked prefill, and preemption plus recompute when memory is short
-
Article: 3.6 ModelRunner, CUDA Graph, and performance benchmarks | Sequence tensorization, global Context injection, KV Cache memory splits, CUDA Graph capture and replay, plus official benchmarks vs vLLM
Changes
- The old vLLM quick start moves under Chapter 1 as 1.2 vLLM quick start, as a hands-on entry to inference basics.
2026-10-08
New content
Module 4 · Inference optimization Chapter 4 “Quantization” — 6 articles live:
-
Article: 4.1 Quantization basics: Scale, granularity, and PTQ/QAT | Understand quantize/dequantize from numeric mapping; bit-width, Scale, zero-point, granularity, plus calibration and PTQ vs QAT
-
Article: 4.2 W8A8 quantization: how SmoothQuant handles activation outliers | From Activation Outliers derive SmoothQuant equivalent scaling; smoothing coefficients, INT8 GEMM, and the offline deploy flow
-
Article: 4.3 Weight-only INT4: GPTQ, AWQ, and Marlin | GPTQ error compensation, AWQ activation-aware scaling, and how Marlin turns INT4 storage into inference gains
-
Article: 4.4 KV Cache quantization: KIVI and FP8 cache | Streaming constraints of KV Cache quantization, KIVI’s different Key/Value grouping, and vLLM FP8 KV Cache calibration plus benefit bounds
-
Article: 4.5 FP8 and NVFP4/MXFP4: how low-bit floats work | Integer vs float quantization; FP8 range and precision; NVFP4/MXFP4 microblock scaling and hardware run conditions
-
Article: 4.6 Quantization selection and vLLM practice | Pick quantization candidates by bottleneck; relate vLLM quantization formats to Kernels; run FP16 vs AWQ INT4 throughput and generation-quality comparisons
2026-10-07
New content
Module 4 · Inference optimization Chapter 5 “Speculative Decoding” — 5 articles live:
-
Article: 5.1 Speculative decoding core ideas: draft a stretch, then let the big model verify | From the autoregressive serial bottleneck derive Draft + Verify, accept probability, and the residual distribution; prove the final output follows the Target distribution
-
Article: 5.2 Draft models and draft-free methods: cheaper ways to guess better | Compare an independent small model, N-gram Prompt Lookup, and Suffix Decoding; Tokenizer compatibility, history match, candidate coverage, and accept-rate tradeoffs
-
Article: 5.3 Medusa and EAGLE: reuse big-model info for better drafts | Medusa multi-head prediction, EAGLE feature-level drafts, EAGLE-2 dynamic trees and EAGLE-3 training improvements; Tree Attention and path commit
-
Article: 5.4 Benefit boundaries of speculative decoding: high accept rate can still be slower | Build a performance ledger from mean commit length and full-round time; shared constraints from batching, long context, quantization, scheduling, and streaming latency
-
Article: 5.5 vLLM speculative decoding in practice: measure config, accept rate, and performance together | Against vLLM v0.31.0, start plain Decode, N-gram, and independent Draft in turn; collect accept metrics, quality output, and latency/throughput
2026-09-15
New content
Module 4 · Inference optimization adds Chapter 12 “On-device inference”:
- Article: 12.1 On-device inference basics: from model export to heterogeneous hardware | Build a systematic view of on-device inference hardware, model formats, runtimes, quantization, memory, and Benchmark, and walk a minimal deploy path with ExecuTorch/XNNPACK
2026-07-09
New content
Module 4 · Inference optimization Chapters 1 and 2 articles live:
-
Article: 1.1 LLM inference basics | One article on the underlying logic of LLM inference: Prefill/Decode phases, the KV Cache memory ledger, metrics such as TTFT/TPOT, and using Roofline to explain why Decode is Memory Bound
-
Article: 2.1 PagedAttention | Deep dive into vLLM’s core technique PagedAttention: borrow virtual-memory paging, use a Block Table to remove KV Cache fragmentation, and support prefix sharing plus Copy-on-Write
-
Article: 2.2 Continuous Batching | Deep dive into Continuous Batching: Iteration-level Scheduling lets requests join as they arrive and leave when done, pulling GPU utilization from about 30% to 80%+
-
Article: 2.3 Prefix Cache and RadixAttention | Deep dive into prefix cache: how vLLM’s hash-based automatic prefix cache reuses shared-prefix KV blocks, and how SGLang’s RadixAttention uses a Radix Tree for more efficient prefix sharing
-
Article: 2.4 Chunked Prefill and unified scheduling | Deep dive into Chunked Prefill: chunk long-prompt Prefill so it stops interfering with Decode; how vLLM V1’s unified Token-budget scheduler flattens the Prefill/Decode boundary
-
Article: 2.5 Attention backends and graph optimization | Deep dive into vLLM’s pluggable Attention backends (FlashAttention/FlashInfer/FlashMLA/Triton) and graph opts: CUDA Graph and torch.compile cut Decode-phase CPU launch overhead
Module 3 · Distributed training collective communication, optimizer, and data-parallel chapter articles live:
-
Article: 2.1 Collective communication primitives in depth | AllReduce/ReduceScatter/AllGather/All-to-All input/output semantics, Ring AllReduce algorithm, and how to quantify one communication’s cost
-
Article: 3.1 Optimizers: from SGD to AdamW and memory-cost analysis | Optimizer evolution from SGD to AdamW, optimizer-state memory cost and why FP32 master weights matter, plus LARS/LAMB for large-batch settings
-
Article: 4.1 Data parallelism in depth | Evolution of DP/DDP/FSDP, gradient sync, and math for communication volume / memory ledgers; the real difference between AllReduce and parameter sharding
-
Article: 4.2 PyTorch data parallelism from principle to practice | Full PyTorch data-parallel stack: communication primitives, DDP, FSDP parameter sharding, and multi-node multi-GPU practice with complete code samples
2026-07-07
New content
-
Article: 1.1 Distributed training overview: the memory ledger and five parallel strategies | Hand-compute the training memory ledger; overview of five parallel strategies and how to choose
-
Article: 1.2 Environment setup and distributed launch | rank/local_rank/world_size, torchrun launch and rendezvous, NCCL debugging, minimal DDP script
2026-07-06
New content
Updated the distributed-training tutorial outline:
| Chapter | Main content |
|---|---|
| Chapter 1 Distributed training overview | Why distributed training, training memory ledger, five parallel strategies, environment setup and DDP launch |
| Chapter 2 Collective communication primitives | AllReduce/ReduceScatter/AllGather/All-to-All semantics, Ring algorithm, communication-volume analysis |
| Chapter 3 Optimizers | SGD/Adam/AdamW evolution, optimizer-state memory cost, large-batch optimizers LAMB/LARS |
| Chapter 4 Data parallelism | DataParallel, DistributedDataParallel, FSDP |
| Chapter 5 ZeRO series | ZeRO-1/2/3 sharding and communication cost, ZeRO-Offload/Infinity heterogeneous memory offload |
| Chapter 6 Tensor parallel and sequence parallel | Megatron Column/Row Parallel Linear, where communication is inserted, sequence-parallel activation memory opts |
| Chapter 7 Pipeline parallel | Bubble problem, GPipe, 1F1B, Interleaved schedule |
| Chapter 8 Mixed precision and memory opts | FP16/BF16/FP8 mixed precision, gradient accumulation, Activation Checkpointing |
| Chapter 9 Long-sequence training and context parallel | Attention’s O(s²) problem, Ring Attention, DeepSpeed-Ulysses, Megatron Context Parallel |
| Chapter 10 MoE parallel | Router mechanism, Expert Parallelism All-to-All, EPĂ—DPĂ—TPĂ—PP mixes and load balance |
| Chapter 11 3D parallel and hybrid strategies | TPĂ—PPĂ—DPĂ—EPĂ—CP topology design, communication-domain mapping, rank layout, cluster selection |
2026-06-25
New content
-
Article: 6.1 FlashAttention V1 in depth | Deep dive into FlashAttention V1 principles and implementation: how Tiling + Online Softmax cuts Attention’s extra memory from to , sharply reduces HBM traffic, and speeds up without accuracy loss
-
Article: 6.2 FlashAttention V2 in depth | This this article deep-dives FlashAttention V2’s core changes vs V1: swap inner/outer loop order, improve work distribution inside the thread block, and cut non-matmul ops
2026-06-06
New content
- Article: 5.2 CUDA Online Softmax implementation and optimization | From algorithm derivation to implementing and optimizing an Online Softmax CUDA Kernel — core prerequisite knowledge for understanding FlashAttention
2026-06-05
New content
- Article: 5.1 CUDA Softmax naive implementation and optimization | From a naive implementation, gradually introduce Safe Softmax, block-level parallelism, Warp Shuffle, vectorized loads, and related opts
2026-06-02
New content
- Article: 4.1 CUDA GEMM operator performance optimization | From a naive implementation through Block-Warp-Thread three-level tiling, vectorized loads, Bank Conflict removal, double buffering, and more — a full methodology for CUDA GEMM optimization
2026-05-18
New content
- Article: 3.1 CUDA Reduce operator optimization | Reduce is one of the most basic GPU ops and one of the best for learning parallel thinking. From the simplest implementation, this article gradually adds Warp-level primitives, vectorized loads, multi-element processing, and more — each step with performance comparisons and explanations so you really understand how to write a fast Kernel.
2026-05-16
New content
- Module 3 · Distributed training adds Chapter 2: Optimizers | Follow SGD → Momentum → Adam → AdamW evolution and internal state makeup; hand-compute optimizer memory cost (AdamW is 75% of training memory); know when large-batch optimizers LAMB/LARS fit; foundation for later ZeRO memory opts and mixed-precision training.
Structure changes
- Module 3 · Distributed training chapters renumbered: old Chapters 2–7 become 3–8; new Chapter 2 “Optimizers” sits between the overview and data parallelism
- Synced the AI Infra learning path and the Module 3 chapter list in the README
2026-05-13
New content
- Article: 2.4 Synchronization and atomic operations | Correct sync is the base of bug-free parallel programs. This article covers CUDA block sync
__syncthreads(), Warp-level sync, Memory Fence, and when to use atomics, their cost, and how to optimize — so you can write high-performance parallel code without giving up correctness.
2026-05-12
New content
-
Article: 2.2 Memory-access optimization | Memory-access efficiency is the biggest lever on CUDA Kernel performance. Coalesced Access principles and checks, shared-memory Bank Conflict causes and Padding fixes, and vectorized loads (float4/int4) to raise bandwidth use.
-
Article: 2.3 Occupancy and resource allocation | Occupancy measures active Warps on an SM vs the theoretical max — a core CUDA Kernel tuning metric. Definition, how to compute it, three limiters (registers/shared memory/Block size), and why higher Occupancy is not always better — the real goal is balance between latency hiding and resource use.
2026-05-11
New content
- Article: 2.1 Warp and the execution model | Deep dive into the GPU’s core execution unit — the Warp
2026-05-09
New content
-
Article: CUDA memory model | Global memory, shared memory, registers, constant memory, and unified memory — traits, fit cases, and optimization tips
-
Article: First useful Kernel: vector add | A full vector-add case that walks the whole CUDA programming path
2026-05-08
New content
- Article: 1.1 CUDA development environment setup
- Article: 1.2 CUDA programming model
2026-05-07
New content
2026-04-28
New content
- Article: 🔥 Cluster communication networks and NCCL: the communication backbone of distributed training
2026-04-27
New content
2026-04-26
New content
2026-04-25
New content
- Article: 🔥 PyTorch framework quick start
2026-04-24
New content
- Article: 3.9 Tokenization and word embeddings
2026-04-23
New content
2026-04-21
New content
- Article: 3.6 LayerNorm and residual connections in depth
- Article: 3.7 Transformer Decoder Block full breakdown
2026-04-20
New content
- Article: 3.4 Transformer feed-forward network (FFN) in depth
- Article: 3.5 Transformer positional encoding in depth
2026-04-18
New content
2026-04-17
New content
- Interview section: 180+ real interview sessions covering 40+ companies, organized by tier. Browse online →
- Companies include ByteDance, Alibaba, Tencent, Baidu, Kuaishou, Meituan, Ant Group, NVIDIA, MiniMax, NIO, XPeng, Li Auto, and more.
- Community entry: new WeChat discussion group, welcome to join
Content improvements
- Updated the README and added a project intro image
2026-04-16
Major update
- Knowledge-base framework overhaul: finer four learning modules, page layout adjusted
- Module 1 · Prerequisites (6 chapters): Programming language basics, Math basics, Transformer architecture in depth, PyTorch framework, GPU hardware overview, Collective communication basics
- Module 2 · CUDA programming and operator optimization (8 chapters): CUDA intro, Performance optimization basics, Reduce practice, GEMM practice, Softmax practice, Attention operators, AI compilers, Performance analysis toolchain
- Module 3 · Distributed training (7 chapters): Distributed training overview, Data parallelism, ZeRO series, Tensor parallel and sequence parallel, Pipeline parallel, 3D parallel and hybrid training strategies, Training framework practice
- Module 4 · Inference optimization (8 chapters): Inference basics, Core inference-engine techniques, Inference frameworks, Quantization, Speculative Decoding, PD disaggregation, Performance analysis and Benchmark, Selection and end-to-end practice
New content
- Transformer series:
- Course material: 150+ detailed sub-articles live, covering all chapters of the four modules
- Blog: new About the author page
2026-04-15
Project launch
- AIInfraGuide knowledge base officially live
- Published [AI Infra learning path](/AIInfraGuide/guides/AI Infraĺ¦äą č·Żçşż) overview
- Published Understanding AI Infra from scratch starter guide
2026-04-14
Project init
- Website framework built (Astro + Starlight)
- Repo initialized, first commit published