Skip to main content
🌐 Distributed training 17 articles · 11 chapters

Module 3: Distributed training

Starts with an overview, then covers data parallelism, the ZeRO family, tensor parallelism, pipeline parallelism and 3D parallelism, and ties them together with a training framework.

Start learning

Chapters

1 Chapter 1: Distributed training overview 2 articles
From model-scale bottlenecks, hand-calculate the training memory ledger, survey the five parallelism strategies, then land on rank/torchrun/NCCL setup and your first DDP script
2 Chapter 2: Collective communication primitives 1 articles
The physical foundation of distributed training—semantics of AllReduce/ReduceScatter/AllGather/All-to-All and other collectives, how the Ring algorithm works, and quantitative communication-volume analysis
3 Chapter 3: Optimizers 1 articles
How mainstream optimizers evolved and what state they keep inside, how to analyze optimizer memory cost, and the foundation for ZeRO memory optimization and mixed-precision training
4 Chapter 4: Data parallelism (DP, DDP, FSDP) 2 articles
How three generations of data parallelism evolved, how they work in practice, and what really differs between AllReduce gradient sync and parameter sharding
5 Chapter 5: The ZeRO memory-optimization series
How ZeRO-1/2/3 shard state and what they cost in communication, plus how ZeRO-Offload and Infinity move state onto heterogeneous memory
6 Chapter 6: Tensor parallelism (TP) and sequence parallelism (SP)
Megatron-LM's tensor-parallel design (Column/Row Parallel Linear), where communication is inserted, and how sequence parallelism cuts activation memory
7 Chapter 7: Pipeline parallelism (PP)
What causes the pipeline bubble, and how GPipe, 1F1B, and Interleaved scheduling reduce it—with quantitative analysis
8 Chapter 8: Mixed precision and memory optimization
FP16/BF16/FP8 mixed-precision training, gradient accumulation, Activation Checkpointing, and what each costs in compute and memory
9 Chapter 9: Long-sequence training and context parallelism
Breaking the sequence-length wall—Attention's O(s²) problem, and a comparison of Ring Attention, DeepSpeed-Ulysses, and Megatron Context Parallel
10 Chapter 10: MoE parallelism
How sparse expert models parallelize: the Router, All-to-All in Expert Parallelism, combining EP×DP×TP×PP, and load balancing
11 Chapter 11: 3D parallelism and hybrid parallelism strategies
Putting every parallelism dimension together—topology design for TP×PP×DP×EP×CP, communication-domain mapping, rank layout, and how to pick the best mix for a given cluster

Study tips

Prerequisites

The prerequisites module, especially chapter 6 on collective communication

  • Read the overview first. It gives you the big picture.
  • Spend extra time on ZeRO. Modern distributed training is hard to follow without it.
  • The training framework chapter connects the earlier theory.