📚 AI Infra prerequisites 20 articles · 6 chapters
Module 1: Prerequisites
GPU architecture, programming basics, the math you need, the Transformer architecture, PyTorch and collective communication. Everything later in the guide builds on this.
Start learningChapters
The Python advanced topics, C/C++ core ideas, and Linux development environment you need for AI Infra, so later CUDA programming and systems work have a solid base
Build the linear algebra, probability, and calculus intuition AI Infra needs: the math behind GEMM tiling, Softmax, backpropagation, and mixed-precision training
Every Transformer component in depth: Self-Attention, FFN, positional encoding, normalization, and the architecture path from MHA to GQA/MLA and from FFN to MoE
- 🔥 Transformer architecture: a quick start
- 3.1 Why AI Infra engineers must understand Transformer
- 3.2 Transformer overview and code implementation
- 3.3 A deep dive into Self-Attention
- 3.4 A deep dive into the Transformer feed-forward network (FFN)
- 3.5 A deeper look at Transformer positional encoding
- 3.6 A deeper look at LayerNorm and residual connections
- 3.7 Full breakdown of the Transformer Decoder Block
- 3.8 From Transformer to LLM autoregressive generation
- 3.9 Tokenization and word embeddings
Master PyTorch Tensors, autograd, the Module training loop and debugging tools, then build a GPT-2-scale small model from scratch and train it end to end
GPU architecture design philosophy, the memory hierarchy, mainstream GPU specs and interconnect topology, so you have the hardware picture before writing CUDA
The communication primitives of distributed training (AllReduce, AllGather and others), the Ring and Tree algorithms, and how to use and tune NCCL
Study tips
- Read the chapters in order. Each one stands on its own, but they build on each other.
- If you already know a topic, skip that chapter and go straight to the CUDA module.
- Every later module assumes you know the Transformer architecture and PyTorch well.