Skip to main content
⚡ CUDA programming and kernel optimization 24 articles · 8 chapters

Module 2: CUDA programming and kernel optimization

Starts with CUDA basics, works through classic kernels (Reduce, GEMM, Softmax, Attention), then covers AI compilers and profiling tools.

Start learning

Chapters

1 Chapter 1: CUDA programming intro 4 articles
Set up a CUDA development environment, learn the Grid/Block/Thread programming model and the memory model, and write your first practical CUDA kernel
2 Chapter 2: CUDA performance optimization basics 4 articles
Warp execution, memory-access tuning, Occupancy, and synchronization—the core playbook for making CUDA kernels fast
3 Chapter 3: Classic operator implementations — Reduce 1 articles
Three progressive versions of Reduce (naive, shared-memory tree reduce, Warp Shuffle) as a way to learn step-by-step CUDA operator optimization
4 Chapter 4: Classic operator implementations — GEMM 1 articles
From naive matrix multiply through Block-Warp-Thread tiling, vectorized loads, bank-conflict removal, and double buffering — a full playbook for CUDA GEMM optimization
5 Chapter 5: Classic operator implementations — Softmax and operator fusion 2 articles
Numerically stable Softmax and Online Softmax, plus the ideas and practice of operator fusion
6 Chapter 6: The Attention operator 2 articles
How FlashAttention V1/V2/V3 work and how they are implemented, plus Flash-Decoding and the PagedAttention CUDA path for the decode stage
7 Chapter 7: AI compilers 1 articles
Triton's Block-level programming model, torch.compile's compilation modes, and how TVM and XLA differ in role
8 Chapter 8: Performance profiling toolchain
Nsight Systems, Nsight Compute, and PyTorch Profiler—the three tools for systematic performance diagnosis

More resources

Practice Hand-written CUDA kernels · interview practice

Graded exercises for memory access, Reduce, Softmax, Attention, GEMM and more, with hints, self-check lists and an in-browser code editor
Practice LeetGPU

LeetCode for GPU programming. Kernel problems graded easy, medium and hard, solvable in CUDA, Triton, PyTorch and more. You can submit without a local GPU and inspect PTX/SASS and the performance leaderboard.

Study tips

Prerequisites

The prerequisites module, especially chapter 5 on GPU hardware

  • Finish the GPU hardware chapter in the prerequisites module first.
  • Every kernel chapter comes with code. Run it and change it yourself.
  • The profiling tools come up throughout the module, so learn them early.