6.4 Data parallelism and expert parallelism

TP and PP both cut one replica. When throughput is short and one replica already meets latency, the simplest move is to run more replicas—that is DP. For Dense models there is almost nothing to say: replicas do not talk; the hard part is load balance.

MoE is different. Of DeepSeek-V3’s 671B parameters, about 97% are routed experts, and each token uses only 8 of them. Experts belong whole on different cards (EP); attention fits better as one copy per card, each serving its own requests (DP). Put together, that is today’s mainstream layout for large MoE inference. This section covers DP first, then EP communication, sync, and load balance, and finally DeepSeek-V3’s real deployment.

⏰ Currency: vLLM descriptions follow v0.32.0 (released 2026-10-18) docs and source; DeepEP numbers are from its v1.2.1 README; DeepSeek-V3 deployment is from its technical report (arXiv 2412.19437).

📑 Contents

• 1. Dense-model DP: open more shops
• 2. Why MoE cuts by expert
• 3. Dispatch and Combine: two All-to-Alls
• 4. Attention DP + expert EP
• 5. Uneven load: the busiest expert sets the pace
• 6. Comm kernels: DeepEP and vLLM backends
• 7. Case: DeepSeek-V3 inference deployment
• Summary
• Self-check list
• References

---

1. Dense-model DP: open more shops

1.1 Replicas do not communicate

DP is the “open more shops” metaphor from section 6.1: each shop has a full kitchen set and takes its own guests. Each replica holds a full model (or a full TP/PP group), schedules on its own, and manages its own KV Cache.

In a Dense forward there is no sync across replicas; throughput grows roughly linearly with replica count. It does not save memory; the premise is that one replica already fits.

1.2 The hard part is how requests are split

Replicas do not talk, so how requests are split decides the outcome:

• Load: one replica piled with long prompts sees worse TTFT while others sit idle. Assigning by “running + queued” request count beats round-robin.
• Prefix cache: each replica’s KV Cache is independent. If requests that share a System Prompt scatter across replicas, every replica recomputes the prefix. Routing shared-prefix requests to the same replica raises the prefix-cache hit rate from Chapter 2 section 2.3.

1.3 Two approaches in vLLM

For Dense models the direct approach is several independent vllm serve processes behind a load balancer. vLLM’s docs say the same for Dense: external load-balance mode only supports MoE; Dense should start independent instances without --data-parallel-*.

Or let vLLM manage it:

# One 8-GPU machine: 4 replicas, TP=2 each, one external port
vllm serve $MODEL --data-parallel-size 4 --tensor-parallel-size 2

There is then one HTTP port. API Server process count defaults to DP size (here 4); they assign requests from each replica’s running and waiting queues. Note --max-num-seqs is a per DP rank cap, while --max-num-queued-reqs is service-wide; convert by replica count when you set them.

---

2. Why MoE cuts by expert

2.1 Almost all parameters sit in experts

Take DeepSeek-V3 ($h=7168$, 61 layers, first 3 Dense and last 58 MoE, 256 routed experts plus 1 shared expert per layer, expert intermediate 2048). One expert is three matrices gate, up, down:

$$3\times7168\times2048\approx44\text{M parameters}$$

58 layers × 256 routed experts ≈ 654B, about 97% of 671B total. Each token activates only 8 per layer, so activated parameters are 37B.

So MoE inference memory is almost entirely a pile of experts that are idle most of the time. How you place those experts is the main deploy question.

2.2 Problems with TP-cutting experts

The direct move is to TP-cut every expert like a Dense FFN. Two problems:

• Matrices get too thin: expert intermediate is already only 2048; at TP=8 each card keeps 256 columns. GEMMs that small rarely fill Tensor Cores.
• Every card touches every token: every expert lives across all cards, so no matter which expert a token picks, every card joins the compute and AllReduce.

2.3 EP: whole experts on one card each

EP flips the idea: do not cut experts; place whole ones on different cards. At EP=8 each card holds 32 full experts; at EP=32 each holds 8.

In FP8 weights, DeepSeek-V3 routed experts at EP=8 are about 82 GB per card—H100 80GB does not fit—so vLLM’s single-node EP examples use H200 or H20; at EP=32 that drops to about 20 GB per card.

The cost is that tokens must “travel”: each token goes to the card that owns its chosen experts, then comes back. That is the “stations by cuisine” metaphor from section 6.1.

---

3. Dispatch and Combine: two All-to-Alls

3.1 Dataflow inside one MoE layer

Each MoE layer has four steps:

1. Route: each card runs the Router on its tokens and gets top-$k$ experts and weights per token.
2. Dispatch: by the route, copy each token’s hidden state $k$ times and send to the cards that own those experts. Every card both sends and receives—one All-to-All.
3. Expert compute: each card runs its experts on tokens it received.
4. Combine: send results back to the token’s home card and weighted-sum by route weights—another All-to-All.

The figure is EP=4, 8 experts, top-2 per token; color marks which card a token came from. Left is each card’s own tokens after attention; right is tokens each card received after Dispatch:

GPU0 got 12 copies, GPU3 only 4. Combine waits for every card; this layer’s time is set by GPU0. Section 5 covers that in depth.

3.2 Communication volume

Round-trip volume per token per MoE layer is about $2\times k\times h\times b$. For DeepSeek-V3, $k=8$, $h=7168$:

Dispatching in FP8 is DeepSeek-V3 and DeepEP’s practice; it saves about a quarter versus BF16 end-to-end. Tokens whose experts already sit on the home card never leave the node.

Versus TP, EP communication only happens on MoE layers, twice per layer; but volume scales with $k$ and the traffic is irregular many-to-many, so cross-node pressure is much higher than PP.

---

4. Attention DP + expert EP

4.1 Why attention uses DP

Once experts are EP-cut, attention still needs a layout. For MLA models like DeepSeek-V3, TP is a bad deal: section 6.2 said MLA caches one compressed latent; under TP every card stores the full latent KV, so KV memory wastes linearly with TP.

DP sidesteps that: each card holds a full attention weight copy, only serves its own request set, and only stores those requests’ KV. Attention weights are a small share of an MoE model; a few copies are cheap.

vLLM docs recommend the same: for MoE, especially DeepSeek with MLA, use DP on attention and EP or TP on experts.

4.2 EP size is TP × DP

vLLM has no separate EP-size flag. With --enable-expert-parallel:

$$\text{EP\_SIZE}=\text{TP\_SIZE}\times\text{DP\_SIZE}$$

Without --enable-expert-parallel, MoE layers default to one TP group of size TP × DP; every expert is cut, and you are back to the problems in 2.2.

4.3 DP ranks are no longer independent: idle ranks still run

Dense DP replicas ignore each other; MoE DP ranks must stay in step. All-to-All needs every rank; if one card has no requests this step, tokens other cards send it have nowhere to land and the group stalls.

So vLLM requires: as long as any rank still has running requests, ranks with none must run a dummy forward and walk every MoE layer’s communication with the others. In the figure GPU3 has no requests for now but still joins Dispatch and Combine:

A separate DP Coordinator process handles this. It talks to every rank and every few steps runs a collective to see whether all ranks are idle; only then do they pause together. In source that interval is dp_sync_interval, default 16 steps.

📌 Key point: Two consequences. First, requests across DP ranks should be as even as possible, or low-request cards mostly dummy-run. Second, every group step waits on the slowest rank; a long Prefill on one rank slows Decode on all ranks.

4.4 Launch command

Single-node 8 GPUs (vLLM doc example; needs larger-memory cards like H200 or H20):

vllm serve deepseek-ai/DeepSeek-V3-0324 \
    --tensor-parallel-size 1 \
    --data-parallel-size 8 \
    --enable-expert-parallel

Each DP rank is its own EngineCore process with its own scheduler and KV Cache, talking to the front API Server over ZMQ. Multi-node launch is in section 6.5.

---

5. Uneven load: the busiest expert sets the pace

5.1 Why load is uneven

Training usually adds a load-balance aux loss, or like DeepSeek-V3 a bias term without aux loss, so expert token counts stay roughly even on the training distribution. Inference traffic over a window may concentrate in one domain (e.g. all code); hot experts then get far more than average.

Under EP every layer waits for every card’s expert compute before Combine, so layer time is set by the card that received the most tokens. The metric is “busiest card ÷ average”; 1 means fully balanced.

5.2 Hands-on sim: larger EP, worse imbalance

The script assumes expert popularity follows a power law (exponent 0.4 is assumed, not any real model), DeepSeek-V3-style 256-choose-8 routing on 4096 tokens in one step, and compares three placements:

# Simulate MoE load imbalance: expert popularity ~ power law (assumed),
# compare how much the busiest card exceeds the average under three placements.
# Count tokens only; ignore communication and compute efficiency.
import random

random.seed(0)
EXPERTS, TOP_K, TOKENS = 256, 8, 4096      # DeepSeek-V3-style route: 256 choose 8; 4096 tokens / step
SKEW = 0.4                                  # power-law exponent; 0 is fully even; assumed

weights = [1 / (i + 1) ** SKEW for i in range(EXPERTS)]
random.shuffle(weights)                     # scatter hot experts across IDs
counts = [0] * EXPERTS
for _ in range(TOKENS):
    chosen = set()
    while len(chosen) < TOP_K:              # each token picks 8 distinct experts
        chosen.add(random.choices(range(EXPERTS), weights=weights)[0])
    for e in chosen:
        counts[e] += 1


def contiguous(ep):
    """Default placement: experts assigned to cards in contiguous ID ranges"""
    per = EXPERTS // ep
    load = [sum(counts[g * per:(g + 1) * per]) for g in range(ep)]
    return max(load) / (sum(load) / ep)


def balanced(ep, redundant):
    """Reorder by observed load: give redundant slots to experts with highest
    load-per-replica, then place replicas heaviest-first onto the currently
    lightest card that still has a free slot"""
    replicas = [1] * EXPERTS
    for _ in range(redundant):
        e = max(range(EXPERTS), key=lambda i: counts[i] / replicas[i])
        replicas[e] += 1
    copies = sorted((counts[e] / replicas[e] for e in range(EXPERTS) for _ in range(replicas[e])), reverse=True)
    slots = (EXPERTS + redundant) // ep
    load, used = [0.0] * ep, [0] * ep
    for c in copies:
        g = min((g for g in range(ep) if used[g] < slots), key=lambda g: load[g])
        load[g] += c
        used[g] += 1
    return max(load) / (sum(load) / ep)


avg = TOKENS * TOP_K / EXPERTS
print(f"hottest expert {max(counts)} tokens, coldest {min(counts)}, average {avg:.0f}")
print(f"{'EP':>4} {'experts/GPU':>6} {'contiguous':>7} {'rebalance':>7} {'+1 redun/GPU':>8}")
for ep in [8, 32, 64, 256]:
    print(f"{ep:>4} {EXPERTS // ep:>8} {contiguous(ep):>10.2f}x {balanced(ep, 0):>11.2f}x {balanced(ep, ep):>11.2f}x")

Output:

hottest expert 640 tokens, coldest 67, average 128
  EP   experts/GPU    contiguous   rebalance   +1 redun/GPU
   8       32       1.14x        1.01x        1.00x
  32        8       1.41x        1.17x        1.00x
  64        4       1.93x        1.64x        1.00x
 256        1       5.00x        5.00x        1.01x

Under these assumptions:

• Larger EP, worse imbalance. At EP=8 each card has 32 experts; hot and cold cancel; the busiest card is only 14% above average. At EP=256 each card has 1 expert; the hottest expert’s 640 tokens all land on one card—5× average.
• Reorder alone without replicas does nothing at large EP. With one expert per card there is nowhere to move; before and after reorder both are 5.00×.
• One redundant expert per card nearly fully balances. Hot experts get multiple copies; tokens split across copies; the busiest card falls within 1.01× of average.

⚠️ Note: This sim reorders from the same step’s stats—as if you already knew this step’s routes. Ideal case. Real systems only adjust from recent history; traffic shifts lag.

5.3 EPLB: redundant experts plus periodic rebalance

vLLM’s Expert Parallel Load Balancer (EPLB) is the “redundant + rebalance” idea above: every forward records per-expert load, periodically reassigns experts to cards from those stats, and hot experts can get extra copies.

vllm serve deepseek-ai/DeepSeek-V3-0324 \
    --tensor-parallel-size 1 \
    --data-parallel-size 8 \
    --enable-expert-parallel \
    --enable-eplb \
    --eplb-config '{"window_size":1000,"step_interval":3000,"num_redundant_experts":8,"log_balancedness":true}'

A few points:

• --enable-eplb must pair with --enable-expert-parallel, or startup fails.
• With redundancy, experts per card are $(\text{total experts}+\text{redundant})/\text{EP}$. The command above is $(256+8)/8=33$.
• Redundant experts cost memory. Per vLLM’s estimate, one extra expert per DeepSeek-V3 card needs about 2.4 GB, matching 2.1’s math: 58 layers × 44M params ≈ 2.55 GB in FP8 (2.4 GiB). Weigh that when KV space is tight.
• For large multi-node deploys, vLLM docs suggest num_redundant_experts 32 so hot experts always have a spare.

💡 Tip: Turn on log_balancedness for a while first, then decide whether and how much redundancy. If balancedness is already near 1, redundant experts only waste memory.

---

6. Comm kernels: DeepEP and vLLM backends

6.1 DeepEP’s two kernel families

DeepSeek’s open-source DeepEP is a comm library written for MoE Dispatch/Combine. It offers two kernel families for Prefill vs Decode:

• Normal Kernel: throughput first. Cross-node traffic goes RDMA to the target node, then NVLink inside the node, so inter-node traffic is merged as much as possible. Fits Prefill’s large messages of thousands of tokens.
• Low-latency Kernel: latency first. Pure RDMA, CUDA Graph friendly; compute/comm overlap via hooks without taking SMs. Fits Decode’s small messages of tens to hundreds of tokens per step.

DeepEP v1.2.1 README numbers (H800, one 400 Gb/s CX7 NIC per GPU):

Setup is hidden 7168, top-8, Dispatch FP8, Combine BF16. Low-latency latency from EP=64 to EP=256 only grows about 20 µs (Dispatch) and 50 µs (Combine)—the premise for large EP in Decode.

6.2 vLLM All-to-All backends

vLLM picks the EP comm implementation with --all2all-backend. Common options in the docs:

DeepEP backends need DeepEP and DeepGEMM installed first. Docs also note DeepEP’s two families are designed for PD separation; mixed Prefill+Decode may perform poorly. In PD-separated deploy, Prefill instances use deepep_high_throughput and Decode instances use deepep_low_latency (Chapter 7).

A related flag is --enable-dbo (Dual Batch Overlap): split one batch into two so one does All-to-All while the other computes, hiding comm behind compute—same idea as the dual micro-batch overlap in the DeepSeek-V3 case below.

⚠️ Note: Older pplx and naive backends are gone in v0.32.0; setting them warns and falls back to allgather_reducescatter. Old tutorials online still mention them.

---

7. Case: DeepSeek-V3 inference deployment

DeepSeek-V3’s technical report publishes its online inference layout—a full combination of this section’s pieces. Prefill and Decode are deployed separately (PD separation) with different parallelism:

Points that map back:

• Small EP for Prefill, large EP for Decode. Prefill handles many tokens at once; every expert has enough work; EP32 is enough. Decode has few tokens per step; EP320 gives each card one expert, so each card reads the least expert weight and cuts per-step latency. Matches DeepEP Low-latency numbers above: raising EP from 64 to 256 barely grows Dispatch latency.
• Redundant experts come from load stats. Prefill picks high-load experts for redundancy from online stats and retunes about every 10 minutes. That is what EPLB in 5.3 does.
• Shared experts treated as routed. In Decode the shared expert is a must-pick routed expert, so each token actually picks 9 experts.
• Attention is TP4 + DP, not pure DP. The report only says a small TP like 4 limits TP comm cost; it does not explain why not pure DP. A reasonable reading: TP4 groups stay small enough for in-node, while cutting per-card attention weights and compute.

⚠️ Note: This is DeepSeek’s own inference stack on H800, not vLLM defaults. With vLLM you can combine --data-parallel-size, --enable-expert-parallel, --all2all-backend, and EPLB along the same lines, but concrete numbers need your GPU count and measured load.

---

📝 Summary

• DP copies whole replicas. Dense replicas have zero communication; the hard part is request assignment. --max-num-seqs is per DP rank.
• Almost all MoE parameters sit in routed experts (≈97% for DeepSeek-V3). TP-cutting experts shreds matrices; EP places each expert whole on one card.
• EP does two All-to-Alls per MoE layer: Dispatch sends tokens to expert cards, Combine returns and weighted-sums; round-trip volume per token per layer ≈ $2kh b$.
• Common MoE layout is attention DP + expert EP; in vLLM EP size = TP × DP. All DP ranks must stay in step; idle ranks still dummy-run.
• Layer time is set by the card that got the most tokens; larger EP worsens imbalance. EPLB uses redundant experts and periodic rebalance at a memory cost.
• DeepEP Normal Kernel fits Prefill; Low-latency Kernel fits Decode—matching vLLM’s deepep_high_throughput and deepep_low_latency.

🎯 Self-check list

• Can state the difference between Dense DP and MoE DP, and why MoE needs dummy runs
• Can compute one DeepSeek-V3 expert’s parameter count, routed-expert share of total, and per-card expert weight at a given EP
• Can write Dispatch and Combine volume per token per layer, and the benefit of FP8 Dispatch
• Can explain why MLA attention prefers DP over TP
• Can compute vLLM EP size from TP and DP, and experts per card after redundancy
• Can explain why larger EP worsens load imbalance, and why redundant experts beat reorder alone
• Can pick vLLM --all2all-backend separately for multi-node Prefill and Decode

📚 References

• GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• DeepSeek-V3 Technical Report
• DeepEP (v1.2.1)
• vLLM docs: Data Parallel Deployment
• vLLM docs: Expert Parallel Deployment
• vLLM source: vllm/config/parallel.py (v0.32.0)