2.4 Chunked Prefill and unified token-budget scheduling

The Continuous Batching section left a loose end: when a very long prompt’s Prefill squeezes into a compute step, it slows every Decode request in the same batch, and users feel a clear “hitch.” Chunked Prefill cleans that up—it cuts a long Prefill into small pieces and slips them into the gaps between Decodes so neither side stalls the other. vLLM V1 goes further: one unified token-budget scheduler erases the Prefill/Decode line entirely. This section covers both.

⏰ Currency note: the chunking idea follows the SARATHI paper; vLLM’s scheduling logic, parameters, and tuning advice are based on v0.32.0 (released 2026-10-18).

📑 Contents

• 1. How Prefill stalls Decode
• 2. Chunked Prefill: split a long Prefill into chunks
• 3. Token Budget: one currency for scheduling
• 4. vLLM V1’s unified scheduler
• 5. Source walk-through: how Token Budget cuts a Chunk
• 6. Key parameters and tuning
• 7. Tradeoffs and when it helps
• Summary
• Self-check list
• References

---

1. How Prefill stalls Decode

First, the interference mechanism. Two takeaways from the last chapter:

• Prefill is compute bound: one step parallel-processes every token in the prompt; the longer the prompt, the heavier that step’s matmuls and the longer it takes.
• Decode is memory bound: each step handles only 1 token, so it is fast by itself; the token rate users feel (TPOT) depends on those steps advancing steadily.

The problem: Continuous Batching rebuilds the batch every step. If a step packs Prefill of a 4000-token prompt together with a batch of Decode requests, that step’s wall time is stretched by the huge Prefill—every Decode request in the batch has to wait for Prefill to finish before that token can come out together.

Analogy: Decode requests are small packages moving at a steady pace on a conveyor, one every few seconds; suddenly a giant shipping container (long Prefill) also needs to pass inspection on the same line, the whole line jams, and every small package behind it waits. On the user side that looks like: tokens flow smoothly, then hitch once, then flow again—a TPOT spike (P99 jitter).

📌 Key point: this is not “batch or not”; it is “one step packed too much compute-bound work.” If the long Prefill is the culprit, splitting it into small pieces across several steps should stop it from locking the line at once.

---

2. Chunked Prefill: split a long Prefill into chunks

Chunked Prefill is straightforward: a long prompt does not have to Prefill in one step; cut it into Chunks and spread them across multiple steps. The SARATHI paper cuts equal-length chunks; vLLM does not fix a chunk length up front—each chunk’s size is whatever token budget remains in that step (see §5).

For example, a 4000-token prompt cut at 512 tokens per chunk is 7 full chunks plus a 416-token tail, 8 chunks total. Each step handles only one chunk; the remaining compute keeps Decode requests in the same batch producing tokens normally. So:

• Each step’s compute stays in a predictable range and is not suddenly blown up by a huge Prefill.
• Decode requests can keep advancing steadily while the long Prefill is still running, and TPOT spikes are smoothed out.

The figure below puts the §1 scene and the chunked scene on one timeline. Each cell is a step: blue is 8 Decode requests each emitting 1 token; yellow is Prefill of the long prompt; black outlines mark steps that include Prefill; the orange dashed line is when the long prompt’s first token appears. Step widths use a simplified “fixed overhead + proportional to Prefill token count” model, only to show relative length.

Back to the conveyor: now that giant container is broken into standard crates; each inspection pass admits only one crate, with small packages still slipping through in between. The container finishes slowly, but the line keeps moving at a steady pace and nothing is fully blocked.

🔑 Core idea: Chunked Prefill trades “spreading a large Prefill across many steps” for “steady Decode”—it flattens one compute spike into several controllable small ones and stabilizes TPOT tail latency.

⚠️ Note: chunking makes this long prompt’s TTFT a bit larger (Prefill spans multiple steps, so the first token arrives a little later). That is a deliberate tradeoff—give up a little per-request TTFT for stable Decode TPOT across the batch. For services that care about smooth token streaming, that trade usually pays off.

---

3. Token Budget: one currency for scheduling

To land Chunked Prefill, the scheduler needs one yardstick for how much work goes in each step. That yardstick is Token Budget.

The insight: whether Prefill or Decode, to the GPU it is still “forward compute over some number of tokens.”

• A Decode request contributes 1 token this step (the next one to generate).
• A Prefill request (or one of its Chunks) contributes N tokens this step (that chunk’s length).

So the scheduler sets a total token-budget cap per step (e.g. “at most 2048 tokens this step”), then packs requests: first keep Decode requests that are already generating (1 token of budget each), then give leftover budget to Prefill chunks. When the budget is spent, the step ships.

$$\underbrace{\sum \text{Decode requests}}_{\text{1 token each}} + \underbrace{\sum \text{Prefill Chunk}}_{\text{chunk\_size tokens each}} \leq \text{token\_budget}$$

💡 Tip: with token budget as one ruler, “how big should a Chunk be,” “how many Decodes fit in a step,” and “how can long Prefill share a step with Decode” all collapse to the same packing problem—fill the GPU within budget, while keeping Decode from stalling.

---

4. vLLM V1’s unified scheduler

In early vLLM (V0), Prefill and Decode were two phases treated differently; the scheduling logic had to handle each, which made the code complex and left gaps when switching between them.

The vLLM V1 engine simplifies that cleanly: drop the Prefill/Decode split entirely. To the V1 scheduler there are no “Prefill requests” vs “Decode requests,” only “how many tokens each request processes this step.” A scheduling decision is a simple dict:

# Each V1 scheduler step is essentially this mapping
{
    "req_A": 1,      # Decoding: process 1 new token
    "req_B": 1,      # Decoding: process 1 new token
    "req_C": 512,    # One Prefill chunk of a long prompt
    "req_D": 128,    # Short prompt: Prefill finishes in one step
}
# Constraint: sum of all values ≤ token_budget

🔑 Core idea: V1 uses the single representation {request_id: num_tokens} to cover Chunked Prefill, Prefix Cache, and Speculative Decoding at once. They are all different values of “how many tokens this request processes this step”—a Prefill chunk is a larger number, Decode is 1, speculative verification is several candidate tokens. One scheduling frame unifies every case.

That is also why V1 can make Chunked Prefill the default behavior: scheduling already works in tokens and does not distinguish phases, so a long prompt is naturally cut into chunks by the budget, with no special-case logic.

---

5. Source walk-through: how Token Budget cuts a Chunk

Now that idea in vLLM V1 source. Scheduling lives in Scheduler.schedule() in vllm/v1/core/sched/scheduler.py. This section does not repeat the two-queue structure from §2.2; it focuses on “how token budget naturally turns a long Prefill into Chunks” (code keeps moving; treat your installed version as truth).

The key: when the scheduler decides “how many tokens this request processes this step,” it truncates by remaining budget. A long prompt that cannot finish in one step only takes what the budget allows—chunking is not a separate “chunker” function; it is the natural result of budget truncation.

# Pseudocode: reconstructs "budget truncation is chunking" inside schedule() (not literal source)
def schedule(self):
    token_budget = self.max_num_batched_tokens
    scheduled = {}

    # Running requests: Decode needs 1 token each; may also be a Prefill continuation from last step
    for req in self.running:
        if token_budget == 0:
            break
        # How many tokens this request still needs (unfinished Prefill, or 1 for Decode)
        need = req.num_tokens_with_spec - req.num_computed_tokens
        # Key: truncated by remaining budget → a long Prefill is "cut" into a chunk here
        num_tokens = min(need, token_budget)
        blocks = self.kv_cache_manager.allocate_slots(req, num_tokens)
        if blocks is None:
            ...                                  # OOM: preempt and retry; details in §2.2.6.2
            continue
        scheduled[req.id] = num_tokens          # {request_id: num_tokens}
        token_budget -= num_tokens

    # Budget left: pull new requests from waiting (also truncated into a first chunk)
    # In source, if this step already preempted, skip this block and admit no new requests
    while self.waiting and token_budget > 0 and self._can_admit():
        req = self.waiting.peek_front()
        # Check prefix cache first: hit tokens need no more compute (see §2.3)
        _, num_cached = self.kv_cache_manager.get_computed_blocks(req)
        need = req.num_tokens - num_cached
        num_tokens = min(need, token_budget)     # a long prompt's first chunk is truncated here too
        if self.kv_cache_manager.allocate_slots(req, num_tokens) is None:
            break                                # not enough memory: stop admitting new requests
        self.waiting.pop_front()
        self.running.append(req)
        scheduled[req.id] = num_tokens
        token_budget -= num_tokens

    return SchedulerOutput(scheduled)

The figure below walks that loop with concrete numbers. Budget 2048: 8 Decode requests take 1 each (blue); long request R still in running needs 1440 Prefill tokens and takes them all (yellow); the remaining 600 go to waiting request W1 with a 900-token prompt, truncated into its first Chunk (green); W2 gets no budget and stays in the queue (gray dashed).

Two points worth remembering:

1. min(need, token_budget) is the essence of chunking: a 4000-token prompt with only 512 budget left Prefills 512 this step and leaves 3488 for later—it automatically becomes a Chunk, with no dedicated “chunker.”
2. Continuations are tracked by num_computed_tokens: “how far this request has been computed” is persisted; the next step resumes from that point until num_computed_tokens catches num_prompt_tokens (Prefill done), then it naturally moves to Decode at +1 per step. That is §2.2’s “no Prefill/Decode phase split” on the data side.

📌 Key point: V1 scheduling in one line—num_computed_tokens is pushed toward num_tokens_with_spec; each step advances by min(remaining need, remaining budget). Chunked Prefill, Decode, and speculative verification are just that expression taking different values on different requests.

---

6. Key parameters and tuning

In vLLM, three parameters matter most for this section. When conditions allow, V1 turns Chunked Prefill on by default and prefers Decode when packing a batch, so you usually do not flip a switch by hand:

Tuning centers on the max_num_batched_tokens tradeoff:

• Throughput first: raise the budget, pack more tokens per step, fill the GPU more, raise system throughput; but Prefill takes a larger share and Decode TPOT stability drops.
• Latency stability first: lower the budget, keep per-step compute controllable, keep Decode smoother and TPOT tails lower; but each step does less and peak throughput may suffer.

vLLM’s official tuning docs give a reference scale: smaller values (e.g. 2048) get better ITL (Inter-Token Latency, the gap between adjacent output tokens, basically synonymous with TPOT) because less Prefill stalls Decode; larger values shorten TTFT because one step can Prefill more tokens; for throughput, set 8192 or above, especially small models on large-memory GPUs.

💡 Tip: there is no one best value. The right move is to load-test under your target traffic (real prompt-length mix and concurrency), watch TTFT, TPOT P95/P99, and throughput, and find the highest throughput that still meets your SLOs. Concrete tuning with examples comes in Chapter 3, “Tuning key vLLM settings.”

---

7. Tradeoffs and when it helps

✅ Where Chunked Prefill pays off clearly:

• Prompt lengths vary a lot, with very long prompts mixed in (long-document QA, long-context RAG)
• Online services that care about smooth token streaming and want to crush TPOT tails (e.g. live chat)
• High-concurrency online loads that mix Prefill and Decode

❌ Where gains are limited:

• Pure offline batch inference that only cares about total throughput, not per-request latency jitter—scheduling overhead from chunking may not be worth it
• Prompts are all short, so a long Prefill never monopolizes a step anyway

📌 Key point: Chunked Prefill targets interference when Prefill and Decode share the run—it is a “smoothness” optimization, not a “peak throughput” one. Whether to use it and how to tune it depends on whether your service cares about TPOT stability and whether the load has long prompts.

⚠️ Note: Chunked Prefill and Prefix Cache are natural partners—a prefix hit first cuts some Prefill compute, then the remaining miss portion is chunked into Decode steps. Stacked together, long-prompt workloads can improve both TTFT and TPOT.

---

📝 Summary

• A long prompt’s Prefill packed into one step stretches that step’s wall time, so Decode requests in the same batch see TPOT spikes.
• Chunked Prefill splits a long Prefill into chunks across many steps (in vLLM, chunk length is set by remaining budget), trading a little TTFT for steady Decode.
• Token Budget is the shared scheduling currency: each Decode request costs 1 token of budget, each Prefill Chunk costs its length, and a step packs within budget.
• vLLM’s V1 unified scheduler uses {request_id: num_tokens} to erase the Prefill/Decode boundary; one frame covers Chunked Prefill, Prefix Cache, and speculative decoding.
• In source, chunking is the natural result of min(remaining need, remaining budget), with num_computed_tokens tracking continuations—no separate chunker.
• The main knob is max_num_batched_tokens (throughput ↔ TPOT stability): official guidance is around 2048 for ITL, 8192+ for throughput, and you still need load tests on your traffic; with many long prompts, long_prefill_token_threshold can cap one request’s Prefill per step.

🎯 Self-check list

• Explain why a long Prefill slows TPOT for Decode requests in the same batch
• Describe how Chunked Prefill flattens one compute spike into several small ones
• State what Chunked Prefill trades (TTFT) for what gain (stable TPOT)
• Explain why Token Budget can measure Prefill and Decode in one unit
• Explain the benefit of V1’s {request_id: num_tokens} unified representation
• From source, explain “budget truncation is chunking” and how num_computed_tokens continues a chunk
• Describe how raising / lowering max_num_batched_tokens affects throughput and TPOT
• Judge whether your load is a good fit for Chunked Prefill

📚 References

• SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills
• vLLM V1 Alpha Release (unified token-budget scheduler)
• vLLM V1 source — vllm/v1/core/sched/scheduler.py (unified scheduling)
• vLLM — Optimization and Tuning (Chunked Prefill and max_num_batched_tokens)
• vLLM — Engine Arguments (long_prefill_token_threshold and other scheduling params)