5.4 Benefit boundaries of speculative decoding

“Draft acceptance is 80% — so it must be faster?” That is the most common misread of speculative decoding. Acceptance only says how often the guess is right; it does not say how long a guess takes, how much compute verifying a stretch costs, or how many requests the extra cache squeezed out. Each round commits more Tokens, and each round also takes longer — the ratio is the real gain.

This section first derives the speedup condition from a simplified ledger, then folds in concurrency, long context, quantization, scheduling, and streaming one by one. The goal is not a universal “best speculative length,” but the ability to explain why one setup got faster and another was not worth it.

⏰ Freshness: the formulas are framework-agnostic; vLLM advice, parameters, and limits follow v0.32.0 (released 2026-10-18) docs. Acceptance rate and mean acceptance length are defined in §5.2 §7.

📑 Contents

• 1. Define the gain first: how many Tokens one round commits
• 2. Performance ledger: derive the speedup condition from the accepted prefix
• 3. Speculative length: when guessing more becomes waste
• 4. Code vs chat: task labels are not performance conclusions
• 5. High concurrency and long context: spare compute changes
• 6. Stacking with quantization: fix the correctness baseline first
• 7. Scheduling and streaming: after each advance gets longer
• 8. Hands-on accounting and deploy gates
• Summary
• Self-check list
• References

---

1. Define the gain first: how many Tokens one round commits

A round proposes $\gamma$ candidates. Let $K\in\{0,\ldots,\gamma\}$ be the length of the continuously accepted prefix. When not truncated by EOS or a length cap, Tokens committed this round are:

$$L=K+1$$

The extra 1 is the correction Token at the first reject, or the bonus Token after full accept. So even if every candidate fails, the round still advances one step.

📌 Key point: “at least one Token of progress” is a progress guarantee, not a performance guarantee. A wipeout round spends Draft plus verify time and only buys one ordinary Decode step of output.

---

2. Performance ledger: derive the speedup condition from the accepted prefix

2.1 Expected commit length

For nonnegative integer $K$, the tail-probability identity holds:

$$\mathbb E[L]=1+\sum_{i=1}^{\gamma}\Pr(K\ge i)$$

Let $a_i$ be the conditional probability that “the $i$-th is also accepted given the first $i-1$ already were.” By the chain rule:

$$\Pr(K\ge i)=\prod_{j=1}^{i}a_j$$

This step does not need independence across positions. If you further simplify every $a_i$ to the same $a$:

$$\mathbb E[L]=1+a+a^2+\cdots+a^\gamma=\frac{1-a^{\gamma+1}}{1-a}\quad(a\ne1)$$

When $a=1$, the limit is $\gamma+1$. Dynamic-length and tree methods have a different candidate count each round; use the measured mean acceptance length $1+C/R$ directly.

2.2 The denominator is a full round

Let ordinary Decode’s mean time per Token be $T_0$, and a speculative round’s mean time be:

$$T_{\text{round}}=T_{\text{draft}}+T_{\text{verify}}+T_{\text{other}}$$

$T_{\text{other}}$ covers sampling, cache maintenance, CPU/GPU sync, and scheduling. Under the simplified single-request, steady Decode, ignore first/last boundary-round conditions, the speedup is:

$$S\approx\frac{T_0\cdot\mathbb E[L]}{\mathbb E[T_{\text{round}}]}$$

To get faster you need $\mathbb E[T_{\text{round}}]<T_0\cdot\mathbb E[L]$: one round must cost less than ordinary Decode producing the same number of Tokens. In measurement use “total committed Tokens / total time”; do not arithmetic-mean per-round speed ratios.

2.3 Verification is not free

Verifying $\gamma+1$ positions at once can reuse one weight read and raise arithmetic intensity. But compute, Attention, writing candidate KV, and Logits work all grow with position count. At small Batch, $T_{\text{verify}}$ is near $T_0$; with long candidates, large Batch, or long context it clearly exceeds $T_0$.

Back to the editor metaphor: reviewing a stretch is faster than writing it yourself, but reviewing ten pages cannot cost the same as reviewing one sentence — and the assistant’s writing time still counts.

---

3. Speculative length: when guessing more becomes waste

In the fixed-$a$ simplified model, growing candidates from $\gamma$ to $\gamma+1$ only adds:

$$\Delta\mathbb E[L]=a^{\gamma+1}$$

to expected commit length. Farther out, the chance that everything before passed shrinks. At $a=0.8$, the 1st candidate brings 0.8 expected Tokens; the 8th only about 0.168. Draft and verify cost per extra position do not shrink with that, so past some length extra guessing is a net loss.

3.1 Depth and width are two budgets

Chain speculation only tunes depth; tree methods (§5.3) also tune branches per level. A wider tree has a better chance of finding an acceptable path, but unpicked nodes are all computed too. Reporting only “speculative depth 4” is not enough — also report how many nodes each round actually verified.

3.2 Dynamic length: adjust with load

Optimal length shifts with load. vLLM’s Dynamic Speculative Decoding uses num_speculative_tokens_per_batch_size to set length by concurrency bands, e.g. [[1, 64, 3], [65, 128, 1], [129, 512, 0]] means guess 3 at concurrency 1–64, 1 at 65–128, and none at 129+. The docs’ reason: with Batch $B$ and $\gamma$ candidates per request, the effective verify Batch is about $B\times\gamma$; past a threshold that slows TPOT.

In v0.32.0 this was measured on EAGLE, EAGLE-3, and DFlash; with Data Parallel on it falls back to fixed length. There is also Adaptive Verification that picks verify length per request, supporting model-based drafts like EAGLE, MTP, DFlash, DSpark, and standalone Draft: DSpark uses its own confidence head; the others use online-estimated acceptance probability. It needs a full CUDA Graph and does not support LoRA or Pipeline Parallel.

A good policy optimizes “qualified output advanced per unit time,” not acceptance rate alone.

---

4. Code vs chat: task labels are not performance conclusions

“Code has high acceptance; open chat has limited gain” is a common rule of thumb, but what actually matters is history repetition and distribution match:

Temperature, Top-p, alignment style, and Prompt mix all affect acceptance. One “copy the code as-is” problem does not stand for all code tasks; one high-temperature creative prompt does not stand for all chat.

⚠️ Note: random-token benchmarks are fine for controlling performance variables, but they lack real workloads’ repetition structure. To judge N-gram/Suffix’s real gain, use real Prompts.

---

5. High concurrency and long context: spare compute changes

5.1 Continuous Batching already ate some idle capacity

At low concurrency each step handles few requests; Target’s compute units sit mostly idle, and speculation uses that spare. As concurrency rises, Continuous Batching lets one weight read serve more requests — ordinary Decode itself fills the compute.

Adding candidates then shifts from “use idle compute” to “fight other requests for compute.” vLLM docs also position speculative decoding as cutting inter-Token latency in mid/low QPS, memory-bound settings. Its method-choice table’s qualitative call: EAGLE and MTP gain high at low QPS, mid-to-high at high QPS; N-gram and Suffix gain low-to-mid at low QPS, mid at high QPS. The docs also note these simpler methods accelerate modestly but do not add load at traffic peaks.

High concurrency is not automatically useless: a better Draft, parallel drafting, and a fitting Kernel can still keep gains. Draw curves by load; do not dismiss from one anecdote.

5.2 Long context adds reads/writes and temporary state

Longer context makes Attention per verify position more expensive, and candidates need extra KV; a standalone Draft’s own cache also grows with context. With $B$ requests in the Batch and about $\gamma$ candidate slots reserved each, a rough order for Target’s extra KV is:

$$M_{\text{candidate}}\approx B\cdot\gamma\cdot m_{\text{KV/token}}$$

This estimate ignores page rounding, lookahead slots, shared prefixes, and tree layout — it only names the extra term. Real capacity comes from engine logs.

5.3 Preemption can eat paper gains

Draft weights and temporary cache shrink available KV space, which may trigger more preemption and recompute, or let the engine admit fewer requests. One-round verify getting faster while the whole request finishes slower is not a contradiction.

So when you log acceptance length, also log KV capacity, preemption count, successful request count, and real throughput.

---

6. Stacking with quantization: fix the correctness baseline first

6.1 Quantizing Draft and quantizing Target are different

• Quantize Draft only: changes the proposal distribution $q$. Per the §5.1 proof, final output is still the unquantized Target’s $p$; only speed and acceptance change.
• Quantize Target: the verification baseline becomes $p_{\text{quant}}$. Speculative decoding guarantees consistency with the quantized model; it does not restore original precision.
• Approximate acceptance + quantization: two changes stack; evaluate quality separately.

Modules like EAGLE and MTP that read Target features saw full-precision features in training. Change Target’s quantization and the feature distribution shifts; acceptance may drop — remeasure. v0.32.0 release notes include a fix for 0% Draft acceptance on DeepSeek-V4-series NVFP4 checkpoints. Combinations like that deserve a dedicated check.

6.2 Speedups do not multiply

Quantization and speculation both ease Decode’s memory bottleneck. Quantization cuts Target weight-read volume, so verify compute that used to hide behind memory traffic can become more visible; it also frees GPU memory for more candidates and requests.

So “quantization 1.5×, speculation 2×” does not imply 3× combined. Measure original baseline, quant-only, spec-only, and the combo separately, and state each quality baseline.

📌 Key point: Chapter 4’s precision ledger and Chapter 5’s sampling guarantee have to align on the same Target before you talk about combined gain.

---

7. Scheduling and streaming: after each advance gets longer

7.1 The scheduler manages more than one new position

With speculation on, each step the engine reserves slots for candidates, allocates Token budget, then updates each request’s real progress from acceptance. Inside one Batch, different requests may differ in candidate count, acceptance length, and remaining output quota.

The figure’s top half is one scheduler round; the bottom is three requests’ state at the end of that round in the same Batch (green = accepted candidates, blue = Tokens Target filled in, gray dashed = rolled-back KV slots):

That path interacts with Continuous Batching, Chunked Prefill, CUDA Graph’s fixed shapes, and async scheduling. For example Dynamic SD in v0.32.0 only has full CUDA Graph on the V2 Model Runner. Which combinations work follows the docs and startup logs of your version.

7.2 TPOT can drop without more even streaming

One speculative verify may return several Tokens, then the next round waits a while — what users see looks more like “bursts of chunks.”

• TPOT: after the first Token, mean time per Token.
• ITL / stream gap: pause between adjacent returns. One chunk may hold several Tokens; that is not the same as single-Token latency.
• TTFT: includes queueing and Prefill; fewer Decode rounds barely help it.

Eval should put TTFT, TPOT, stream pauses, and end-to-end time together. Interactive workloads also need P95/P99 and the longest pause the product can accept.

7.3 Stop conditions and structured output must take effect promptly

If the accepted prefix hits EOS, a Stop String, or the output cap, only commit what came before the end. Stateful constraints like structured output and Tool Calling must update with each candidate prefix; vLLM’s per-request num_draft_tokens already subtracts candidates voided early by structured-output constraints.

When something breaks, first get ordinary generation and the simplest speculative config working, then add features one by one so you can see which step changed correctness or performance.

---

8. Hands-on accounting and deploy gates

8.1 Same acceptance, completely different denominators

Below is a stdlib-only hypothetical cost model. Time constants are for demo only — they match no GPU or real model.

def expected_length(conditional_acceptance):
    """E[L] = 1 + Σ Pr(K >= i); input is each position's conditional acceptance a_i."""
    survival, total = 1.0, 1.0
    for a in conditional_acceptance:
        assert 0 <= a <= 1
        survival *= a
        total += survival
    return total


base_ms = 10.0  # assume ordinary Decode is 10 ms per Token
print(f"{'γ':>2} {'E[L]':>6} {'cheap draft':>8} {'costly draft':>8}")
for gamma in [1, 2, 4, 8]:
    length = expected_length([0.8] * gamma)
    cheap_round = 10 + 0.7 * gamma + 0.1 * gamma ** 2  # draft fast, verify grows slowly
    costly_round = 10 + 5.0 * gamma + 0.8 * gamma ** 2  # draft slow, verify grows fast
    print(f"{gamma:>2} {length:>6.3f} "
          f"{base_ms * length / cheap_round:>8.3f} "
          f"{base_ms * length / costly_round:>8.3f}")

assert expected_length([0.0] * 4) == 1
assert expected_length([1.0] * 4) == 5

Both schemes share the same conditional acceptance: the cheap one peaks near 2.3× at $\gamma=4$ then falls by $\gamma=8$; the costly one is only slightly faster for $\gamma\le2$ and slower than ordinary Decode from $\gamma=4$ on. In a real experiment, replace the two hypothetical cost functions with measured round times and commit stats.

8.2 Four gates before going live

When mean acceptance length grows, round time also grows, and TPOT does not improve, localize Draft, verify, sampling, and cache maintenance one by one. If it only looks good idle, remeasure at target load, then decide whether to enable by scenario or hand off to Dynamic SD to auto-degrade.

---

📝 Summary

• Per-round commit length is the continuous accepted prefix plus 1; boundary rounds may be truncated by stop conditions.
• $\mathbb E[L]=1+\sum_i\Pr(K\ge i)$ does not require position independence; a fixed conditional acceptance $a$ is only a simplification.
• Speedup requires a full round to cost less than ordinary Decode producing the same Token count.
• The $(\gamma+1)$-th candidate only adds $a^{\gamma+1}$ expected Tokens — diminishing returns; tree width has a cost too.
• Dynamic SD adjusts speculative length by concurrency band and can cut or disable speculation at high concurrency.
• Code and chat are only experiment groups; repetition structure and distribution match explain the gaps.
• High concurrency, long context, and preemption change spare compute and memory budget; vLLM positions speculation for mid/low QPS.
• Quantizing Draft does not change the output baseline; quantizing Target does; the two speedups do not multiply.
• TPOT, stream pauses, capacity, and product quality together decide whether to deploy.

🎯 Self-check list

• Derive expected commit length from the continuous accepted prefix
• Explain why chaining conditional acceptance probabilities does not need independence
• Write the speedup condition that includes Draft, verify, and other overhead
• Use $a^{\gamma+1}$ to explain diminishing returns on speculative length
• Write a concurrency-banded num_speculative_tokens_per_batch_size config for vLLM and explain the rationale
• Explain why random Prompts are not enough to evaluate N-gram/Suffix
• Analyze how high concurrency, long context, and preemption eat speculative gains
• Distinguish quantizing Draft vs Target for the output baseline
• Explain why a better TPOT does not guarantee better stream pauses
• Make deploy decisions against the four gates: correctness, quality, latency, capacity

📚 References

• Fast Inference from Transformers via Speculative Decoding (Leviathan et al., 2023)
• Accelerating Large Language Model Decoding with Speculative Sampling (Chen et al., 2023)
• vLLM — Speculative Decoding (method selection table)
• vLLM — Dynamic Speculative Decoding
• vLLM — Adaptive Verification
• vLLM — Per-Request Acceptance Metrics
• vLLM v0.32.0 Release Notes
• vLLM v0.32.0 — V1 Scheduler