5.5 Speculative decoding with vLLM in practice

The first four sections broke “guess then verify” into sampling rules, candidate sources, helper networks, and a performance ledger. Now wire them back into a serving stack: same Target, same GPU, same request set — what actually changes when speculation turns on? Acceptance alone without an ordinary-Decode baseline, or Token/s without looking at outputs, cannot answer that.

This section gives a reproducible flow: run the baseline first, then N-gram, standalone Draft, and EAGLE-3 in order; split code-edit vs open chat stats; record acceptance, outputs, and performance together. Commands follow vLLM v0.32.0 docs and source. GPU experiments need your own environment — this text does not invent unmeasured scores.

⏰ Freshness: based on vLLM v0.32.0 (released 2026-10-18). Per-request acceptance metrics are experimental; fields may change across versions — pin the version.

📑 Contents

• 1. Pin the environment and control variables
• 2. Prepare code and chat input sets
• 3. Start baseline, N-gram, and Draft servers in order
• 4. EAGLE-3, Suffix, and MTP: check artifacts before configuring
• 5. Collecting acceptance: see the real denominator per request
• 6. Performance comparison: same workload and concurrency
• 7. Quality and how to read results
• 8. Debugging and the next tuning round
• Summary
• Self-check list
• References

---

1. Pin the environment and control variables

1.1 Install and record versions

Use a Linux GPU environment that vLLM supports. First confirm driver, CUDA, and Wheel compatibility with the GPU install docs. GPU memory must fit Target weights, KV Cache, workspace, and CUDA Graph; Draft groups need extra space.

Install in an isolated environment and archive version info:

python -m venv .venv
source .venv/bin/activate
python -m pip install 'vllm[bench]==0.32.0'  # bench subcommand needs extra deps
python -c "import torch, vllm; print('torch', torch.__version__, 'cuda', torch.version.cuda, 'vllm', vllm.__version__)"
nvidia-smi
mkdir -p results
python -m pip freeze > results/environment.txt
vllm serve --help > results/serve-help.txt
vllm bench serve --help > results/bench-help.txt

On another version, first check that version’s docs for method names and fields. Speculative-decoding knobs moved a lot recently — e.g. model-specific MTP method names unified to mtp, and the old --speculative-model flag is deprecated in favor of --speculative-config (short -sc).

1.2 Change only the proposal method

Target is always Qwen/Qwen3-8B. That choice gives three ready partners: same-family Qwen/Qwen3-0.6B as a standalone Draft (the official vLLM docs example), RedHatAI/Qwen3-8B-speculator.eagle3 as EAGLE-3 weights trained for it, and N-gram needing no extra weights. All four experiment groups share one baseline.

Qwen3 enters Thinking mode by default; this example forces enable_thinking=False. That controls variables and matches how these EAGLE-3 weights were trained (the model card notes Thinking was off during training). If the product needs Thinking on, output length and acceptance both change — remeasure.

Formal experiments should record each model’s revision, ideally download a fixed local snapshot for reruns.

---

2. Prepare code and chat input sets

2.1 Generate checkable JSONL

Save the code below as prepare_workloads.py. Running it yields two sets of 64 inputs each. The code set asks the model to change one function and keep the rest verbatim; the chat set asks for explanations on different topics with almost nothing to copy.

import json
from pathlib import Path

Path("data").mkdir(exist_ok=True)
source = "\n\n".join(
    f"def scale_{i}(value):\n    return value * {i + 1}" for i in range(12)
)
topics = [
    "explain why an inference service must watch both throughput and latency",
    "list possible reasons cache hit rate is high yet users still wait",
    "discuss how a team picks a deployment plan that stays maintainable",
    "explain why average metrics can hide a few severe failures",
]

for task in ["code", "chat"]:
    rows = []
    for i in range(64):
        if task == "code":
            prompt = (
                f"Only change the multiplier of scale_{i % 12} to {100 + i}; keep every other function unchanged. "
                "Output the full Python file with no explanation and no code fence.\n" + source
            )
        else:
            prompt = f"Case {i}: Please write two full paragraphs that {topics[i % len(topics)]}."
        rows.append({"prompt": prompt, "output_tokens": 128})
    with open(f"data/{task}.jsonl", "w", encoding="utf-8") as f:
        f.writelines(json.dumps(r, ensure_ascii=False) + "\n" for r in rows)

print("Wrote data/code.jsonl and data/chat.jsonl, 64 rows each (flow demo)")

python prepare_workloads.py
sha256sum data/*.jsonl > results/input-sha256.txt

2.2 What these two sets can show

This is a mechanism demo. The code set deliberately amplifies copyable text; it does not represent all code generation. The chat set does not represent every language, domain, or temperature. Formal evals should swap in enough real traffic samples, and every server version must read the same file.

The two sets have different input lengths, so comparisons stay inside the same task: code speculation vs code baseline, chat vs chat.

---

3. Start baseline, N-gram, and Draft servers in order

3.1 In terminal A, define shared parameters

Bash example below. Shared flags go in one array; each group only appends speculative config:

TARGET_MODEL=Qwen/Qwen3-8B
COMMON=(
  --served-model-name spec-demo
  --host 127.0.0.1
  --port 8000
  --dtype bfloat16
  --tensor-parallel-size 1
  --kv-cache-dtype auto
  --max-model-len 4096
  --max-num-seqs 16
  --max-num-batched-tokens 4096
  --gpu-memory-utilization 0.85
  --no-enable-prefix-caching
  --generation-config vllm
  --seed 42
  --per-request-spec-decode-metrics summary
)

# Group 1: ordinary Decode — do not pass --speculative-config
vllm serve "$TARGET_MODEL" "${COMMON[@]}"

What a few flags do: --generation-config vllm makes the server ignore the model repo’s default sampling params so all sampling comes explicitly from the client; --per-request-spec-decode-metrics summary attaches this request’s acceptance stats to each response (§5).

After model load, compile, and CUDA Graph capture finish, check from terminal B:

curl http://127.0.0.1:8000/health
curl http://127.0.0.1:8000/v1/models

After finishing §5 and §6 baseline collection, stop the server in terminal A with Ctrl+C and confirm GPU memory is free with nvidia-smi. Run groups sequentially — do not let several servers fight for the same GPU.

3.2 Group 2: N-gram

vllm serve "$TARGET_MODEL" "${COMMON[@]}" \
  --speculative-config '{"method":"ngram","num_speculative_tokens":4,"prompt_lookup_min":2,"prompt_lookup_max":4}'

Lookup window 2–4 Tokens; at most 4 proposals per round. With no window set, vLLM defaults both ends to 5. Steps with no match do not speculate, so watch candidate coverage (§5.2 §7).

3.3 Group 3: standalone Draft

Stop the previous server, confirm memory is free, then start:

vllm serve "$TARGET_MODEL" "${COMMON[@]}" \
  --speculative-config '{"method":"draft_model","model":"Qwen/Qwen3-0.6B","num_speculative_tokens":4}'

The official example uses 5 candidates; 4 here aligns with the N-gram group. Startup logs print Draft config and KV Cache capacity — record that beside the baseline group’s capacity to see how much space Draft squeezed.

--speculative-config only covers speculation itself. temperature, top_p, and other sampling params belong on the client request — do not put them in this JSON.

📌 Key point: every group uses the same served name spec-demo, so client scripts need no edits. A server that starts only means the config passed checks; outputs and performance still need item-by-item verification.

---

4. EAGLE-3, Suffix, and MTP: check artifacts before configuring

4.1 Group 4: EAGLE-3

EAGLE-3 weights must match the Target. The RedHatAI/Qwen3-8B-speculator.eagle3 model card states its verifier is Qwen/Qwen3-8B, so it plugs straight onto this example’s Target:

vllm serve "$TARGET_MODEL" "${COMMON[@]}" \
  --speculative-config '{"method":"eagle3","model":"RedHatAI/Qwen3-8B-speculator.eagle3","num_speculative_tokens":3}'

Candidate count 3 comes from the model card’s vLLM example; the card’s acceptance-length table also shows gains shrink clearly past 3–4. The card notes Thinking was off in training and that you should use /v1/chat/completions — §5’s collection script satisfies both.

Swap Targets and you must swap matching EAGLE-3 weights and re-run an ordinary Decode baseline for the new Target. method="eagle" is first-generation EAGLE weights; the two generations are not interchangeable by renaming the method.

4.2 Suffix Decoding

Suffix depends on Arctic Inference. Keep the vLLM version pinned on install so dependency resolution does not quietly swap the control environment:

python -m pip install 'vllm==0.32.0' arctic-inference
python -m pip check
python -m pip freeze > results/environment-suffix.txt

vllm serve "$TARGET_MODEL" "${COMMON[@]}" \
  --speculative-config '{"method":"suffix","num_speculative_tokens":16,"suffix_decoding_max_cached_requests":0}'

suffix_decoding_max_cached_requests: 0 turns off the cross-request global tree and keeps only the current request’s history — easier to explain in a demo. 16 is the per-round candidate cap; actual length varies dynamically. On dependency conflicts, pick a compatible combo from Arctic Inference’s notes and re-run the baseline.

4.3 MTP: only for models that ship MTP

Qwen3-8B has no MTP module, so MTP is not in this example’s control set. For a model that ships MTP, config is simplest:

vllm serve XiaomiMiMo/MiMo-7B-Base \
  --speculative-config '{"method":"mtp","num_speculative_tokens":1}'

Official advice starts tuning at 1 candidate. Such experiments must use that model’s own ordinary Decode as baseline.

⚠️ Note: rejection_sample_method: "synthetic" accepts candidates at a preset acceptance rate to stress-test system limits without a good Draft. It does not preserve the distribution — performance testing only; never in production config.

---

5. Collecting acceptance: see the real denominator per request

5.1 Acceptance stats in the response

With --per-request-spec-decode-metrics summary, an n=1 request carries this request’s stats in metrics.speculative_decoding. Main fields:

detailed also attaches per-round per_step_accepted and per_step_drafted, useful for dynamic-length methods like Suffix. When speculation is on but a request proposed zero candidates, the fields still exist with an all-zero histogram; a baseline server has no speculation, so the field is null.

5.2 Collection script

Save the code below as collect_acceptance.py. It runs four prompts via the non-streaming Chat API, saves each output and stats, then aggregates by task:

import argparse
import json
import time
from pathlib import Path
from urllib.request import Request, urlopen

parser = argparse.ArgumentParser()
parser.add_argument("label", help="e.g. baseline / ngram / draft / eagle3 / suffix")
args = parser.parse_args()

code = "def add(a, b):\n    return a + b\n\ndef multiply(a, b):\n    return a * b\n"
cases = [
    {"id": "edit", "task": "code",
     "prompt": "Rename add to subtract and make it subtraction; keep multiply unchanged. Output full code only, no fence.\n" + code},
    {"id": "copy", "task": "code", "prompt": "Output the following code verbatim, with no explanation or fence:\n" + code},
    {"id": "explain", "task": "chat", "prompt": "In two paragraphs, explain the difference between throughput and per-request latency in an inference service."},
    {"id": "tradeoff", "task": "chat", "prompt": "A team must trade off low cost against stable latency. Give a reasoned discussion method."},
]

rows = []
for case in cases:
    payload = {
        "model": "spec-demo",
        "messages": [{"role": "user", "content": case["prompt"]}],
        "temperature": 0, "top_p": 1, "seed": 42, "n": 1,
        "max_tokens": 256, "stream": False,
        "chat_template_kwargs": {"enable_thinking": False},
    }
    req = Request("http://127.0.0.1:8000/v1/chat/completions",
                  data=json.dumps(payload).encode(),
                  headers={"Content-Type": "application/json"})
    start = time.perf_counter()
    with urlopen(req, timeout=180) as resp:
        result = json.load(resp)
    choice = result["choices"][0]
    spec = (result.get("metrics") or {}).get("speculative_decoding")
    if args.label != "baseline" and spec is None:
        raise RuntimeError("missing acceptance metrics: check version, startup flags, and n=1")
    rows.append({**case, "output": choice["message"]["content"],
                 "finish_reason": choice["finish_reason"], "usage": result.get("usage"),
                 "request_seconds": time.perf_counter() - start, "speculative": spec})

Path("results").mkdir(exist_ok=True)
Path(f"results/{args.label}-acceptance.json").write_text(
    json.dumps(rows, ensure_ascii=False, indent=2), encoding="utf-8")

for task in ["code", "chat"]:
    stats = [r["speculative"] for r in rows if r["task"] == task and r["speculative"]]
    C = sum(s["num_accepted_draft_tokens"] for s in stats)
    D = sum(s["num_draft_tokens"] for s in stats)
    R = sum(s["num_spec_steps"] for s in stats)
    hist = [sum(col) for col in zip(*(s["acceptance_histogram"] for s in stats))]
    print(task, {"C": C, "D": D, "R": R,
                 "acceptance_rate": C / D if D else None,
                 "mean_acceptance_length": 1 + C / R if R else None,
                 "histogram": hist})

5.3 Run once per server group

# Run while the matching server is up
python collect_acceptance.py baseline
python collect_acceptance.py ngram
python collect_acceptance.py draft
python collect_acceptance.py eagle3

The four prompts only validate the collection flow. Formal comparisons need larger, length-comparable samples and a warmup first. The script’s non-streaming wall time includes Prefill and queueing — it is not TTFT or TPOT.

When aggregating, sum $C$, $D$, $R$ first, then take ratios — do not average per-request percentages. When $D=0$, print None: there is no denominator, which is not the same as “0% acceptance.” Histogram entry 0 (rounds that accepted nothing) often best explains why the chat group gains little.

5.4 Cross-check with server aggregate metrics

In /metrics, vllm:spec_decode_num_drafts_total, vllm:spec_decode_num_draft_tokens_total, and vllm:spec_decode_num_accepted_tokens_total match the per-request fields $R$, $D$, $C$ summed. vllm:spec_decode_num_accepted_tokens_per_pos_total carries a position label; the count at position $i$ is rounds that accepted at least $i+1$ candidates — divide by num_drafts_total for per-position cumulative acceptance, which shows how fast it decays with position.

These are cumulative since process start; take the difference of snapshots before and after the experiment. Server aggregates do not split code/chat; warmup and n>1 requests count too. Streaming requests put per-request metrics in the last usage chunk — set stream_options.include_usage: true or start the server with --enable-force-include-usage.

---

6. Performance comparison: same workload and concurrency

6.1 Run the benchmark from terminal B

Set LABEL to whichever group the current server is, then run the loop below to save one result per “task × concurrency” combo:

LABEL=baseline  # after switching servers: ngram / draft / eagle3
for TASK in code chat; do
  for CONCURRENCY in 1 8; do
    vllm bench serve \
      --backend openai \
      --base-url http://127.0.0.1:8000 \
      --endpoint /v1/completions \
      --model spec-demo \
      --tokenizer Qwen/Qwen3-8B \
      --dataset-name custom \
      --dataset-path "data/${TASK}.jsonl" \
      --chat-template-kwargs '{"enable_thinking":false}' \
      --custom-output-len 128 \
      --num-prompts 64 \
      --num-warmups 4 \
      --max-concurrency "$CONCURRENCY" \
      --request-rate inf \
      --temperature 0 \
      --top-p 1 \
      --ignore-eos \
      --seed 42 \
      --percentile-metrics ttft,tpot,itl \
      --metric-percentiles 50,95,99 \
      --save-result \
      --result-dir results \
      --result-filename "${LABEL}-${TASK}-c${CONCURRENCY}.json"
  done
done

The custom dataset applies Target’s Tokenizer Chat Template on the client (Thinking explicitly off here), then sends the formatted text to the Completions API. §5’s collection script uses the Chat API, with the server applying the template. The two paths have different jobs, but template settings must match.

6.2 Fixed length is for performance only

--ignore-eos makes every request generate a full 128 Tokens so groups do not differ in workload because natural endings differ. The cost is output that may run past a normal answer — so this batch of text is not for quality judgment.

64 requests is a workable starting point; P95/P99 are unstable on such a small sample. Formal experiments need more samples and repeats, keep the raw JSON, and also measure under a finite arrival rate (--request-rate set to a concrete number).

6.3 Record by task, concurrency, and baseline pairing

Speculation mainly improves TPOT and output throughput; TTFT is mostly queueing and Prefill. One streaming chunk may hold several Tokens, so ITL’s shape differs from the baseline — §5.4 §7 explains why.

---

7. Quality and how to read results

7.1 Rule out obvious config mistakes first

Compare each group’s *-acceptance.json: did code prompts keep unmodified functions and apply the right edit; did chat prompts finish normally. When finish_reason is "length", check whether the output budget was too small before looking elsewhere.

This example uses greedy decoding, so group outputs should match. On differences, first check template and sampling params, then Batch shape and Kernel numeric differences (§5.1 §8). Under random sampling, the same Seed does not guarantee word-for-word identity — compare task-metric distributions.

7.2 Then run a separate task eval

Formal quality evals must be separate from Draft or EAGLE training data, cover code execution correctness, math, structured output, long text, multi-turn chat, and restore normal EOS/Stop behavior. Reports should state the acceptance mode (standard rule or other) and whether Target is quantized.

With Draft-only quantization you can keep the original Target baseline; after quantizing Target or switching Target, rebuild the baseline so several changes are not mixed.

7.3 How to read four common outcomes

• Code group acceptance high, speedup weak: check verify length, Draft time, and whether the baseline was already fast.
• N-gram acceptance high, but many steps had no candidates: check the histogram and $D/R$; report candidate coverage together.
• Concurrency 1 faster, concurrency 8 throughput down: at high load speculation fights other requests for compute, or KV capacity shrank — consider Dynamic SD (§5.4 §3).
• Mean TPOT better, stream pauses longer: check ITL P95/P99; interactive feel is not just mean Token/s.

Fix the configs and samples you will compare before looking at results. Keep no-gain and failed configs too — they mark where the method applies.

---

8. Debugging and the next tuning round

8.1 Common symptoms and check order

Change one variable at a time. If you change candidate count, model, and concurrency together, you cannot explain why results got better or worse.

8.2 What to tune next

• Candidate count: at fixed concurrency sweep 1, 2, 3, 4…, plot TPOT and mean acceptance length, find where gains start falling (§5.4 §3).
• Concurrency bands: if low and high concurrency want different optimal lengths, set num_speculative_tokens_per_batch_size in bands; high-load bands can be 0.
• N-gram window: watch acceptance and candidate coverage together; a wider window raising coverage while lowering acceptance is normal — look at TPOT under both.
• Switch method: copy-heavy workloads compare N-gram vs Suffix first; new-content-heavy workloads compare standalone Draft vs EAGLE-3; if the model ships MTP, try MTP first.

✅ Ready-to-ship signal: at target load, TPOT and throughput both beat baseline, quality evals pass, capacity loss is acceptable.

❌ Abandon signal: gains only at concurrency far below target load, or acceptance is high but end-to-end time does not improve.

---

📝 Summary

• Every speculative group shares one ordinary Decode baseline; only --speculative-config changes; servers run in sequence.
• Code and chat are tallied separately; comparisons stay inside one task.
• N-gram and Suffix need no extra weights; standalone Draft wants same family and vocab; EAGLE-3 weights must match Target; MTP only for models that ship the module.
• --per-request-spec-decode-metrics puts $C$, $D$, $R$ and the acceptance histogram on each response; aggregate by summing first, then ratios.
• Performance tests use --ignore-eos to fix workload; quality judgment uses normally finished outputs — keep the two batches separate.
• Read results with KV capacity, preemption, and P95/P99; record no-gain configs too.

🎯 Self-check list

• List variables that must stay fixed in a controlled experiment, and why
• Write --speculative-config for N-gram, standalone Draft, EAGLE-3, Suffix, and MTP
• Explain why EAGLE-3 weights cannot be reused on another Target as-is
• Enable per-request acceptance metrics and read $C$, $D$, $R$, and the histogram from a response
• Aggregate acceptance rate and mean acceptance length by task, and distinguish $D=0$ from 0% acceptance
• Cross-check per-request stats with before/after /metrics snapshot diffs
• Use vllm bench serve to compare baseline and speculative groups under the same workload and concurrency
• Explain why --ignore-eos is only for performance tests
• Localize causes from symptoms like “high acceptance, no speedup” or “high-concurrency throughput drop”

📚 References

• vLLM v0.32.0 Release Notes
• vLLM — GPU install
• vLLM — Speculative Decoding
• vLLM — N-Gram Speculation
• vLLM — Draft Models
• vLLM — EAGLE Draft Models
• vLLM — Suffix Decoding
• vLLM — Multi-Token Prediction (MTP)
• vLLM — Per-Request Acceptance Metrics
• vLLM — Dynamic Speculative Decoding
• vLLM — vllm bench serve CLI
• vLLM v0.32.0 — Benchmark dataset implementation (CustomDataset)
• Hugging Face — RedHatAI/Qwen3-8B-speculator.eagle3
• Hugging Face — Qwen/Qwen3-8B
• Hugging Face — Qwen/Qwen3-0.6B
• Snowflake — Arctic Inference