5.2 Draft models and model-free methods

The previous section showed that as long as verification and correction are done right, a bad Draft guess will not bias the output. From a performance view, though, a draft that is slow to write and usually vetoed makes speculation wasted work. So this section asks: how do you keep the cost of “guessing a stretch” below the time saved by “calling the large model fewer times”?

The answer is not always another neural net. Model outputs often copy the prompt, code repeats the same identifiers, agents walk similar steps again and again — that history is already a draft source. This section puts a standalone small model, N-gram, and Suffix Decoding on one table, then unifies how acceptance rate is counted; 5.4 and 5.5 reuse those definitions.

⏰ Freshness: framework behavior and parameter defaults are based on vLLM v0.32.0 (released 2026-10-18) source and docs.

📑 Contents

• 1. What makes a good Draft: fast, accurate, and plug-compatible
• 2. Standalone small models: smaller is not always better
• 3. Tokenizer and context: both models must speak the same language
• 4. N-gram: find the next stretch already written in history
• 5. Suffix Decoding: from one match to history statistics
• 6. Hands-on: match window and candidate coverage
• 7. Acceptance rate: get the denominator straight first
• 8. Engineering choices: pick candidates by workload and resources
• Summary
• Self-check list
• References

---

1. What makes a good Draft: fast, accurate, and plug-compatible

The component that proposes candidates is the Proposer. A useful Proposer needs three things at once:

1. Cheap: low latency, memory, and communication cost to generate candidates.
2. Accurate: on real traffic, candidates often match Target’s choices.
3. Plug-compatible: token IDs, context, cache, and sampling rules can hook into the inference engine.

Like hiring an assistant: low pay is only one factor; you also care how much the editor has to rewrite the draft, and whether the files open in the editor’s software.

📌 Key point: “model-free” only means no extra neural weights. Lookup, index maintenance, CPU↔GPU data movement, and Target verification each still have a cost.

---

2. Standalone small models: smaller is not always better

2.1 Why a small model can be enough

Not every position in a text is equally hard. Punctuation, common connectives, grammar, variable names already in context — a small model can guess those reasonably well. Hand those “easy positions” to the small model; Target only has to verify.

Parameter count is a rough cost proxy. Draft latency also depends on depth, Attention structure, KV length, kernel implementation, and batch size. A deep 0.5B model generating token by token is not necessarily faster than a shallower 1B model.

2.2 Matching Target matters more than a high score on its own

When picking a Draft, look at how consistent it is with Target on your workload:

• Target is strong at code while Draft has barely seen code: syntax may pass, algorithm choices get rejected often.
• Different training data, alignment, or language mix shifts common continuation habits. A same-family small model trained on the same data is usually the first choice; the vLLM docs example is Qwen/Qwen3-8B with Qwen/Qwen3-0.6B.
• A larger Draft often accepts more, but the extra Draft time can eat the gain.

So the selection metric is measured “(Draft time + verify time) / committed tokens”; leaderboard scores are only a reference.

2.3 Parallel Draft: several candidates from one forward

A classic Draft runs $\gamma$ serial forwards, so longer drafts get slower. PARD (Parallel Draft Model) trains differently so one Draft forward predicts several positions; Draft time then barely grows linearly with $\gamma$. vLLM v0.32.0 enables this with "parallel_drafting": true; the official example pairs amd/PARD-Qwen3-0.6B with 12 candidates. These methods need specially trained Draft weights; flipping the switch on an ordinary small model does not make it a parallel Draft.

2.4 An extra model squeezes serving capacity

Besides weights, a standalone Draft has its own KV Cache, activations, and workspace. Total GPU memory is fixed, so space left for Target KV shrinks, and how many concurrent requests you can serve shrinks with it.

A single request getting faster while max concurrency drops can happen at the same time. Section 5.5’s walkthrough measures low-concurrency latency and serving capacity separately.

---

3. Tokenizer and context: both models must speak the same language

3.1 Same text, not necessarily the same token IDs

Section 5.1’s $p(x)$ and $q(x)$ assume the same token space. Two tokenizers both encoding “hello” does not mean they split the same way, let alone that each integer ID means the same thing.

Check the vocab’s token → ID map, special tokens (BOS/EOS/chat template markers), and the chat template Target actually uses. Matching vocab_size only means the counts match.

The check below needs transformers; it only downloads the two models’ tokenizer files — no weights, no inference (first run needs the network):

from transformers import AutoTokenizer

target = AutoTokenizer.from_pretrained("Qwen/Qwen3-8B")
draft = AutoTokenizer.from_pretrained("Qwen/Qwen3-0.6B")

t_vocab, d_vocab = target.get_vocab(), draft.get_vocab()
diff = {tok for tok in t_vocab.keys() | d_vocab.keys()
        if t_vocab.get(tok) != d_vocab.get(tok)}
print("Tokens with mismatched mapping:", len(diff))
print("EOS:", target.eos_token_id, draft.eos_token_id)

messages = [{"role": "user", "content": "你好"}]
same_template = target.apply_chat_template(messages, tokenize=True) == \
    draft.apply_chat_template(messages, tokenize=True)
print("Same dialog template encoding:", same_template)

At inference time Draft reads Target’s already-encoded history; it does not wrap the template again itself. So template differences affect whether Draft has “seen” that format, which in turn affects acceptance rate.

3.2 Different vocabs: a separate engineering path

vLLM by default requires a shared vocab. v0.32.0 adds use_heterogeneous_vocab: true, using Token-Level Intersection (TLI) so Drafts from different families and tokenizers work — but only for the draft_model method, and only with the default draft_sample_method="greedy".

3.3 Draft’s context must not quietly shrink

Draft must handle the total length of “actual history + candidates.” When max context, RoPE config, and long-context ability disagree, long requests may fail outright, or acceptance may drop sharply on long text. Before deploy, align Draft’s max_model_len with Target, or explicitly record that Draft only sees part of the history by design.

---

4. N-gram: find the next stretch already written in history

4.1 Match the tail, then copy what follows

Suppose the current history ends with [A, B, C], and earlier there was [A, B, C, D, E, F]. The lookup finds a matching tail and hands [D, E, F] to Target as candidates.

The figure below walks the lookup with $n=3$, $k=3$ (blue cells are the 3-gram matching the end; hatched yellow cells are the copied candidates):

Matching is on the token ID sequence, unrelated to Chinese word segmentation or character counts. Two independent parameters:

• Lookup window $n$: in vLLM, prompt_lookup_min and prompt_lookup_max. Neither set defaults to 5; set only one and the other takes the same value.
• Proposal length: num_speculative_tokens, how many following tokens to copy at most.

vLLM v0.32.0’s CPU implementation searches the entire request history (prompt plus generated content), prefers the longest match in $[\text{min},\text{max}]$, and on ties takes the earliest occurrence. There is also an ngram_gpu method that runs lookup on GPU, usable by both V1 and V2 Model Runners; under the V2 Model Runner, ngram also takes the GPU path, and v0.32.0 lets it support variable-length drafts.

4.2 Trade-offs on window length

• Long window: stronger constraint, candidates likelier relevant, but fewer hits.
• Short window: easy to hit, but you may only match a comma plus “的”, with wildly divergent continuations after.
• Long proposal: copy more at once, maybe save more Target calls — or get rejected on the first few and waste the rest.

Prompt Lookup fits tasks where “output reuses a lot of input”: summaries quoting the source, code changing a few lines, template field extraction.

4.3 No probabilities — how do you verify?

N-gram gives deterministic tokens. Per section 5.1 §3.3, treat that as one-hot $q$; acceptance collapses to “accept with probability $p(y)$,” and the residual is $y$ removed then renormalized by $p$. The conclusion is unchanged: the output distribution is still Target’s.

---

5. Suffix Decoding: from one match to history statistics

N-gram’s idea is “we wrote the same tail before, so we can probably copy what followed,” looking at one match each time. SuffixDecoding goes further: organize history with a suffix tree, record how often each continuation appeared, then pick the most confident path.

5.1 Frequency as cheap confidence

Suppose after some suffix, return appeared 20 times and raise twice; proposing return first is clearly better. The more scattered the continuations, the less worth guessing deep. vLLM filters tokens whose estimated probability is too low with suffix_decoding_min_token_prob (default 0.1).

These frequencies are only the Proposer’s internal ranking, not Target’s conditional probabilities; whether to keep them is still Target’s verification call.

5.2 History scope and dynamic length

Suffix Decoding maintains two kinds of suffix tree:

• Prompt tree: for the current request’s prompt and generated content.
• Global tree: caches historical outputs across requests, sized by suffix_decoding_max_cached_requests (default 10000); set to 0 to disable.

How many tokens to propose each step is decided dynamically: longer, more confident matches guess more. So here num_speculative_tokens is an upper bound; docs suggest 16 or 32. In v0.32.0, if unset, the code takes suffix_decoding_max_tree_depth (default 24). It shines on high-repetition work like code edits, agent self-reflection loops, and RL rollouts; when generating wholly new content, more history does not help.

5.3 Integration cost

💡 Tip: a global tree lets one user’s output influence another user’s candidates. Candidates still go through Target verification, so the output distribution is unaffected; in multi-tenant settings, whether that sharing is allowed is a business isolation call.

📌 Key point: Prefix Cache reuses KV already computed, cutting repeated Prefill; N-gram/Suffix propose tokens that may need generating, which still need verification. Both “reuse history,” but their metrics must not be mixed.

---

6. Hands-on: match window and candidate coverage

The code below uses integers as token IDs and implements a simplified lookup under vLLM’s rules: search from long windows to short, and on equal length take the earliest occurrence. It only demos the rules; vLLM actually uses a KMP variant that finishes in one scan.

def propose(history, min_n=2, max_n=4, k=3):
    """Return (hit window length, candidates); on miss return (None, [])."""
    for n in range(min(max_n, len(history) - 1), min_n - 1, -1):
        tail = history[-n:]
        # start < len - n excludes the tail itself; scan front to back for earliest match
        for start in range(len(history) - n):
            if history[start:start + n] == tail:
                return n, history[start + n:start + n + k]
    return None, []


h1 = [10, 11, 12, 20, 21, 99, 10, 11, 12]
print("h1:", propose(h1))                      # (3, [20, 21, 99])
print("h1 4-gram only:", propose(h1, min_n=4))  # (None, [])

h2 = [7, 8, 1, 7, 8, 2, 7, 8]                  # tail [7, 8] appeared twice
print("h2:", propose(h2))                      # earliest: (2, [1, 7, 8])
print("no repeats:", propose([1, 2, 3, 4, 5]))

h1 hits a 3-gram; forcing 4-gram yields no candidates. In h2 the tail [7, 8] was followed by both 1 and 2; the rule picks the earliest, proposing 1. If Target wants 2 this time, the candidate is rejected at the first position.

You can append more 7, 8, 2 to h2 and think about what Suffix Decoding would propose by frequency — the difference between “pick one match” and “pick a continuation by historical frequency.”

---

7. Acceptance rate: get the denominator straight first

7.1 Three related but different quantities

The rest of this chapter uses these definitions. In an observation window, $D$ candidate tokens were proposed, $C$ of them accepted, across $R$ verification rounds:

$$\text{Draft Acceptance Rate}=\frac{C}{D}$$ $$\text{Mean Acceptance Length}=1+\frac{C}{R}$$

The “+1” in mean acceptance length is the correction or bonus token every round commits. vLLM’s per-request metrics and Prometheus metrics both use these two definitions.

The third quantity is candidate coverage: how many steps actually proposed candidates. When N-gram finds nothing, that step does not speculate; if you only count rounds that had candidates, a 90% acceptance rate may sit on only 10% of steps actually speculating.

7.2 Fixed length and dynamic length do not convert into each other

Only when every round proposes the same $\gamma$ candidates do you have $D=\gamma R$, and mean acceptance length can be written $1+\gamma\cdot C/D$. For Suffix, dynamic SD, and other per-round variable lengths, you must compute from actual $D$ and $R$.

Another common mix-up is positional acceptance: “probability that position $i$ is accepted given earlier ones passed” versus “probability of accepting all the way from 1 through $i$” are different; the latter decays much faster as $i$ grows. vLLM’s vllm:spec_decode_num_accepted_tokens_per_pos_total (split by the position label) counts the latter.

⚠️ Note: 90% acceptance is not 90% speedup. Draft generation, verify length, memory capacity, and scheduler load all enter the time denominator; section 5.4 derives that ledger.

---

8. Engineering choices: pick candidates by workload and resources

Start with real prompts and sampling settings, measure candidate coverage, mean acceptance length, and end-to-end latency, then decide whether a more complex method is worth it. “Code” and “chat” are only task labels; what actually drives results is repetition, distribution match, and system cost.

✅ Upside: cheap, frequently hitting candidates can cut Target call count.

❌ Cost: a standalone model eats GPU memory; model-free methods need reusable history; both need extra cache and verification plumbing.

---

📝 Summary

• A good Draft must be cheap, accurate, and plug-compatible; few parameters are only one piece.
• A standalone small model can guess new content, but count weights, KV, and capability match; PARD cuts draft latency with parallel Draft.
• Same vocab_size is not same token IDs — check special tokens, templates, and context length; heterogeneous vocabs need TLI.
• N-gram matches the history tail then copies what follows; lookup window and proposal length are separate parameters; vLLM takes the longest earliest match.
• Suffix uses a suffix tree and frequencies to set proposal length dynamically; cost sits in tree build, cache, and dependencies.
• Deterministic candidates equal one-hot $q$; with standard verification they stay lossless.
• Report acceptance rate $C/D$, mean acceptance length $1+C/R$, and candidate coverage separately.

🎯 Self-check list

• Explain why a larger Draft may be more accurate yet slower
• List the extra GPU-memory items a standalone Draft occupies
• Write a short script checking two tokenizers’ token → ID maps
• Distinguish N-gram’s lookup window from proposal length, and state vLLM defaults
• Describe which match vLLM N-gram picks when several match
• Explain why Suffix frequencies still need Target verification, and the isolation risk of a global tree
• Distinguish Prefix Cache from proposing historical tokens
• Compute acceptance rate and mean acceptance length from $C$, $D$, $R$, and name the conversion trap under dynamic length

📚 References

• Fast Inference from Transformers via Speculative Decoding (Leviathan et al., 2023)
• Transformers — Assisted Decoding and Prompt Lookup
• SuffixDecoding: Extreme Speculative Decoding for Emerging AI Applications (Oliaro et al.)
• Snowflake — Arctic Inference official implementation
• PARD: Accelerating LLM Inference with Low-Cost PARallel Draft Model Adaptation (An et al.)
• vLLM — Draft Models
• vLLM — Parallel Draft Models
• vLLM — N-Gram Speculation
• vLLM — Suffix Decoding
• vLLM v0.32.0 — N-gram Proposer implementation