3.1 Project overview and the four-layer architecture

📑 Contents

• 1. Why nano-vllm is worth reading
• 2. Directory layout at a glance
• 3. The four-layer architecture
• 4. Lifecycle of one request
• 5. How it relates to vLLM
• Summary
• Self-check checklist
• References

1. Why nano-vllm is worth reading

Production frameworks like vLLM and SGLang run past 100k lines of code. When you read them, it is hard to point at one line and say “this is where the batch is assembled.” nano-vllm rebuilds the backbone of an inference engine in about 1200 lines of Python, which makes it a good companion for reading the source side by side.

Think of vLLM as a skyscraper with basement plumbing, fire systems, and elevator scheduling. nano-vllm is the show-home with the structure left exposed: walls, load-bearing columns, and utility runs are all visible, so you can see how the building stands up. It skips the production extras (dozens of model adapters, every quantization scheme, cluster orchestration) and keeps only the algorithm and engineering backbone.

It fits two groups especially well: developers learning inference / serving systematically for the first time, and people preparing for LLM inference interviews who need to “talk at source-code level.” If you know nanoGPT, this analogy helps:

Once you can write a model’s forward, an inference engine still has to solve a different set of problems: how batches are assembled dynamically, how KV is stored in blocks, when to prefill and when to decode. nano-vllm’s code is organized around those questions.

💡 Tip: skim Chapter 2 before you start the source. This chapter does not re-explain PagedAttention or Continuous Batching themselves; it looks at how they show up in the code.

2. Directory layout at a glance

Clone the repo and the core lives under the nanovllm/ package. At the root you also get example.py (minimal usage) and bench.py (throughput benchmark). Build a mental map from filename to responsibility first:

nanovllm/
├── __init__.py          # Public exports: LLM, SamplingParams
├── config.py            # Config: global settings (dataclass)
├── llm.py               # LLM: subclasses LLMEngine; body is only pass
├── sampling_params.py   # SamplingParams: sampling hyperparameters
├── engine/
│   ├── llm_engine.py    # Engine main loop: schedule → forward → sample → update
│   ├── scheduler.py     # Prefill/Decode scheduling and preemption
│   ├── sequence.py      # Per-request state machine
│   ├── block_manager.py # KV Cache block allocate / free
│   └── model_runner.py  # GPU-side execution: tensor prep, Context, CUDA Graph
├── layers/
│   ├── attention.py     # Triton KV write + FlashAttention forward
│   ├── linear.py        # Column / row / QKV / merged parallel Linear
│   ├── embed_head.py    # Vocab-parallel Embedding and LMHead
│   ├── layernorm.py     # RMSNorm
│   ├── activation.py    # SiluAndMul (SwiGLU)
│   ├── rotary_embedding.py  # RoPE rotary position embedding
│   └── sampler.py       # Temperature sampling (exponential noise + argmax)
├── models/
│   └── qwen3.py         # Qwen3 model implementation
└── utils/
    ├── context.py       # Global inference context (is_prefill, cu_seqlens, ...)
    └── loader.py        # safetensors weight loading

By directory: engine/ decides who runs this step and how; layers/ and models/ do the matrix math; utils/ carries context across modules and loads weights. The top-level __init__.py only exports the names LLM and SamplingParams, so users only need those two entry points.

File size also shows where the weight sits. The four largest files are model_runner.py (257 lines), qwen3.py (216 lines), linear.py (156 lines), and block_manager.py (120 lines)—execution, model, tensor parallelism, and KV management. llm.py is only 5 lines.

3. The four-layer architecture

Group the files by “who calls whom” and you get the four layers below. Solid orange arrows are downward calls; dashed arrows are the execution layer handing KV Cache and logits back to the engine layer:

🔌 Layer 1: interface (llm.py + sampling_params.py + config.py)

Users only touch LLM (provides generate) and SamplingParams (temperature, max generation length, and so on). At engine init, Config locks in global constraints such as GPU memory fraction, max batch size, tensor-parallel degree, and KV block size. The public API here is tiny; you can run inference without understanding the scheduler.

🧠 Layer 2: engine (engine/llm_engine.py)

LLMEngine holds the scheduler, model executor, and tokenizer, and runs a loop: pull a batch of sequences from the scheduler → hand them to the executor for one forward pass and sample the next token → write the result back into sequence state, until every request finishes.

🗂️ Layer 3: scheduling (scheduler.py + sequence.py + block_manager.py)

This layer decides “who runs this step, and where their KV lives.” Scheduler chooses whether this round is prefill or decode and which sequences form a batch; Sequence tracks each request’s tokens, cached length, and state; BlockManager allocates physical KV blocks in the PagedAttention style. This layer only manages resource allocation and lifecycle; it never touches matrix multiplication, which is why BlockManager sits under engine/ rather than layers/.

⚙️ Layer 4: execution (model_runner.py + models/qwen3.py + layers/*)

ModelRunner turns the scheduler’s sequences into tensors: builds input_ids, positions, slot_mapping, block_tables, and so on, sets the global Context, then calls Qwen3 forward. layers/ are the operator-level implementations; attention.py has two jobs: write new K/V into the cache, and call FlashAttention for the attention compute.

4. Lifecycle of one request

Layering describes the static structure. The diagram below is the data flow: the path a string takes from entering the engine to coming out as full text. Sections 3.2–3.6 each expand one segment of that chain.

Diamonds are the main-loop step(); blue boxes belong to the scheduler; green boxes run on the GPU. The six boxes underneath are the body of each round:

Walk it step by step (interviews often ask “which code path does a request take once it comes in”):

1. 📥 The user calls llm.generate(prompts, sampling_params). A prompt can be a string or an already-encoded list of token ids; strings are encoded by the HuggingFace tokenizer inside add_request.
2. add_request creates a Sequence for each prompt and puts it on the scheduler’s waiting queue.
3. ⚙️ Each step() of the main loop: the Scheduler picks the sequences for this round from the Token Budget and remaining KV, and marks whether this is a prefill (process the prompt) or a decode (generate token by token).
4. ModelRunner turns that batch into GPU tensors, writes the global Context, and runs one Qwen3 pass. During the forward, Attention stores the newly computed K/V into the KV Cache, then uses FlashAttention for the attention compute.
5. The model’s hidden state goes through ParallelLMHead to get logits; Sampler samples the next token by temperature.
6. Scheduler.postprocess writes the new token back into the matching Sequence and checks for EOS or max_tokens. Finished sequences free their KV blocks; unfinished ones stay for the next decode round.
7. 📤 When every sequence is done, generate sorts by seq_id to restore submission order, decodes to strings, and returns.

📌 Key point: Prefill and Decode never share the same batch, which matches vLLM V0’s default scheduling. Prefill processes the whole prompt at once and is compute-bound; decode computes one token per sequence per step and is memory-bound. Each scheduler round is either all prefill or all decode; 3.5 expands that policy.

5. How it relates to vLLM

Interviews often ask “how does nano-vllm relate to vLLM.” A fair summary is same ideas, trimmed engineering.

✅ Preferred phrasing: nano-vllm compresses vLLM’s backbone algorithms and engineering path into a “readable range.” The goal is not to replace vLLM in production. It inherits PagedAttention, Continuous Batching, FlashAttention, TP, and related mechanisms, and leaves out engineering details such as quantization, huge model-adapter catalogs, and cluster orchestration.

❌ Don’t say this: “nano-vllm is behind / cruder than vLLM.” That confuses a teaching project with a production system’s role.

📝 Summary

• nano-vllm implements an inference-engine backbone in about 1200 lines of Python, a useful sample before reading vLLM-class systems.
• The code is organized in four layers—interface → engine → scheduling → execution. Calls go top-down; KV Cache and logits flow back bottom-up.
• One request’s lifecycle is a pipeline: tokenize → enqueue → schedule → tensorize → forward (write KV + attention) → sample → write back → decode.
• Prefill and Decode are scheduled in separate batches; that is the key premise for understanding the scheduler later.

🎯 Self-check checklist

• Can name nano-vllm’s author, code scale, and positioning, and state its relationship to vLLM in one sentence
• Can sketch the nanovllm/ directory layout from memory and say what engine/, layers/, and utils/ each own
• Can draw the four-layer call graph: interface / engine / scheduling / execution
• Can retell, in a short paragraph, the full data path from a string request to output text
• Can explain why BlockManager belongs in the scheduling layer rather than layers/
• Can explain why Prefill and Decode are scheduled separately, and what resource each one bottlenecks on

📚 References

• nano-vllm GitHub repository
• nanoGPT GitHub repository
• Efficient Memory Management for Large Language Model Serving with PagedAttention (Kwon et al., 2023)
• vLLM docs — architecture overview