3.9 Tokenization and word embeddings

A Transformer does not take raw text. It takes vectors. Before a piece of writing becomes model input, it goes through two transforms: Tokenization and Embedding (word embedding). They look simple, but they decide what the model can “see” and “understand” — the start of the whole Transformer pipeline, and basic literacy for language modeling.

📑 Contents

• 1. Why text has to become vectors
• 2. Tokenization: splitting text into basic units
• 3. One-hot encoding: the first attempt that failed
• 4. Word Embedding: a semantic map in geometric space
• 5. Missing position: why Positional Encoding is still needed
• 6. From Token ID to input matrix: the Transformer input layer
• Self-check list
• References

---

1. Why text has to become vectors

Computers know numbers, not text. “猫”, “cat”, “Katze” — to a computer these are all meaningless symbol strings. It does not know that “猫” and “狗” are both animals, or that “happy” and “joyful” mean something similar.

Neural nets need numeric matrices as input. So before text enters a Transformer, two problems have to be solved:

1. Segmentation: how do you split continuous text into processable “units” (Tokenization)?
2. Encoding: how do you turn each “unit” into a numeric vector that can capture semantic relations (Embedding)?

How you solve those two problems shapes how well the model understands language and how efficient inference is.

---

2. Tokenization: splitting text into basic units

2.1 What a Token is

A Token is the smallest unit a Transformer uses for language — not a character, not a word, but a “word piece” in between.

Different split strategies turn the same string into different token sequences:

Input: "Tokenization"

Character-level:  T / o / k / e / n / i / z / a / t / i / o / n      (12 tokens)
Word-level:       Tokenization                       (1 token)
Subword:          Token / ization                    (2 tokens)  ← used by mainstream LLMs

2.2 Subword splitting: what large models actually use

Modern large models mostly use subword splitting. The most common method is BPE (Byte Pair Encoding).

📌 Key point: BPE starts from characters and repeatedly merges high-frequency character pairs into larger tokens, until you get a vocabulary that covers common words and builds rare words from subword pieces.

Intuitively: BPE merges frequent combinations into single tokens and leaves rare ones as concatenated subwords. Common words (“play”, “run”) usually stay as whole tokens; rare or new words (like “tokenization”) may become “token” + “ization”, so you avoid unknown-token (OOV) problems. One caveat: BPE splits are purely data-driven statistics. They do not have to follow linguistic roots or suffixes; the result depends on the training vocabulary.

Vocabulary sizes in real models:

⚠️ Note: A larger vocabulary covers more, but the Embedding matrix (covered below) also gets bigger. LLaMA-3 grew the vocab from 32K to 128K and quadrupled Embedding-layer parameters.

2.3 Tokenization for Chinese

Chinese has no natural space separators, so the handling differs from English:

• Most large models do character-level Tokenization on Chinese (one token per Chinese character), or split on UTF-8 byte sequences of characters
• Models tuned for Chinese (Qwen, ChatGLM series, and similar) usually keep a larger vocabulary with more Chinese phrases as single tokens, which improves generation efficiency

---

3. One-hot encoding: the first attempt that failed

Once you have Token IDs, the simplest vectorization is One-Hot Encoding: if the vocabulary has $V$ tokens, each token is a length-$V$ vector with a 1 at its index and 0 elsewhere.

Example: vocabulary = [“猫”, “狗”, “鱼”, “河”, “银行”], size $V = 5$:

$$\text{猫} = [1, 0, 0, 0, 0] \qquad \text{狗} = [0, 1, 0, 0, 0] \qquad \text{鱼} = [0, 0, 1, 0, 0]$$

It looks usable, but two fatal problems show up:

Problem 1: dimension explosion

Real vocabularies have tens of thousands of tokens, so each vector has tens of thousands of dimensions — storage and compute blow up. Worse, feeding that sparse vector into a neural net forces later layers to use huge weight matrices. Efficiency is terrible.

Problem 2: lost semantics

One-hot vectors are fully orthogonal — the inner product of “猫” and “狗” is 0, and so is “猫” and “鱼”. Mathematically every token is the same distance from every other; nothing is related. The model cannot learn any semantic relation from the encoding: “猫” and “狗” are both pets and both animals, and that information is simply not there.

$$\text{猫} \cdot \text{狗} = 0, \quad \text{猫} \cdot \text{鱼} = 0 \quad \Rightarrow \text{to the model, "cat–dog" = "cat–fish"}$$

---

4. Word Embedding: a semantic map in geometric space

4.1 Core idea: similar words sit close together

The core idea of Word Embedding is: map each token into a low-dimensional continuous vector space so that semantically similar words are close in that space.

You compress the sparse $V$-dimensional one-hot vector into a dense $d_{model}$-dimensional vector (usually $d_{model} \ll V$):

A 2D sketch (real Embedding spaces are much higher-dimensional):

         Animals
    狗 ●   ● 猫
             ● 兔
       ● 鸟
                   Finance
              ● 银行  ● 钱
                    ● 贷款

         ● 河流   Geography
    ● 湖泊

“猫” and “狗” sit close (both common pets). Note: in static Word Embedding (Word2Vec and the like), the token “银行” has one fixed vector, placed by its average co-occurrence pattern across the whole corpus — finance contexts and riverbank contexts both pull on it, so the vector lands somewhere between. True disambiguation is Self-Attention’s job: it adjusts each token’s representation from other words in context (“money” or “river”) and produces different contextual vectors for “银行”. The Embedding layer is the starting point; Self-Attention does dynamic disambiguation.

4.2 Surprising geometry in Embedding space

Word Embeddings trained on large corpora show striking geometric structure:

Word analogies:

$$\text{king} - \text{man} + \text{woman} \approx \text{queen}$$

So in Embedding space, the direction from “king” to “queen” is almost parallel to the direction from “man” to “woman”. Semantic relations become geometric relations.

Parallelism of grammatical transforms:

$$\text{walking} - \text{walk} \approx \text{swimming} - \text{swim} \approx \text{running} - \text{run}$$

Displacement vectors from base form to progressive are almost the same for all these verbs — the tense change becomes one shared displacement direction.

💡 Tip: Nobody hand-designed these properties. The model learns them from massive text. Across centuries of human writing, co-occurrences like “狗追猫” and “猫抓鱼” show up far more often than “猫追猫”, so word positions settle into the layout we see today.

4.3 The Embedding matrix: a learnable lookup table

In a Transformer, Word Embedding is a learnable lookup matrix $W_{emb}$:

$$W_{emb} \in \mathbb{R}^{V \times d_{model}}$$

Given a Token ID, take that row and get a $d_{model}$-dimensional vector:

Token ID = 1234  →  W_emb[1234, :]  →  [0.12, -0.34, 0.07, ..., 0.91]  (d_model dims)

That is equivalent to a linear layer with a one-hot input, but implemented as an index lookup, which is far cheaper than a real matrix multiply.

Embedding-layer parameter count: $V \times d_{model}$

For LLaMA-2-7B: $32000 \times 4096 \approx 131M$ parameters. That sounds like a lot, but it is only about 2% of the 6.7B total — the rest lives in the 32 Decoder Blocks.

---

5. Missing position: why Positional Encoding is still needed

Word Embedding solves semantic encoding, but leaves a critical gap: it carries no position information.

Shuffle the three tokens of “猫追狗” into “狗猫追” and the Embedding vectors do not change — the model cannot tell order apart. Meaning depends heavily on word order: “猫追狗” and “狗追猫” mean opposite things.

That is why the Transformer input layer also adds Positional Encoding:

$$\text{Input}_{i} = W\_{emb}[\text{token}_i] + \text{PE}(\text{pos}_i)$$

Word Embedding supplies semantics; Positional Encoding supplies position; add them and you have the full Transformer input. The detailed designs (Sinusoidal and RoPE) are covered in 3.6 A deeper look at positional encoding.

---

6. From Token ID to input matrix: the Transformer input layer

Put the pipeline together with a full example:

User input: "猫在桌子上"

Step 1: Tokenization
────────────────────────────────────────────────────
"猫在桌子上" → Tokenizer → [3421, 528, 11082, 319] (4 Token IDs)
                             猫    在   桌子    上

Step 2: Embedding Lookup
────────────────────────────────────────────────────
Token 3421 (猫)  → W_emb[3421]  → e_1: (d_model,)
Token 528  (在)  → W_emb[528]   → e_2: (d_model,)
Token 11082(桌子)→ W_emb[11082] → e_3: (d_model,)
Token 319  (上)  → W_emb[319]   → e_4: (d_model,)

Stack into input matrix X: (4, d_model)

Step 3: Add Positional Encoding
────────────────────────────────────────────────────
X[0] = e_1 + PE(0)   ← "猫" at position 0
X[1] = e_2 + PE(1)   ← "在" at position 1
X[2] = e_3 + PE(2)   ← "桌子" at position 2
X[3] = e_4 + PE(3)   ← "上" at position 3

Final input matrix X: (4, d_model)  ← into Decoder Block layer 1

In the end, 4 tokens become a $(4, d_{model})$ matrix. Each row is one token’s vector (semantics + position). That matrix is the Transformer Block input.

AI Infra view: The Embedding matrix $W_{emb}$ is sometimes weight-tied with the output LM Head (Weight Tying), which saves $V \times d_{model}$ of memory. GPT-2 does Weight Tying ($V=50257$, $d_{model}=768$, one shared ~38.6M matrix); LLaMA-2 does not (tie_word_embeddings: false in the HuggingFace config), so Embedding and LM Head are separate weights totaling $2 \times 32000 \times 4096 \approx 262M$. When quantizing, the Embedding layer is usually left unquantized (it is a lookup; quantization gains little and can hurt accuracy a lot).

---

🎯 Self-check list

• Explain the three Tokenization granularities (character, word, subword) and why modern large models choose subword
• Explain the two main one-hot failures: dimension explosion and lost semantics
• State the geometric intuition of Word Embedding: similar words sit close in Embedding space
• Explain the geometry behind analogies like king − man + woman ≈ queen
• Write the Embedding parameter formula ($V \times d_{model}$) and check it on LLaMA-2-7B
• Explain why Word Embedding still needs Positional Encoding (missing position)
• Describe the full flow: text → Token ID → Embedding → add positional encoding → input matrix

📚 References

• Attention Is All You Need — original Transformer paper; input-layer design in §3.4
• Neural Machine Translation of Rare Words with Subword Units — original BPE paper (Sennrich et al., 2016)
• Efficient Estimation of Word Representations in Vector Space — original Word2Vec paper (Mikolov et al., 2013), foundational Word Embedding work
• LLaMA 2: Open Foundation and Fine-Tuned Chat Models — LLaMA-2 details, including Tokenizer and Embedding design
• tiktoken — OpenAI’s open-source BPE library used by the GPT series