Tokenisation and Embeddings: From Text to Tensors

A neural network cannot read strings like "The capital of France is". Computers perform matrix multiplications on floating-point tensors. In Transformer Architecture Explained, we began with an integer tensor input_ids of shape [B, T], passed it through embedding tables and transformer blocks, and projected the final hidden state into vocabulary logits.

That leaves two foundational questions unanswered:

1. Where did those input numbers come from? How does a raw text string turn into clean integer IDs, and why does an embedding table turn an integer into a vector?
2. How does a hidden vector physically become vocabulary scores? When the final layer finishes, what exact arithmetic maps a hidden vector of width d back to thousands of token scores?

Before diving into byte-level algorithms and matrix layouts, we must establish four clear boundaries:

• Tokenisation: The deterministic process of splitting a raw text string into discrete subword chunks called tokens.
• Vocabulary IDs: Arbitrary integer identifiers (0 through V - 1) assigned to each unique token in a model’s dictionary. An ID is purely an index in a lookup table; calculating id_1 + id_2 has no mathematical meaning.
• Static Embedding Lookup: Retrieving a fixed, learned row vector E[id] of length d for each token ID. In this table, the token "bank" receives the exact same vector whether the text discusses a river bank or a commercial bank.
• Contextual Representations: The dynamic vectors that emerge after passing through the transformer blocks. Through self-attention and MLP layers, the static vector for "bank" absorbs clues from surrounding tokens like "river" or "deposit", creating a context-rich representation.

In this lesson, you will trace a raw string through tokenisation, embedding lookup, and the vocabulary output projection:

• We will work through an educational 6-token toy tokenizer with explicit rules checkable by hand.
• We will verify the exact token IDs against real GPT-2 byte-level BPE using Python.
• We will unpack the output projection, distinguish mathematical notation ($h W$) from PyTorch’s stored weight layout (weight.T), and calculate vocabulary logits by hand.

The one-sentence version

Raw text → regex-based BPE split → token IDs → embedding lookup → [B, T, d] matrix → positional addition → decoder stack → [B, T, V] logits (via tied or untied output projection).

Everything else in this lesson makes that sentence concrete.

Lab: Run the companion CPU exercise at /labs/track1/02-tiny-decoder/ (tokenisation, embedding lookup, tied/untied projection).

The 6-token educational toy: complete pipeline preview

To understand every matrix multiplication without getting buried in 50,000 vocabulary entries, we begin with an explicit 6-token educational tokenizer.

Its rules are defined as follows:

1. Vocabulary ($V=6$):
   • 0: <pad> (special padding token)
   • 1: a (single letter base token)
   • 2: b (single letter base token)
   • 3: ab (merged token from pair ('a', 'b'), merge rank 0)
   • 4: b (merged token from leading space + b, merge rank 1)
   • 5: space (isolated base space token)
2. Regex split rule: Leading whitespace is greedily absorbed into the following letter ( ?\p{L}+).
3. Merge priority: BPE merges apply in strict rank order (rank 0 before rank 1).
4. Embedding table $E$ ([6, 4]):

   E = [
       [0.0, 0.0, 0.0, 0.0],  # 0: <pad>
       [1.0, 0.0, 0.0, 0.0],  # 1: a
       [0.0, 1.0, 0.0, 0.0],  # 2: b
       [0.5, 0.5, 0.0, 0.0],  # 3: ab
       [0.0, 0.0, 1.0, 0.0],  # 4:  b
       [0.0, 0.0, 0.0, 1.0],  # 5: space
   ]

5. **Learned positional vectors ($P$):** $P[0] = [0.1, 0.1, 0.1, 0.1]$, $P[1] = [0.2, 0.2, 0.2, 0.2]$.
6. **Untied output projection matrix $W$ (`[4, 6]`):**
   ```python
   W = [
       [0.1, 0.0, 0.0, 0.0, 0.0, 0.0],
       [0.0, 0.1, 0.0, 0.0, 0.0, 0.0],
       [0.0, 0.0, 0.1, 0.0, 0.0, 0.0],
       [0.0, 0.0, 0.0, 0.1, 0.0, 0.0],
   ]

Trace prompt "a b" step-by-step:

• Step 0 — Raw Input: "a b"
• Step 1 — Regex Split: The regex splits "a b" into ["a", " b"]. The leading space before "b" is absorbed into the second piece.
• Step 2 — BPE Merge Application:
  • Piece "a" is a single base character → Token ID 1.
  • Piece " b" matches merge rank 1 (' ', 'b') → Token ID 4.
  • Output input_ids: [1, 4] (shape [B=1, T=2]).
• Step 3 — Embedding Lookup & Positional Addition:
  • Lookup row E[1]: [1.0, 0.0, 0.0, 0.0]
  • Lookup row E[4]: [0.0, 0.0, 1.0, 0.0]
  • Add position vector P[0] to token 0: [1.0, 0.0, 0.0, 0.0] + [0.1, 0.1, 0.1, 0.1] = [1.1, 0.1, 0.1, 0.1].
  • Add position vector P[1] to token 1: [0.0, 0.0, 1.0, 0.0] + [0.2, 0.2, 0.2, 0.2] = [0.2, 0.2, 1.2, 0.2].
  • Combined sequence tensor shape: [1, 2, 4].
• Step 4 — Output Projection at Position 0: Suppose the transformer passes this tensor through its blocks, and position 0 has hidden state $h = [1.1, 0.1, 0.1, 0.1]$:
  • Tied projection (h @ E^T): Reuses embedding table $E$. Multiplying $h$ by $E^T$ produces vocabulary logits: [0.0, 1.1, 0.1, 0.6, 0.1, 0.1] (shape [1, 6]). Top candidate is token ID 1 (a) with logit 1.1.
  • Untied projection (h @ W): Uses independent matrix $W$. Multiplying $h$ by $W$ produces vocabulary logits: [0.11, 0.01, 0.01, 0.01, 0.0, 0.0] (shape [1, 6]). Top candidate is token ID 0 (<pad>) with logit 0.11.
  • Note: Neither vector is direct text. Logits are unnormalized affinity scores. A decoding policy (greedy argmax or top-p/temperature sampling) selects the observed output token.

---

Verifying against production: Real GPT-2 Byte-Level BPE

Our toy example uses 6 tokens so the numbers fit on a single line. In production, GPT-2 uses byte-level Byte-Pair Encoding (BPE) with a vocabulary of $V = 50,257$.

You can verify the exact behavior in Python in two lines:

import tiktoken

enc = tiktoken.get_encoding("gpt2")

# Verify prompt "a b":
ids_ab = enc.encode("a b")
print("IDs for 'a b':", ids_ab)
# -> [64, 275]
print("Tokens:", [enc.decode([t]) for t in ids_ab])
# -> ['a', ' b']

# Verify prompt "ab c":
ids_abc = enc.encode("ab c")
print("IDs for 'ab c':", ids_abc)
# -> [397, 269]
print("Tokens:", [enc.decode([t]) for t in ids_abc])
# -> ['ab', ' c']

# Verify prompt "The capital of France is":
ids_prompt = enc.encode("The capital of France is")
print("Prompt IDs:", ids_prompt)
# -> [464, 3139, 286, 4881, 318]

Notice the exact parallel with our toy model:

• In GPT-2, 'a' is ID 64 and ' b' is ID 275.
• In prompt "ab c", 'ab' is merged into ID 397, and the leading space before 'c' is absorbed into ' c' (ID 269).
• The raw text "ab c" produces 2 token IDs, not 4 characters or 3 words!

Why byte-level BPE?

Older tokenizers (such as word-level tokenizers) suffered from the Out-Of-Vocabulary (OOV) problem: any word not seen during training became an unknown token <unk>, destroying information.

Byte-level BPE solves this by starting with all 256 possible bytes (0x00 through 0xFF) as the base vocabulary:

1. Every string is ultimately a sequence of UTF-8 bytes.
2. Because all 256 bytes exist in the vocabulary, the tokenizer can represent any string in any language, including emojis and code, without ever emitting <unk>.
3. Frequent byte sequences (like common words and subwords) are merged into single tokens to keep sequence lengths manageable.

A consistent shape trace: realistic toy setup

For tracing the byte-level explanation and the architecture diagram in Section 5, we use a consistent, realistic toy setup ($V = 258, d = 64$):

Symbol	Meaning	Value in this trace
B	Batch size (sequences processed together)	1
T	Sequence length (tokens per sequence)	2
d	Model width (hidden dimension)	64
V	Vocabulary size (256 bytes + 2 merges)	258

Every token ID in this trace is $< V$. GPT-2 small follows this exact pattern with $V = 50,257$ and $d = 768$.

1. Text → BPE token IDs (regex split)

GPT-2 uses byte-level BPE. The encode method in encoder.py runs the regex on raw text first, splitting it into pieces. Then each piece’s UTF-8 bytes are mapped to Unicode code points via bytes_to_unicode(), and finally BPE merges are applied. The regex pattern is defined at encoder.py line 48:

r"""'s|'t|'re|'ve|'m|'ll|'d| ?\p{L}+| ?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+"""

Crucial regex behaviour: space absorption. The ? prefix on the word-pattern alternatives is greedy: if a space precedes a sequence of letters (\p{L}) or numbers (\p{N}), the space is absorbed into that token. A space is its own token only when the letter/digit alternatives cannot take it: when it is followed by another space, by end of string, or by a character the other alternatives do not cover. That is the \s+(?!\S) / \s+ branch. Trailing whitespace, for example, becomes a standalone token.

Input	Regex split	regex pieces	Behaviour
"abcd"	["abcd"]	1	No space; \p{L}+ matches all letters
"ab c"	["ab", " c"]	2	Space absorbed into c via ?\p{L}+
"a b c"	["a", " b", " c"]	3	Each leading space absorbed into following word
"hello world"	["hello", " world"]	2	Space absorbed into world
" "	[" "]	1	Isolated space is its own token
""	[]	0	Empty string

Worked example (GPT-2 style, byte-level, simplified): Prompt "ab c" is a raw UTF-8 string. The GPT-2 regex runs on the text first and splits into ["ab", " c"]. Then for each regex token, GPT-2 byte-encodes it to UTF-8, maps bytes to unicode via bytes_to_unicode, and applies BPE merges. The first regex token ab byte-encodes to [0x61, 0x62]; merge rank 0 ('a','b') applies → token ab (id 256). The second regex token c byte-encodes to [0x20, 0x63]; merge rank 1 (' ','c') applies → token c (id 257). Result: 2 token IDs, not 3.

Toy example (simplified GPT-2 byte-level BPE):

• Byte mappings: a → 0x61, b → 0x62, space → 0x20, c → 0x63
• BPE merge rules (rank order, lowest first):
  • rank 0: ('a', 'b') → token ab, id 256
  • rank 1: (' ', 'c') → token c, id 257
• Vocabulary: 256 single-byte ids (0–255) plus the two merge ids = V = 258
• Prompt: "ab c" (four bytes: a, b, space, c)

Step 1 — Regex split (on raw text):

• re.findall(pat, "ab c") → ["ab", " c"]
• The regex operates on the raw UTF-8 string, not on bytes. The first piece ab is pure letters: ? matches 0 spaces, \p{L}+ matches ab.
• The second regex token c has a leading space: the greedy ? matches 1 space, then \p{L}+ matches c, producing the combined piece c.
• Regex split result: ["ab", " c"] — 2 pieces.

Step 2 — Byte-encode + bytes_to_unicode per regex token:

• Regex token "ab" → UTF-8 bytes [0x61, 0x62] → bytes_to_unicode → unicode "ab"
• Regex token " c" → UTF-8 bytes [0x20, 0x63] → bytes_to_unicode → unicode "Ġc" (where Ġ = space byte)

Step 3 — BPE merge application on the unicode strings (inference):

• For the unicode string "ab": merge rank 0 ('a', 'b') applies → token ab. No further merges. BPE output: ab → id 256.
• For the unicode string "Ġc": merge rank 0 does not match, rank 1 ('Ġ', 'c') applies (where Ġ = space byte). BPE output: Ġc → id 257.
• Token IDs: [256, 257]. In this toy, ids 0-255 are the single bytes and 256, 257 are the merges. Real GPT-2 also gives every byte an id in 0-255, but in bytes_to_unicode order, so its base ids are not byte values (a is 64, the space byte is 220). These are toy ids: in the real GPT-2 vocabulary id 256 is Ġt and 257 is Ġa, so never copy ids across vocabularies. Without the rank 1 merge, " c" would be two token IDs (space and c), giving [256, <space_id>, <c_id>] and T = 3. That is the difference between a regex piece and a BPE token: the regex decides the substring, the merge list decides how many ids that substring produces.

Key point: The regex split happens on raw text before byte-encoding and BPE. Each regex piece is independently BPE-merged. The split determines which substrings BPE operates on. The greedy ? means that in "ab c", the space is not a separate token — it is absorbed into the c token, so BPE only ever sees two substrings: ab and c. Then each substring is independently byte-encoded (UTF-8 → bytes_to_unicode) and BPE-merged. The regex split and the BPE merge list are separate stages: the split decides the substrings, the merge list decides how many token IDs each substring produces.

1.1 Regex split checklist for readers

• Does the input contain a space followed by a letter/digit? If yes, the space is absorbed into the following token ( ?\p{L}+ or ?\p{N}+).
• Does the space precede punctuation or symbols? If yes, the greedy ?[^\s\p{L}\p{N}]+ also absorbs that single leading space into the punctuation token. A space is its own token only when followed by another space or end of string via \s+(?!\S) or \s+.
• Does the input have no spaces? If yes, the whole string is one \p{L}+ or \p{N}+ token.
• After the regex split, BPE merges are applied to each substring independently, in rank order.

2. Token IDs → embedding vectors

The embedding table E has shape [V, d]. For each token ID i in the sequence, the model looks up row E[i], producing a d-dimensional vector. These vectors are stacked to form the token embedding matrix of shape [B, T, d] (batch × sequence × model width).

• GPT-2 source: model.py line 154 — wte = tf.get_variable('wte', [hparams.n_vocab, hparams.n_embd], ...); the lookup h = tf.gather(wte, X) + tf.gather(wpe, ...) is at line 157, which also adds the position table.
• Llama 2 source: model.py line 437 — self.tok_embeddings = ParallelEmbedding(params.vocab_size, params.dim, ...); h = self.tok_embeddings(tokens)

Toy example: Our token IDs are [256, 257] (2 tokens). With B = 1, T = 2, d = 64, and V = 258, both ids are valid rows of E:

• Embedding lookup: E[256] → vector of length 64; E[257] → vector of length 64.
• Stacked output: [1, 2, 64] — two vectors of width 64, one for each token position.

If we add learned positional embeddings (GPT-2 style), a second table P of shape [max_position, d] provides row P[t] for position t. The positional addition is:

embedding_matrix[t] = E[token_id[t]] + P[t]   # broadcast over batch

Result: still [1, 2, 64], now with positional information baked in.

3. Hidden states → vocabulary logits: tied vs. untied

After the transformer stack processes the sequence, the hidden state at position t has shape [B, d]; for the entire batch and sequence, the tensor is [B, T, d]. The output projection (also known as the un-embedding layer or lm_head) maps this representation into vocabulary scores.

How a hidden vector becomes vocabulary scores: the dot product intuition

Before looking at matrices, understand the physical operation: Suppose the final hidden state for a single token position is a vector h of width d. The model must output V numbers—one logit score for every candidate word in its vocabulary.

To do this, the model assigns a d-dimensional prototype vector w_v to each vocabulary token v (from index 0 to V - 1). The logit for token v is simply the dot product between the context vector h and the token’s prototype vector w_v:

logit_v = h · w_v = sum_{k=0}^{d-1} (h[k] × w_v[k])

• If h aligns closely with w_v (pointing in the same directional subspace), their dot product is large and positive: the model strongly favors token v as the next word.
• If h is orthogonal to w_v, the logit is near zero.
• If h opposes w_v, the logit is negative.

Matrix notation versus framework stored layout

Engineers frequently encounter confusion when translating between theoretical papers and PyTorch source code. Here is how the dimensions map:

1. In mathematical papers: The hidden state is written as a row vector h with shape [1, d]. The projection matrix is written as W with shape [d, V], where column v is the prototype vector for token v. The forward pass is:

   logits = h @ W    # [1, d] × [d, V] = [1, V]

2. In PyTorch (nn.Linear): PyTorch defines nn.Linear(in_features=d, out_features=V, bias=False). To optimize BLAS operations and memory cache lines, PyTorch stores the underlying parameter tensor linear.weight with shape:

   stored weight shape = [out_features, in_features] = [V, d]

   Each row v of linear.weight is the prototype vector for token v. When evaluating output = linear(h), PyTorch executes:

   output = h @ weight.T    # [1, d] × [d, V] = [1, V]

3. Weight tying in code: Notice what happens when weights are tied (W = E.T in math notation): The embedding table E has shape [V, d]. PyTorch’s nn.Linear expects a weight tensor of shape [V, d]. Therefore, tying weights in PyTorch requires no matrix transpose or copying:

   # Tied weights in PyTorch:
   lm_head = nn.Linear(d, V, bias=False)
   lm_head.weight = embedding_table.weight  # Shared pointer, shape [V, d]
   logits = lm_head(h)  # Computes h @ embedding_table.weight.T

   The same physical tensor of floats in GPU memory serves both as the lookup table at the input and the linear projection at the output.

A checkable calculation: tied versus untied

Let’s verify this dot product using position 0’s hidden state h = [1.1, 0.1, 0.1, 0.1] from our 6-token toy setup:

Tied Projection (reuses embedding table E of shape [6, 4]):

Here, row v of E is the prototype vector for token v:

• Token 0 (<pad>): w_0 = [0.0, 0.0, 0.0, 0.0] → 1.1(0) + 0.1(0) + 0.1(0) + 0.1(0) = 0.0
• Token 1 (a): w_1 = [1.0, 0.0, 0.0, 0.0] → 1.1(1.0) + 0.1(0) + 0.1(0) + 0.1(0) = 1.1
• Token 2 (b): w_2 = [0.0, 1.0, 0.0, 0.0] → 1.1(0) + 0.1(1.0) + 0.1(0) + 0.1(0) = 0.1
• Token 3 (ab): w_3 = [0.5, 0.5, 0.0, 0.0] → 1.1(0.5) + 0.1(0.5) + 0 + 0 = 0.55 + 0.05 = 0.6
• Token 4 ( b): w_4 = [0.0, 0.0, 1.0, 0.0] → 1.1(0) + 0.1(0) + 0.1(1.0) + 0.1(0) = 0.1
• Token 5 ( ): w_5 = [0.0, 0.0, 0.0, 1.0] → 1.1(0) + 0.1(0) + 0.1(0) + 0.1(1.0) = 0.1

Tied logits: [0.0, 1.1, 0.1, 0.6, 0.1, 0.1]. The highest-scoring candidate is token 1 (a) with score 1.1.

Untied Projection (uses independent matrix W of shape [4, 6]):

Column v of W is the prototype vector for token v:

• Token 0 (<pad>): Column 0 is [0.1, 0.0, 0.0, 0.0]^T → 1.1(0.1) = 0.11
• Token 1 (a): Column 1 is [0.0, 0.1, 0.0, 0.0]^T → 0.1(0.1) = 0.01
• Token 2 (b): Column 2 is [0.0, 0.0, 0.1, 0.0]^T → 0.1(0.1) = 0.01
• Token 3 (ab): Column 3 is [0.0, 0.0, 0.0, 0.1]^T → 0.1(0.1) = 0.01
• Tokens 4 and 5: Columns 4 and 5 are all zeros → 0.0

Untied logits: [0.11, 0.01, 0.01, 0.01, 0.0, 0.0]. The highest-scoring candidate is token 0 (<pad>) with score 0.11.

The two projections disagree because their weights are completely different. Tied projection constrains token embeddings and output prototypes to live in the same geometric space; untied projection lets the output layer learn specialized decision boundaries.

Larger logits are candidates, not outputs

Both projections return raw scores, [B, T, V]. Turning a score row into text is a separate step: greedy decoding takes argmax, while sampling (temperature, top-k, top-p) converts logits into probabilities using softmax and draws from that distribution. Section 4 walks through this complete exercise.

Engineering judgment (Staff/Principal)

Question	Tied (GPT-2)	Untied (Llama 2)	Staff/Principal takeaway
Parameter count	Saves V × d params. For GPT-2 small (V = 50,257, d = 768) that is 38.6M. Does not halve the model.	Adds V × d params on top of the embedding table.	Tied weights are a compactness technique, not a quality enhancer. The saving is real but modest next to total parameter count.
Expressive power	Output projection is constrained to E^T; it cannot learn a different mapping.	Output projection is a free d × V matrix.	Untied weights give the output layer extra capacity; whether that helps is an empirical question for the target task.
Training dynamics	One weight matrix receives gradient from both the input lookup and the output projection, which couples the two roles.	The two matrices are updated by separate gradient paths.	Tied weights act as weight sharing, which changes optimisation dynamics. Reported gains are model- and data-dependent; verify on your own validation split instead of assuming a regularisation benefit.
Inference cost	Same matmul as untied: h @ E^T is [T, d] × [d, V].	Same matmul: h @ W.	Arithmetic is identical. Tying saves memory (one V × d tensor to store and shard), not FLOPs per token.

Staff/Principal judgment: Tying the output projection to the embedding table is a memory-saving choice, not a quality decision. It saves V × d parameters (the output projection matrix) without changing per-token FLOPs. For a decoder-only model in the 100M–1B parameter range, tying is a reasonable default if you control the full training pipeline, but it is not universally better: untied weights give the output layer independent capacity, which may help when embedding and output roles diverge. The choice should be recorded as an architecture decision with the parameter delta and validation evidence, not treated as a rule of thumb.

4. Practical exercise

Problem. Work through a fully specified 6-entry toy vocabulary. Nothing here is byte-level; the vocabulary is small so every number can be checked by hand. Its base symbols a, b, and space each have an ID before the two merges are applied.

id	token	note
0	<pad>	unused in this exercise
1	a	single character
2	b	single character
3	ab	merge rank 0: ('a','b')
4	b	merge rank 1: (' ','b')
5		base space symbol

Embedding table E ([V, d] = [6, 4]):

E = [[0.0, 0.0, 0.0, 0.0],    # id 0
     [1.0, 0.0, 0.0, 0.0],    # id 1
     [0.0, 1.0, 0.0, 0.0],    # id 2
     [0.5, 0.5, 0.0, 0.0],    # id 3
     [0.0, 0.0, 1.0, 0.0],    # id 4
     [0.0, 0.0, 0.0, 1.0]] # id 5 (space)

Position embeddings: P[0] = [0.1, 0.1, 0.1, 0.1], P[1] = [0.2, 0.2, 0.2, 0.2].

Untied output matrix W ([d, V] = [4, 6]), used only in step 4:

W = [[0.1, 0.0, 0.0, 0.0, 0.0, 0.0],
     [0.0, 0.1, 0.0, 0.0, 0.0, 0.0],
     [0.0, 0.0, 0.1, 0.0, 0.0, 0.0],
     [0.0, 0.0, 0.0, 0.1, 0.0, 0.0]]

Input: "a b".

1. Run the GPT-2 regex split on "a b". How many pieces, and what are they?
2. Apply the merge list to each piece and give the final token IDs.
3. Compute the token-embedding matrix, then add positions. Give both shapes.
4. Take the position-0 hidden state h and compute the tied logits h @ E^T and the untied logits h @ W. State shapes and the top-ranked id in each case. Is either one a sampled output?
5. What-if: the input is "a b " (trailing space). How does the regex split change, and what are the resulting token IDs and sequence length?

Answers.

1. ["a", " b"] — 2 pieces. The first a has no leading space, so ? matches zero spaces. The second regex token starts with a space; the greedy ? consumes it and \p{L}+ matches b, so the space travels with the b.
2. "a" has length 1, so no merge can apply: id 1. " b" is the pair (' ', 'b'); rank 0 is ('a','b') (no match), rank 1 is (' ','b') (match), so it collapses to id 4. Final ids: [1, 4], T = 2.
3. E[[1, 4]] = [[1.0, 0.0, 0.0, 0.0], [0.0, 0.0, 1.0, 0.0]], shape [1, 2, 4]. Adding P[0] and P[1] gives [[1.1, 0.1, 0.1, 0.1], [0.2, 0.2, 1.2, 0.2]], still [1, 2, 4].
4. Position 0 hidden state h = [1.1, 0.1, 0.1, 0.1].
   • Tied: h @ E^T with E^T of shape [4, 6] gives [0.0, 1.1, 0.1, 0.6, 0.1, 0.1], shape [1, 1, 6]. Top-ranked id 1 (a).
   • Untied: h @ W with W of shape [4, 6] gives [0.11, 0.01, 0.01, 0.01, 0.0, 0.0], shape [1, 1, 6]. Top-ranked id 0 (<pad>).
   • Neither vector is an output. A decoder still has to turn scores into a token: greedy decoding takes the argmax, while temperature/top-k/top-p sampling draws from a distribution built from the same logits. The two projections disagree here precisely because they are different matrices.
5. "a b " splits into ["a", " b", " "] — 3 pieces, because the trailing space is followed by end-of-string and only \s+(?!\S) / \s+ can match it. The first two pieces still produce ids 1 and 4. The final piece uses base space id 5, so the result is [1, 4, 5], T = 3. A BPE merge that uses space requires that space to exist as a base symbol even when the merged b is also in the vocabulary.

5. Visual and text equivalent

Tokenisation + Embeddings + Logits

Three panels: text to token IDs, token IDs to embeddings plus position, and hidden state to vocabulary logits.

Read this diagram as text

Text equivalent. Panel A: the four bytes of 'ab c' are split by the GPT-2 regex into two pieces, 'ab' and ' c', because the optional leading space is absorbed into the following letters. Two ranked merges, (a,b) then (space,c), turn those pieces into the toy token IDs 256 and 257, so T equals 2 and the vocabulary has 258 entries. Panel B: ids 256 and 257 select two rows of the embedding table with shape [258, 64]; the rows stack into [1, 2, 64], and the learned position rows P[0] and P[1] are added to give the hidden state [1, 2, 64]. Panel C: the final hidden state [1, 2, 64] is projected to logits [1, 2, 258] either with tied weights, where GPT-2 multiplies by the transpose of the embedding table with no bias, or with untied weights, where Llama 2 multiplies by a separate [64, 258] output matrix with no bias. Tying saves V times d parameters, which is 16,512 here and 38.6M for GPT-2 small, and it does not change the matmul cost.

6. Sources checked

• OpenAI GPT-2 report: language-models.pdf - byte-level BPE description and the 50,257-token vocabulary. The report does not argue for tied weights; that detail comes from the code below.
• GPT-2 tokenizer source: encoder.py - byte-level BPE, the regex on line 48, merge application, and decode.
• GPT-2 model source: model.py - embedding at line 154, tied output projection at line 171 (no bias), learned positional embeddings.
• Llama 2 model source: model.py - separate output matrix at line 446 (bias=False), RoPE, RMSNorm, ParallelEmbedding.
• Hugging Face transformers docs: tie_word_embeddings - the configuration flag that switches between tied and untied heads.
• Checks performed for this lesson (2 November 2026): the regex above was run on "ab c", "a b", and "a b " with Python’s regex module; the GPT-2 vocabulary file gpt2/vocab.json was read to confirm len(vocab) = 50257 and that id 220 decodes to the space byte. Both matched the claims in this lesson.

7. Caveats and misconceptions

• Misconception: “Tokens are just words.” BPE merges can split words into sub-word parts. A single word can yield multiple token IDs.
• Misconception: “All models use the same vocabulary.” Each model has its own BPE-trained vocabulary. GPT-2’s ~50k vocab differs from Llama 2’s ~32k.
• Misconception: “GPT-2 normalises/lowercases text before tokenisation.” It does not. It works on raw bytes via the byte→unicode mapping.
• Misconception: “BPE training and application are the same process.” Training builds the merge list by frequency; inference applies the fixed ranked list.
• Misconception: “Positional embeddings are the only way to add order.” RoPE, ALiBi, and other positional methods exist.
• Misconception: “Tied weights halve the model’s parameters.” Tying saves only V × d params, a fraction of total params.
• Misconception: “Tied weights include a bias term.” GPT-2’s tied output projection has no bias.
• Misconception: “The ## prefix is a BPE convention.” The ## prefix belongs to WordPiece (BERT), not BPE. GPT-2 byte-level BPE uses no such prefix; space is a byte that gets absorbed per the regex rule.
• Tokeniser-dependent caveat: If the input contains characters or words not seen during BPE training, the tokenizer may split them differently than expected. Always use the model’s official tokenizer (e.g., tiktoken for OpenAI models, transformers.AutoTokenizer for Hugging Face).

What comes next: What should the model learn from those scores?

We now have the complete picture of how tokens enter and exit a transformer forward pass:

1. Raw text is split into tokens and mapped to arbitrary integer vocabulary IDs.
2. Embedding table $E$ turns integer IDs into static continuous vectors of width d.
3. Transformer blocks contextualize those vectors across positions with causal attention and MLPs.
4. Output projection computes the dot product between the final hidden vectors and vocabulary prototypes to generate logits.

Now comes the crucial question: how does the model learn the right weights in those embedding tables and projection layers in the first place?

When training an autoregressive model, if we feed the prompt ['A', 'B', 'C', 'D'], which target token belongs to each prediction position? Why must the final position’s prediction be dropped from the loss? And when batches contain sequences of different lengths, how do attention masks and loss masks prevent padding noise from destroying the weights?

That is the subject of the next lesson: Preparing Training Sequences: Shift, Pad, and Pack.

Module exit check: tokenisation and projection arithmetic

1. 01Using the 6-token toy vocabulary (0: <pad>, 1: a, 2: b, 3: ab, 4: b, 5: space), trace the input string "a b ". Write down the regex pieces, final IDs, and sequence length. Explain why base space id 5 is needed even though merged token id 4 contains a space.

   Show a hint

   "a b " splits into ["a", " b", " "]. The first two pieces produce ids 1 and 4; the trailing standalone space produces id 5. Final IDs: [1, 4, 5], T = 3. BPE starts from base symbols before merging, so a merge of (space, b) requires the space base symbol to exist.

2. 02Given hidden state h = [0.8, -0.2, 0.5, 0.0] and tied embedding table E of shape [6, 4] where row 2 is [0.0, 1.0, 0.0, 0.0] (token b) and row 3 is [0.5, 0.5, 0.0, 0.0] (token ab), calculate the exact tied logit scores for tokens 2 and 3 by hand. State the shape of linear.weight if implemented in PyTorch nn.Linear(4, 6, bias=False).

   Show a hint

   For token 2 (b): logit_2 = h · E[2] = (0.8)(0) + (-0.2)(1.0) + (0.5)(0) + (0)(0) = -0.20. For token 3 (ab): logit_3 = h · E[3] = (0.8)(0.5) + (-0.2)(0.5) + (0.5)(0) + (0)(0) = 0.40 - 0.10 = 0.30. In PyTorch, nn.Linear(4, 6) stores weight with shape [out_features, in_features] = [6, 4], matching the [V, d] shape of the embedding table.

3. 03A model architecture proposal uses vocabulary size V = 32,000 and hidden dimension d = 4,096. Calculate the exact parameter savings from tying the output projection to the embedding table, and state whether tying reduces forward inference FLOPs per token.

   Show a hint

   Tying removes the independent output matrix W [d, V], saving exactly V × d = 32,000 × 4,096 = 131,072,000 parameters (~131.1M parameters). Forward FLOPs are unchanged: both tied (h @ E^T) and untied (h @ W) compute identical [T, d] × [d, V] matrix multiplications.

---