Positional Information: Absolute Embeddings and RoPE

A transformer has no innate sense of order. Token embeddings are position-independent: the same token ID always maps to the same vector, regardless of where it appears in the sequence. To give the model positional information, we must add it explicitly. This lesson compares two approaches — learned absolute position embeddings (GPT-2 style) and rotary position embeddings / RoPE (Llama 2 style) — traces their tensor shapes, and verifies how each behaves when evaluated at positions never seen during training.

The one-sentence version

Learned absolute positions add a per-position vector to each token embedding (a table lookup that fails beyond the table size), while RoPE rotates Q/K vector pairs by a position-dependent angle (a continuous formula that generalizes to any position).

Everything else in this lesson makes that concrete, with shapes tracked at every step.

Lab: Run the companion CPU exercise at /labs/track1/02-tiny-decoder/ (position embeddings, RoPE, extrapolation test).

A consistent shape trace

Use these dimensions throughout the lesson:

Symbol	Meaning	Value in this trace
B	Batch size (sequences processed together)	2
T	Sequence length (tokens per sequence)	5
h	Number of attention heads	4
d_head	Width per head (d / h)	8
d	Model width (h × d_head)	32
V	Vocabulary size	100

The numbers are deliberately small. A real model uses the same pattern with larger values; the pattern is what matters.

1. Learned Absolute Position Embeddings (GPT-2 style)

1.1 Mechanism

A second embedding table P of shape [max_position, d] is learned alongside the token embedding table E. At position t, row P[t] is added to the token embedding:

embedding[t] = E[token_id[t]] + P[t]   # broadcast over batch

Shape: [B, T, d] + [T, d] → [B, T, d] (broadcast over batch dimension B).

1.2 Forward pass

From GPT-2 source (src/model.py, lines 238–239):

wte = tf.get_variable('wte', [hparams.n_vocab, hparams.n_embd])  # [V, d]
wpe = tf.get_variable('wpe', [hparams.n_ctx, hparams.n_embd])    # [max_position, d]
h = tf.gather(wte, X) + tf.gather(wpe, positions)                # [B, T, d]

1.3 Worked example (B=2, T=5, d=32)

Token embedding lookup:  E[token_ids]  →  [2, 5, 32]
Position table lookup:   P[positions]  →  [2, 5, 32]  (broadcast over batch)
Element-wise addition:   E[token_ids] + P[positions]  →  [2, 5, 32]

Position t directly indexes row P[t] — an O(1) table lookup. Position 5 and position 50 have completely unrelated vectors; the model must learn each position’s representation independently.

1.4 Key properties

Property	Detail
Shape	[B, T, d] preserved; position info added via element-wise addition
Learnable	P is a learned parameter table of size max_position × d
Position lookup	Direct table index P[t] — O(1)
Absolute	Position 5 and position 50 have unrelated vectors
Extrapolation limit	Cannot represent positions beyond max_position
Parameter cost	max_position × d additional parameters

1.5 Failure mode: position overflow

If a sequence is longer than max_position, tf.gather(wpe, positions) raises an index-out-of-range error. The model cannot process positions it has never seen during training.

---

2. Rotary Position Embeddings (RoPE)

2.1 Mechanism

RoPE encodes absolute position m by rotating each pair of dimensions in Q and K by a position-dependent angle m · θᵢ, where θᵢ = 10000^(-2i/d_head) for dimension pair i. The rotation is applied inside the attention block to Q and K vectors only — V is not rotated.

For a 2D vector pair (x, y) rotated by angle φ:

x' = x·cos(φ) − y·sin(φ)
y' = x·sin(φ) + y·cos(φ)

2.2 Forward pass

From Llama 2 source (llama/model.py, precompute_freqs_cis and apply_rotary_emb):

# Precompute frequencies (once, at model init)
theta_i = 10000 ** (-2 * i / d_head) for i in 0..(d_head/2 - 1)  # [d_head//2]
freqs[m, i] = m · theta_i                                         # [max_seq_len, d_head//2]
freqs_cis = cos(freqs) + i·sin(freqs)                              # complex64

# Apply to Q and K (inside attention forward)
xq_ = view_as_complex(xq.reshape(*xq.shape[:-1], -1, 2))  # [B, T, h, d_head//2] complex
xk_ = view_as_complex(xk.reshape(*xk.shape[:-1], -1, 2))
freqs_cis = reshape_for_broadcast(freqs_cis, xq_)          # [1, T, 1, d_head//2]
xq_out = view_as_real(xq_ * freqs_cis).flatten(3)          # [B, T, h, d_head]
xk_out = view_as_real(xk_ * freqs_cis).flatten(3)          # [B, T, h, d_head]

2.3 Worked example (B=2, T=5, h=4, d_head=8)

For d_head=8, there are 4 dimension pairs with base frequencies:

Pair i	θᵢ = 10000^(-2i/8)	Rotation angle at position m
0	1.0	m × 1.0 rad
1	0.1	m × 0.1 rad
2	0.01	m × 0.01 rad
3	0.001	m × 0.001 rad

At position m=3, the first dimension pair rotates by 3 × 1.0 = 3.0 radians; at m=50, it rotates by 50 × 1.0 = 50 radians.

Q = torch.randn(B, T, h, d_head)  # [2, 5, 4, 8]
K = torch.randn(B, T, h, d_head)
V = torch.randn(B, T, h, d_head)

freqs_cis = precompute_freqs_cis(d_head, 10)   # [10, 4]
freqs_cis_T = freqs_cis[:T]                    # [5, 4]
Q_rope, K_rope = apply_rotary_emb(Q, K, freqs_cis_T)  # [2, 5, 4, 8] each

Shapes after RoPE:

• Q_rope: [2, 5, 4, 8] (unchanged)
• K_rope: [2, 5, 4, 8] (unchanged)
• V: [2, 5, 4, 8] (not rotated — bypasses RoPE entirely)

2.4 Key properties

Property	Detail
Shape	[B, T, h, d_head] preserved for Q and K; V unchanged
No learned table	Frequencies from fixed formula θᵢ = 10000^(-2i/d_head)
Position-dependent	Angle at position m is m · θᵢ per dimension pair i
Relative position	Dot product Q_rot[m] · K_rot[n] depends on relative offset m − n
Extrapolation	Works at positions never seen during training
Norm preservation	Rotation preserves vector norm (verified: max diff < 1e-5)
Parameter cost	Zero additional parameters
Applied to Q/K only	V is not rotated

---

3. Position-Offset Test (Verified)

The critical difference between the two methods appears when the model is evaluated at positions beyond its training range.

3.1 Test setup

Train with max_position = 10, then evaluate at position 100.

import torch
import math

torch.manual_seed(123)
torch.set_default_dtype(torch.float32)

d_head = 8
B, T, h = 2, 5, 4

# --- Learned absolute positions (table size 10) ---
pos_table = torch.randn(10, d_head)  # only positions 0..9
positions_100 = torch.tensor([[100]])  # position 100
# pos_table[100] → IndexError: index 100 is out of bounds for dimension 0 with size 10

# --- RoPE (continuous formula, no table) ---
theta_i = 1.0 / (10000.0 ** (torch.arange(0, d_head, 2)[:4].float() / d_head))
# theta_i = [1.0, 0.1, 0.01, 0.001]
angle_at_100 = 100 * theta_i  # [100.0, 10.0, 1.0, 0.1] radians
# Rotation at position 100 is well-defined; no table lookup needed

3.2 Results

Method	Position 5 (in range)	Position 100 (out of range)	Reason
Learned absolute	Works (table lookup P[5])	IndexError (P[100] out of bounds)	Fixed-size table
RoPE	Works (angle 5 · θᵢ)	Works (angle 100 · θᵢ)	Continuous formula

3.3 Norm preservation (verified)

Rotation preserves vector norm. For Q at [B, T, h, d_head]:

||Q_rot||₂ ≈ ||Q||₂   (max diff < 1e-5)

This means RoPE does not change the magnitude of Q/K vectors — only their orientation. The attention computation remains numerically stable.

---

4. Comparison: Learned Absolute vs RoPE

Dimension	Learned Absolute	RoPE
Mechanism	Add learned vector P[t] to token embedding	Rotate Q/K pairs by m · θᵢ per dimension pair
Where applied	Input embedding layer (before transformer blocks)	Inside attention block, on Q and K only
Parameters	max_position × d (learned)	0 (fixed formula)
Position info	Absolute: each position has independent vector	Relative: angle difference depends on m − n
Extrapolation	Fails beyond max_position	Works at any position
V affected?	No	No (Q/K only)
Shape change	None: [B, T, d] → [B, T, d]	None: [B, T, h, d_head] → [B, T, h, d_head]

---

5. Staff/Principal Engineering Judgment

Question	Recommendation	Rationale
Which positional method for a decoder-only generation model?	RoPE	Extrapolates to longer sequences at inference; zero parameters; relative-position property matches the causal LM objective.
When is learned absolute acceptable?	Fixed max length (e.g., classification, fixed-input tasks)	Simple to implement; each position learns its own representation; but fails if sequence exceeds max_position.
Does RoPE need KV-cache changes?	No	RoPE is applied to Q/K before caching; cached keys already carry position info.
Should V be rotated?	No	Values carry content, not position; rotating V would mix position into content vectors unnecessarily.
What if context length doubles at inference?	RoPE handles it; learned absolute does not	RoPE’s continuous angle formula works at any position; learned absolute requires retraining with a larger table.

Staff/Principal judgment: For any decoder-only model that may process sequences longer than its training length — which includes virtually all generation models — RoPE is the default choice. The zero-parameter overhead, relative-position property, and natural extrapolation make it the standard in Llama, Mistral, GPT-NeoX, and most modern decoder-only architectures. Learned absolute embeddings are appropriate only when the maximum sequence length is known, fixed, and small enough that the max_position × d parameter cost is acceptable.

---

6. Visual and text equivalent

Positional Information: Learned Absolute vs RoPE

Where positional information enters: learned absolute embeddings add a looked-up vector; RoPE rotates query and key pairs. A position beyond the learned table is invalid, while RoPE can compute its angle there; model quality still needs testing.

Read this diagram as text

Text equivalent. Panel A: A token ID selects a learned token embedding E[token], and position p selects a learned position embedding P[p]. Adding the vectors supplies the block input. With a 10-row position table, P[50] is out of range. Panel B: Position p determines an angle used to rotate query and key feature pairs. Values are not rotated. At position 50, the angle can be computed without a learned table, but model quality at that position requires evaluation.

Text equivalent: The diagram has two panels. Panel A shows a token ID selecting E[token] and position p selecting P[p]; their vectors are added before the transformer blocks. For a position table with 10 rows, P[50] is out of range. Panel B shows position p supplying an angle for rotation of Q and K feature pairs, while V remains unchanged. At position 50, RoPE can calculate the angle without a learned position table. Whether the model produces useful results beyond its training range must be measured.

---

7. Exercises

Exercise	Prompt
1	Given B=2, T=5, h=4, d_head=8, d=32: how many parameters does a learned absolute position table have if max_position=2048? How many for RoPE?
2	If the model is trained with max_position=2048 but must process a sequence of length 4096 at inference, what happens with learned absolute positions? With RoPE?
3	For d_head=8, what are the four theta_i values? What is the rotation angle at position m=3 for the first dimension pair?
4	After applying RoPE, what are the shapes of Q, K, and V? Does V change?

Try it yourself

1. 01Given B=2, T=5, h=4, d_head=8, d=32: how many parameters does a learned absolute position table have if max_position=2048? How many for RoPE?

   Show a hint

   Learned: max_position × d = 2048 × 32. RoPE: zero parameters (fixed formula).

2. 02If the model is trained with max_position=2048 but must process length 4096 at inference, what happens with learned absolute positions? With RoPE?

   Show a hint

   Learned absolute: table lookup at position ≥ max_position raises IndexError. RoPE: continuous angle formula works at any position.

3. 03For d_head=8, what are the four theta_i values? What is the rotation angle at position m=3 for the first dimension pair?

   Show a hint

   theta_i = 10000^(-2i/8) for i=0,1,2,3. First pair: angle = m × theta_0 = 3 × 1.0.

4. 04After applying RoPE, what are the shapes of Q, K, and V? Does V change?

   Show a hint

   RoPE is applied to Q and K only. V is not rotated.

---

8. Sources

Source	Relevance
Vaswani et al., “Attention Is All You Need” (arXiv:1706.03762), §3.2.2	Learned absolute position embeddings added to input embeddings; sinusoidal position encoding formulas
Su et al., “RoFormer: Enhanced Transformer with Rotary Position Embedding” (arXiv:2104.09864)	RoPE: rotation matrix for absolute position + relative position dependency; extrapolation to longer sequences
Llama 2 source llama/model.py (precompute_freqs_cis, apply_rotary_emb)	Reference RoPE implementation: theta_i = 10000^(-2i/d_head), complex-number rotation of Q/K pairs
GPT-2 source src/model.py (lines 238–239)	Learned position embeddings: wpe table, tf.gather(wte, X) + tf.gather(wpe, positions)
Internal verification (Python, torch 2.5, CPU, float32, seed 123)	RoPE shape preservation, norm preservation (max diff < 1e-5), position-dependent angles, extrapolation to position 100

---