Lab 03 — Cached generation and GQA

Track 1 practice lab

Practice material, not a published lesson. Not counted in the Track 1 chapter count.

Prerequisite lessons

• Pretrained Weights and Text Generation
• Prefill and Decode: Measuring the Two Phases
• KV Cache Implementation and Memory Accounting
• Context Windows and Their Limits
• MHA vs. MQA vs. GQA: Attention Architecture Tradeoffs

Setup

• Python 3.12.10, PyTorch 2.15.0 (CPU), float32, seed 42.
• Run from `labs/track1/03-cache-and-gqa`.
• Cache-byte arithmetic uses a stated FP16 assumption of 2 bytes per element; model runs are float32.

Exercise

Run the broken cache path that overwrites slot 0, observe mismatched logits, then use the correct append at `write_position` and confirm parity.

Verification commands

cd labs/track1/03-cache-and-gqa
python3 test_logit_parity.py
python3 test_cache_growth.py
python3 test_mha_gqa_memory.py
python3 test_broken_cache_position.py

Recorded results

Logit parity passes for 5 decode steps. Cache grows by one column per layer per step. MHA uses 8,192 bytes per layer (16,384 total); GQA uses 4,096 per layer (8,192 total), calculated. Broken cache mismatches all 3 steps; the fix restores parity.

Runnable files

• README.md
• cache_model.py
• test_logit_parity.py
• test_cache_growth.py
• test_mha_gqa_memory.py
• test_broken_cache_position.py
• setup.md
• actual_output.txt
• HANDOFF.md

All practice labs