# HANDOFF — Lab C: Cached Generation and GQA **Directory:** `labs/track1/03-cache-and-gqa/` **Date:** 2026-11-02 **Lesson coverage:** Lessons 11–15 (Generation and context orders 11–14; Modern architectures order 15) **Status:** All four tests pass. Ready for manager integration review. --- ## Summary Built a deterministic tiny attention model on CPU with cached and uncached generation paths. Verified logit parity, cache growth, MHA/GQA cache memory arithmetic, and a broken cache-position diagnostic with repair. Timing is recorded as optional local data only, never as a guaranteed speedup. --- ## Exact setup | Variable | Value | |----------|-------| | Directory | `labs/track1/03-cache-and-gqa/` | | Python | 3.12.10 | | PyTorch | 2.15.0 (CPU) | | Device | CPU only | | Precision | float32 | | Seed | `torch.manual_seed(42)` | | Model dims | D=32, H=2, D_HEAD=16, L=2, V=50, B=1 | | Prompt length | P=3 | | Decode steps | 5 | | Parity tolerance | 1e-5 (FP32) | Files in the lab: | File | Purpose | |------|---------| | `cache_model.py` | Tiny decoder model + MHA/GQA byte calculator | | `test_logit_parity.py` | Cached vs uncached logit parity over 5 decode steps | | `test_cache_growth.py` | One KV append per layer per step | | `test_mha_gqa_memory.py` | Raw MHA vs GQA cache bytes | | `test_broken_cache_position.py` | Broken cache-position write + repair | | `actual_output.txt` | Recorded output from all tests | | `setup.md` | Exact environment setup | | `README.md` | Lab overview | | `HANDOFF.md` | This file | No MDX, curriculum, shared config, or dependency files were modified. Track 2 is untouched. --- ## Actual test results ### Test 1 — Logit parity (test_logit_parity.py) Prompt: `[7, 3, 9]` (P=3). Five greedy decode steps. Each step compares cached-path logits against an uncached full-sequence reference at the last position. ``` Cache length after prefill (P=3): 3 Running 5 decode steps with cached-vs-uncached parity checks: step cache_len max|diff| parity_ok --------------------------------------------- 1 4 1.788e-07 True 2 5 1.788e-07 True 3 6 1.192e-07 True 4 7 2.384e-07 True 5 8 2.384e-07 True --------------------------------------------- All 5 steps within tolerance 1e-05: True PASS: cached and uncached next-token logits agree for every decode step. ``` **Result:** PASS. All 5 decode steps show max logit difference < 1e-5 (FP32 tolerance). Cache grows from 4 → 8 across the 5 steps (P + step). ### Test 2 — Cache growth (test_cache_growth.py) Verifies exactly one K/V column is appended per layer per decode step. ``` Cache length after prefill (P=3): 3 Per-step cache growth check (4 decode steps): step expected actual per_layer ok ------------------------------------------------- 1 4 4 True True 2 5 5 True True 3 6 6 True True 4 7 7 True True ------------------------------------------------- Final cache length: 7 (expected 7) All layers grow by exactly one column per step: True PASS: one KV append per layer per step; cache shape [B, H_kv, P+S, D_HEAD] confirmed. ``` **Result:** PASS. Every layer's cache grows by exactly one column per step. Cache shape follows `[B, H_kv, P + S, D_HEAD]` = `[1, 2, P + step, 16]`. ### Test 3 — MHA vs GQA cache bytes (test_mha_gqa_memory.py) Per-layer bytes use `2 × H_kv × D_HEAD × S × B × b`; totals multiply by `L` (b=2 for FP16). GQA figures are calculated from the formula with `H_kv=1`; no GQA cache was instantiated or measured. ``` Raw KV-cache bytes from the lab model dimensions L (layers) = 2 H_q (Q heads) = 2 D_HEAD = 16 S (cached tok) = 64 B (batch) = 1 b (bytes/elem) = 2 (FP16) Variant H_kv Per-layer bytes Total (2 layers) Ratio vs MHA ---------------------------------------------------------------------- MHA 2 8,192 16,384 1.00 GQA 1 4,096 8,192 0.50 ---------------------------------------------------------------------- MHA: 8,192 bytes per layer; 16,384 bytes total across 2 layers GQA: 4,096 bytes per layer; 8,192 bytes total across 2 layers GQA figures are calculated from H_kv=1, not measured from an instantiated cache. GQA / MHA ratio: 0.50 (50% reduction) PASS: per-layer and total byte figures are consistent. ``` **Result:** PASS. MHA with `H_kv=2` uses 8,192 bytes per layer and 16,384 bytes across two layers; GQA with `H_kv=1` uses 4,096 bytes per layer and 8,192 bytes across two layers. The GQA figures are calculated, not verified against an instantiated GQA cache. These are raw tensor bytes only — not GPU allocator memory, and not a latency guarantee. ### Test 4 — Broken cache position + repair (test_broken_cache_position.py) **Broken case** (overwrite slot 0 instead of appending at end): ``` step 1: max|cached - uncached| = 1.654e-01 (tolerance 1e-05) -> MISMATCH (expected) step 2: max|cached - uncached| = 2.335e-01 (tolerance 1e-05) -> MISMATCH (expected) step 3: max|cached - uncached| = 2.875e-01 (tolerance 1e-05) -> MISMATCH (expected) All 3 steps show mismatch (expected for broken cache): True ``` **Fixed case** (correct append at write_position): ``` step 1: max|cached - uncached| = 2.384e-07 (tolerance 1e-05) -> OK step 2: max|cached - uncached| = 2.384e-07 (tolerance 1e-05) -> OK step 3: max|cached - uncached| = 2.384e-07 (tolerance 1e-05) -> OK All 3 steps within tolerance: True ``` **Result:** PASS. Broken cache position causes max diff > 1e-1 (far outside tolerance). Fix (write at correct slot `k_cache.shape[2]`) restores parity with diff < 1e-6. --- ## Diagnosis and repair **Bug:** The KV cache write overwrote slot 0 instead of appending at `write_position` (the current cache length). The cache never grew, and the newest token displaced the oldest entry, so subsequent reads attended to the wrong keys/values. **Fix:** Write new K/V at slot `k_cache.shape[2]` — exactly the current cache length, which is the next free position. After the fix, cached and uncached logits agree within FP32 tolerance for all decode steps. --- ## Lesson mapping | Lab component | Lesson | Connection | |---------------|--------|------------| | Logit parity over decode steps | 13 (kv-cache-explained) | Cached vs uncached next-token logits agree | | Cache growth one column per step | 12 (prefill-and-decode), 13 | Prefill fills cache once; decode appends one column | | MHA/GQA cache bytes | 15 (mha-mqa-gqa) | H_kv determines cache size; formula from lesson 13 | | Broken cache position | 13 | Per-layer cache write position must be correct | --- ## Timing statement No timing data is included as a guaranteed result. All tests run on CPU. Any timing observed is optional measured local data only and does not constitute a performance claim or speedup guarantee. --- ## Integration notes - All four tests are independent and can be run individually. - No external dependencies beyond Python 3.12 and PyTorch 2.15.0. - No model checkpoint, GPU, API key, or network access required. - Ready for manager integration: inspect code, run tests, reconcile with lesson claims.