# Handoff — Lab B: Tiny Decoder Training

Track 1, Session B (lessons 1–3, 7–10). Directory:
`labs/track1/02-tiny-decoder/`. Only this directory was created/edited.

## Exact setup

| Item | Value |
|---|---|
| Directory | `labs/track1/02-tiny-decoder/` |
| Python | 3.12.10 (CPython, darwin) |
| PyTorch | 2.15.0 (CPU build), dtype float32 |
| pytest | 9.0.2 |
| Device | CPU only |
| Seed | `torch.manual_seed(0)` (train.py), `torch.manual_seed(3)` (test helper) |
| Vocabulary `V` | 32 (ids 0–31; id 0 = pad) |
| Model width `d` | 16 |
| Attention heads `n_heads` | 4 (so `d_head = 4`) |
| Decoder blocks `N` | 2 |
| Max position table | 16 (learned absolute position embeddings) |
| Output projection | Tied to embedding table (`W_out = E^T`, 0 extra params, no bias) |
| Normalisation | RMSNorm (gain only, no bias), pre-norm |
| MLP | SwiGLU: `d → 4d → 2d → d`, no bias |
| Attention | Manual scaled dot-product, explicit lower-triangular causal mask |
| Batch `B` | 2 |
| Sequence length `T` | 6 |
| Fixed `input_ids` | `[[1, 2, 3, 0, 0, 0], [4, 5, 6, 7, 0, 0]]` |
| Fixed `labels` | clone of input_ids, pad (0) → `-100` |
| Optimiser | AdamW, `lr = 1e-2` |
| Steps | 300 |
| Loss | `CrossEntropyLoss(ignore_index=-100)` |

No model checkpoint, tokenizer download, API key, GPU, or internet access was
used. All numbers below are actual measured outputs from this setup.

## Files in this directory

| File | Role |
|---|---|
| `model.py` | Tiny decoder + hand-computed parameter-count formula + weight-shape convention note |
| `train.py` | Fixed-batch run: pairing inspection, wrong vs correct loss, repair verification |
| `test_tiny_decoder.py` | 9-pytest suite (all pass) |
| `HANDOFF.md` | this file |
| `README.md` | setup and how to run |

## Weight-shape convention (mathematical vs PyTorch-stored)

`model.py` documents the distinction at the top of the module and on each
`nn.Linear`:

- Mathematical convention (lessons 8–9): a linear map is written `W [d_in, d_out]`
  applied as `x @ W`.
- PyTorch `nn.Linear(d_in, d_out)` stores its weight as `[d_out, d_in]`
  (out_features × in_features) and applies `x @ weight.T`.

Concrete cases in this lab (mathematical → stored):
- Attention Q/K/V/O: `[d, d]` → `[d, d]` (square, visually identical)
- SwiGLU `w`: `[d, 4d]` → `[4d, d]`
- SwiGLU `w_out`: `[2d, d]` → `[d, 2d]`
- Output projection: `[d, V]` → `[V, d]`

The parameter-count formula uses the mathematical convention; `nn.Linear` stores
the transpose but has the same element count, so the count is unaffected.

## Actual results

### Test suite

```
collected 9 items
test_forward_output_shape            PASSED
test_rmsnorm_preserves_shape         PASSED
test_decoder_block_preserves_shape   PASSED
test_swiglu_intermediate_shapes      PASSED
test_parameter_count_matches_formula PASSED
test_causal_safety                   PASSED
test_correct_shift_loss_decreases    PASSED
test_wrong_vs_correct_loss_on_crafted_logits PASSED
test_specific_pairing_inspection     PASSED
9 passed
```

### Forward shape (lesson 1)

```
input_ids shape (2, 6)  ->  logits shape (2, 6, 32)   OK
```

### Parameter count (lesson 9)

Hand-computed formula = `V·d + max_position·d + N·(10d² + 6d) + d + 0` (tied):

| Component | Calculation | Params |
|---|---|---|
| Token embeddings | 32 × 16 | 512 |
| Position embeddings | 16 × 16 | 256 |
| Decoder blocks (×2) | 2 × (10×16² + 6×16) = 2 × 2656 | 5,312 |
| Final norm | 16 | 16 |
| Output projection (tied) | 0 | 0 |
| **Total** | | **6,096** |

```
actual=6,096  expected=6,096  MATCH
```

### Loss reduction — correct vs wrong (measured data, not proof of alignment)

```
wrong loss:   step0=15.5331  step299=0.0005   (misaligned path)
correct loss: step0=11.2957  step299=0.0002   (repaired path)
```

Both decrease on this tiny batch. **A decreasing loss does NOT prove label
alignment** — the wrong path also decreases through memorisation. Alignment is
established by inspecting the pairing, below.

### Label-alignment inspection (the core demonstration)

Fixed batch: `input_ids[0] = [1, 2, 3, 0, 0, 0]`,
`labels[0] = [1, 2, 3, -100, -100, -100]`.

At sliced position 0:

| Shift | Slices | Pairs | Result |
|---|---|---|---|
| Correct | `logits[:, :-1]` vs `labels[:, 1:]` | `logits[0,0]` → `labels[0,1]` = `2` | Next token — **ALIGNED** |
| Wrong | `logits[:, 1:]` vs `labels[:, :-1]` | `logits[0,1]` → `labels[0,0]` = `1` | Previous-position label — **MISALIGNED** |

Crafted-logit static diagnostic (logits[0,0,5]=100, labels[0,1]=5; loss
averages over all positions, so no "near-zero" claim):

| Shift | Measured loss |
|---|---|
| Correct | 2.7726 |
| Wrong | 3.4657 |

The shifts differ on identical inputs/labels, confirming they are different
computations. The correct shift pairs `logits[t]` with `labels[t+1]`.

### Repair verified

The repair is the code change:
`logits[:, 1:] / labels[:, :-1]`  →  `logits[:, :-1] / labels[:, 1:]`.

After the repair, re-inspection confirms `logits[0,0]` → `labels[0,1] = 2`
(next token — aligned). Alignment is established by inspecting the pairing,
not by observing a decreasing loss.

### Causal safety (lesson 8)

Corrupting a future position (index 2) leaves outputs at positions 0 and 1
unchanged (`allclose(atol=1e-5)`), confirming the causal mask prevents
future-token leakage.

## Completion state

All checks pass:
- `python3 train.py` → all assertions pass
- `python3 -m pytest test_tiny_decoder.py -v` → 9 passed

This lab is ready for the manager integration gate: inspect code, run the
documented commands, and reconcile behavior with lesson claims before any
lesson linking or practice-index integration.