# Lab B — Tiny Decoder Training (Track 1, Session B)

Companion lab for lessons 1–3 and 7–10 of the LLM Internals track. A minimal CPU
decoder-only language model trained on a single fixed tiny batch to demonstrate
forward shapes, parameter counting, label alignment, loss reduction, and a
deliberate label-shift failure with a diagnostic check and repair.

## Scope

- Build a small GPT-style decoder on CPU (no downloads, no API keys, no GPU).
- Run a fixed tiny-batch overfit and verify loss decreases.
- Assert tensor shapes and parameter count against a hand-computed formula.
- Diagnose a label-shift bug (loss pairs `logits[t]` with `labels[t-1]` instead
  of `labels[t+1]`) and apply the repair.

This lab does **not** make performance or timing claims; timing is not measured.

## Contents

| File | Purpose |
|---|---|
| `model.py` | Tiny decoder: RMSNorm, causal self-attention, SwiGLU MLP, pre-norm decoder block, full model with tied output projection; parameter-count formula |
| `train.py` | Fixed-batch overfit run: correct shift (loss decreases), static label-shift diagnostic, repair (correct shift again) |
| `test_tiny_decoder.py` | pytest suite: forward shape, parameter count, shape preservation, causal safety, correct-shift decrease, static label-shift diagnostic, fix |
| `HANDOFF.md` | Exact setup and actual measured results (this lab's record) |

## Prerequisites

- Python 3.12 (tested with 3.12.10)
- PyTorch 2.15.0 (CPU), float32
- pytest 9.0.2 (for the test suite)

No model checkpoint, tokenizer download, API key, GPU, or internet access is required.

## How to run

From this directory:

```bash
# Run the fixed-batch overfit training (prints setup + results)
python3 train.py

# Run the test suite
python3 -m pytest test_tiny_decoder.py -v
```

`train.py` is self-contained and imports only `model.py`. The test suite imports
`model.py` and `train.py` (the shared `next_token_loss` helper).

## Design notes (traced to lessons)

- **Lesson 1**: decoder-only trace; logits shape `[B, T, V]`; input `[B, T]`.
- **Lesson 2**: token + learned position embeddings summed to `[B, T, d]`.
- **Lesson 3**: `input_ids`/`labels` `[B, T]` with pad→`-100`; loss-side shift
  `logits[:, :-1]` vs `labels[:, 1:]` (shift is in the loss, not the collator).
- **Lesson 8**: pre-norm RMSNorm, SwiGLU MLP, residual stream `[B, T, d]`,
  causal safety (future positions cannot affect earlier outputs).
- **Lesson 9**: stacked decoder blocks + final norm + tied output projection;
  parameter counting (`10d² + 6d` per block).
- **Lesson 10**: next-token cross-entropy, AdamW optimizer, overfit diagnostic
  (loss should decrease on a tiny fixed batch); misaligned-label failure mode.

## Reproducibility

`train.py` and `test_tiny_decoder.py` call `torch.manual_seed(...)` so runs are
deterministic on the same PyTorch version/device. Recorded results in
`HANDOFF.md` were produced with Python 3.12.10, torch 2.15.0, CPU, float32.