Track 1 practice lab
Practice material, not a published lesson. Not counted in the Track 1 chapter count.
Prerequisite lessons
- Transformer Architecture Explained: Trace a Decoder-Only LLM
- Tokenisation and Embeddings: From Text to Tensors
- Preparing Training Sequences: Shift, Pad, and Pack
- Positional Information: Absolute Embeddings and RoPE
- Inside the Transformer Block: Residuals and Normalisation
- Assembling a GPT-Style Language Model
- Training and Debugging: Next-Token Loss and Optimiser Steps
Setup
- Python 3.12.10, PyTorch 2.15.0 (CPU), float32, pytest 9.0.2.
- Run from `labs/track1/02-tiny-decoder`.
- No checkpoint, tokenizer download, API key, GPU, or internet access is required.
Exercise
Compare the correct next-token shift with a label-shifted loss, observe the diagnostic failure, then apply the repair and confirm loss decreases.
Verification commands
cd labs/track1/02-tiny-decoder
python3 train.py
python3 -m pytest test_tiny_decoder.py -v