Track 01
LiveLLM Internals & Architecture
Implement and inspect a decoder-only transformer, explain how it learns and generates, and evaluate the design choices behind modern and reasoning models.
What you'll be able to do
By the end of the core course.
- Trace a decoder-only transformer from tokens to logits, with explicit tensor shapes
- Implement attention, the transformer block, training, and cached generation
- Explain how model and context choices change memory, latency, and quality
- Evaluate modern attention, sparse, and reasoning designs against stated tradeoffs
Main subjects
The territory this track covers.
- 01Transformer architecture
- 02Tokenisation and embeddings
- 03Attention and masking
- 04Positional encoding
- 05Pretraining and generation
- 06KV cache and context
- 07MQA, GQA, and MLA
- 08Mixture of Experts
- 09Multi-token prediction
- 10Architecture comparison
- 11Reasoning evaluation
- 12Reasoning distillation
Who this is for
Experienced developers learning model internals for the first time, and senior engineers who want to inspect the machinery behind the API.
Before you start
- Comfort reading Python and basic PyTorch
- Familiarity with vectors and matrix multiplication
- No prior machine-learning course required
Curriculum
2 primers live · 26 of 26 chapters live
Primers
2 of 2 liveShort published pieces that set up the vocabulary before the numbered course.
Inference foundations
Two published primers on the inference loop and the KV cache. Read them first for the vocabulary, then start the core course.
2 of 2 lessons live
- 01How LLM inference worksFollow a prompt through tokenization, prefill, and decode, then see why the second phase is sequential and often limited by memory bandwidth.Read
- 02Understanding the KV CacheWhat a language model stores between decode steps, why it grows with every token, and how to estimate the memory bill.Read
Core course
14 of 14 liveThe main sequence, read in order.
Module 01
Transformer architecture and model inputs
Trace a decoder-only model from input IDs to logits, then turn text into tokenised, correctly shifted training sequences.
3 of 3 lessons live
- 01Transformer Architecture Explained: Trace a Decoder-Only LLMFollow one small decoder-only transformer from token IDs to logits, with explicit tensor shapes and component boundaries.Read
- 02Tokenisation and Embeddings: From Text to TensorsFollow a short text string through GPT-2 byte-level BPE tokenisation and an embedding lookup, with explicit tensor-shape tracing from [B,T] to [B,T,d] to [B,T,V].Read
- 03Preparing Training Sequences: Shift, Pad, and PackOne small batch traced from token IDs through padded input IDs, labels, causal mask, logits, and loss. Covers padding and sequence packing conventions.Read
Module 02
Attention mechanisms from scratch
Implement scaled dot-product attention, control what each token can see, and merge parallel heads.
3 of 3 lessons live
- 04Self-Attention: Queries, Keys, and ValuesBuild self-attention in PyTorch: Q/K/V projections, scaled dot-product scores, stable softmax, weighted value aggregation, verified against torch.nn.functional.scaled_dot_product_attention.Read
- 05Causal and Padding Masks: Implementation and TestsConstruct causal and key-padding masks, combine via broadcast AND, and verify future-token leakage. Covers causal mask, padding mask, combined formula, and fully masked rows.Read
- 06Multi-Head Attention: Splitting, Merging, and ProjectingBuild multi-head attention from scratch: combined in-projection, head splitting, parallel attention, head merging, output projection. Verified against PyTorch reference.Read
Module 03
Building and training a language model
Add position information, assemble the decoder block, stack a GPT-style model, then train and debug it.
4 of 4 lessons live
- 07Positional Information: Absolute Embeddings and RoPECompare learned absolute position embeddings with RoPE: trace shapes, verify position-offset extrapolation, and state the Staff/Principal decision framework.Read
- 08Inside the Transformer Block: Residuals and NormalisationTrace residuals through a decoder block with pre-norm RMSNorm/SwiGLU. Compare LayerNorm/RMSNorm, GELU/SwiGLU, pre-norm vs post-norm with shapes and a PyTorch block.Read
- 09Assembling a GPT-Style Language ModelBuild a small runnable GPT-style decoder-only model by stacking decoder blocks, adding final normalisation, tying or untying the output projection, and verifying forward shapes and parameter counts.Read
- 10Training and Debugging: Next-Token Loss and Optimiser StepsConstruct next-token targets with shifted logits, compute cross-entropy loss and perplexity, implement optimizer steps and gradient accumulation, and run a tiny-batch overfit run.Read
Module 04
Generation, KV cache, and context
Load compatible weights, generate text, implement the KV cache, and measure where the context window stops helping.
4 of 4 lessons live
- 11Pretrained Weights and Text GenerationLearn how to adapt a Hugging Face checkpoint to your model architecture, then generate text using four decoding strategies with explicit traceable examples and stopping rules.Read
- 12Prefill and Decode: Measuring the Two PhasesGiven a prompt length, output length, and batch size, trace the two phases separately, measure them, and explain which latency metric each measurement informs.Read
- 13KV Cache Implementation and Memory AccountingUpdate the cache correctly across positions, verify cached and uncached logits match, and calculate memory at a stated precision.Read
- 14Context Windows and Their LimitsDistinguish the configured context limit from effective context, then test truncation, positional limits, and evidence-position sensitivity.Read
Advanced modules
12 of 12 liveDeep dives for readers who have finished the core course.
Module 05
Modern LLM architectures
Compare the attention, routing, and prediction changes behind current model designs under explicit constraints.
6 of 6 lessons live
- 15MHA vs. MQA vs. GQA: Attention Architecture TradeoffsCompare MHA, MQA, and GQA under explicit constraints: query head sharing, KV-cache formula, ratios, and when architecture choice requires evaluation rather than unconditional ranking.Read
- 16Mixture of Experts: Routing, Dispatch, and Load BalancingTrace MoE routing: scores, top-k dispatch, normalised output. Separate active and stored params, compute, memory, latency; include minimal example, capacity/overflow case, brief dense-to-MoE note.Read
- 17Multi-Head Latent Attention: DeepSeek's KV-Cache DesignDeepSeek MLA: joint K/V compression, decoupled RoPE key, projection absorption during decode. A constrained matrix-order trick; cache ratios depend on stated toy dimensions.Read
- 18Multi-Token Prediction: Objectives and TradeoffsMTP predicts future training targets; DeepSeek uses sequential modules discardable at inference, Meta uses parallel heads. Training quality and inference speed must be evaluated separately.Read
- 19What Can You Remove From a Transformer?Compare structural removal (layers, MLP width, attention) from zeroing weights and quantization. Include compatibility, recovery training, an ablation plan, and the latency/parameter gap.Read
- 20Architecture Comparison: Documenting a Defensible ChoiceCompare modern decoder-only configurations under a fixed baseline with explicit conditions, a defensible decision framework, original ablation design, and labelled arithmetic.Read
Module 06
Reasoning models and test-time computation
Evaluate reasoning, spend test-time compute deliberately, and understand the training methods behind reasoning models.
6 of 6 lessons live
- 21How Reasoning Models Work: Training Paths, Traces, and CostReasoning models are built through several non-exclusive paths. A correct final answer can coexist with an unfaithful trace; token cost and process checks must be measured separately.Read
- 22Evaluating Reasoning: Baselines, Verifiers, and Parsing FailuresEstablish a baseline, build a small dataset, extract and normalize answers, count parser failures separately, apply verifiers with their limits, compute exact match and pass@k, and classify errors.Read
- 23Inference-Time Scaling: Sampling, Best-of-N, and Self-ConsistencyCompare repeated sampling, best-of-N, and self-consistency under shared token, latency, and selection budgets; show selection mechanisms, worked arithmetic, and failure cases.Read
- 24Self-Refinement: Critique, Revision, and Stopping PoliciesCompare generate–critique–revise using model self-feedback versus external tool checks, with worked cases, stopping policy, cost model, and a checkable exercise.Read
- 25Reinforcement Learning for Reasoning: Verifiable Rewards and GRPOTrain reasoning models with verifiable rewards using GRPO. Covers rollouts, group-relative advantages, the GRPO objective, regularisation choices, reward hacking, and a worked numerical example.Read
- 26Reasoning Distillation: Teacher Data, Filtering, and Student EvaluationBuild teacher-generated reasoning data, filter it for quality, train a student model, and evaluate capability against deployment cost.Read
The core course is the shortest complete path. Advanced modules assume you have finished it.
Practice labs
Runnable CPU-only companions for selected lessons. Practice material, not published chapters; not counted in the Track 1 chapter count.
- Lab 01 — Attention and masksImplement single-head and multi-head attention, verify tensor shapes, stable softmax, causal future-token isolation, and numerical parity with PyTorch. Diagnose a causal mask that passes shape checks but leaks future tokens.
- Lab 02 — Tiny decoder trainingAssemble a tiny decoder, overfit a fixed batch, assert shapes and parameter count, and diagnose a label-shift bug with a diagnostic check and repair.
- Lab 03 — Cached generation and GQACompare cached and uncached generation, verify one KV append per layer per step, compute raw MHA versus GQA cache bytes, and diagnose a broken cache write.