Blogs
Every published lesson from int8ai.com, newest first: model internals, inference, and the systems around them.
Published lessons
60 lessons · newest first
- Track 03Diagnose the Gap Before Changing WeightsEvaluate zero-shot, few-shot, and RAG on a held-out domain task before fine-tuning, diagnosing format, factual, and reasoning gaps.12 min read
- Track 03Select a Base Model for the TaskEvaluate open-weights base models across tokenizer fertility on structured JSON, VRAM budgets for LoRA versus serving, base versus instruct checkpoints, and enterprise licensing.13 min read
- Track 03Set a Baseline and Success ThresholdEstablish frozen held-out evaluation sets, behavioral slicing, safety recall asymmetry, general capability regression guardrails, and the formal Ship/Train/Hold decision record.12 min read
- Track 03Turn Real Tasks Into Training ExamplesSpecify desired behaviour, provenance, rights, and sensitive-data handling for each training example.14 min read
- Track 03Format Conversations and Mask the LossInspect token IDs and loss labels for the chosen objective, including chat-template boundaries, EOS, truncation, and packing; assistant-only loss is optional.13 min read
- Track 03Split, Deduplicate, and Audit the DatasetUse group-aware splits and exact or near-duplicate checks to limit leakage, imbalance, and evaluation contamination before training.13 min read
- Track 03Use Synthetic and Teacher Data CarefullyTrack teacher and data provenance, apply task-specific checks, and compare generated examples against real held-out tasks.13 min read
- Track 03Choose Continued Pretraining or Supervised Fine-TuningDistinguish unsupervised domain-adaptive pretraining from instruction tuning, budget training compute, and avoid hallucination pitfalls when injecting knowledge.14 min read
- Track 03LoRA: Add Trainable Low-Rank AdaptersDecompose dense weight updates into low-rank matrices, calculate trainable parameter ratios, select target modules, and scale adapter updates with alpha.14 min read
- Track 03QLoRA: Quantize Base Weights to 4-BitBudget VRAM, compress base models with 4-bit NormalFloat and Double Quantization, and use paged optimizers to fine-tune on consumer GPUs.14 min read
- Track 03Full Fine-Tuning Versus AdaptersCompare full parameter training against low-rank adapters across compute costs, memory sharding, catastrophic forgetting, and multi-tenant serving.14 min read
- Track 03Debug and Reproduce a Training RunApply pre-flight sanity checks, execute tiny-batch overfit tests, track gradient norms, and generate deterministic reproducibility manifests.14 min read
- Track 03Choose a Compact Student ModelSize a student model for the task and deployment constraints before distillation begins, comparing parameter scales, KV-cache footprints, and tokenizer alignment.13 min read
- Track 03Distill Useful Behaviour From a TeacherCompare teacher outputs with task labels and check transfer failures during distillation.14 min read
- Track 03Test Gains and RegressionsCompare paired domain, general, out-of-distribution, and safety results; audit judge bias where qualitative scoring is needed.14 min read
- Track 03Package the Model for ReleaseRecord model and adapter lineage, data rights, evaluation evidence, deployment boundaries, and rollback triggers in a model card.13 min read
- Track 02FlashAttention and GPU Data MovementExplain why reducing HBM reads can matter more than reducing FLOPs, and trace where FlashAttention avoids materialising the full attention matrix.16 min read
- Track 02KV-Cache QuantizationMeasure memory saved by quantizing the KV cache, the quality impact on a target task, and the runtime cost of dequantisation.14 min read
- Track 02Speculative DecodingExplain draft-and-verify, estimate when it reduces generation latency, and identify the conditions where it does not help.15 min read
- Track 02Choose a Serving Runtime and HardwareCompare CPU, single-GPU, and production serving paths using the same workload, and state the constraints that decide between them.15 min read
- Track 02Capacity Planning: From Traffic Profile to GPU CountTurn a latency target and a traffic profile into a GPU count you can defend, using memory bounds, queueing theory, and a load-sweep protocol.18 min read
- Track 02Scale Beyond One GPU: Replication, Tensor Parallelism, and Prefill/Decode DisaggregationChoose a multi-GPU serving topology—replicas, tensor parallelism, or prefill/decode disaggregation—given a model footprint, traffic target, latency SLOs, and GPU interconnect.16 min read
- Track 02Overload and Failure HandlingWhen a burst exceeds serving capacity, decide which requests to admit, reject, cancel, or route to a fallback while protecting latency, useful throughput, quality, and cost.16 min read
- Track 02Serving Workloads and Performance MetricsDefine TTFT, inter-token latency, throughput, and tail latency, then map each to a reproducible measurement and the instrument that actually answers it.12 min read
- Track 02Where Inference Time and Memory GoBreak down inference cost into weights, KV cache, compute, and memory bandwidth, and identify whether prefill or decode is the bottleneck for a stated workload.13 min read
- Track 02Quantization: Scale, Zero Point, and GranularityChoose the quantization granularity (per-tensor, per-channel, per-group) and measure the rounding error it introduces on a weight matrix.15 min read
- Track 02Calibration Data and Quality RegressionPick representative calibration data, run post-training quantization, and decide whether the quality drop on a held-out set is acceptable before deploying.18 min read
- Track 02Low-bit LLM Weights and ActivationsSeparate what is quantized (weights, activations, KV cache) from the numeric format (INT8, FP8, 4-bit), then compare each combination against hardware support and quality tradeoffs.15 min read
- Track 02Batching and ThroughputAnalyze how batching amortizes weight transfers in LLM decode, derive memory-versus-compute bounds with KV traffic, and evaluate queue and capacity tradeoffs.26 min read
- Track 02PagedAttention: Virtual Memory for the KV CacheMap logical KV positions to physical blocks, trace copy-on-write sharing, and calculate capacity and tail waste without assuming memory savings guarantee throughput.35 min read
- Track 02Continuous Batching: Iteration-Level SchedulingTrace iteration-level scheduling, request state machines, packed tensor representations, roofline compute-memory dynamics, and KV memory budget limits in LLM serving engines.30 min read
- Track 02Prefill Optimisation: Prefix Caching, Chunked Prefill, and DisaggregationPrefix caching reuses KV blocks across queries to skip redundant prefill compute, while chunked prefill bounds per-iteration tokens to protect decode latency. We trace both mechanisms and tradeoffs.35 min read
- Track 01Architecture Comparison: Documenting a Defensible ChoiceCompare modern decoder-only configurations under a fixed baseline with explicit conditions, a defensible decision framework, original ablation design, and labelled arithmetic.14 min read
- Track 01How Reasoning Models Work: Training Paths, Traces, and CostReasoning models are built through several non-exclusive paths. A correct final answer can coexist with an unfaithful trace; token cost and process checks must be measured separately.16 min read
- Track 01Evaluating Reasoning: Baselines, Verifiers, and Parsing FailuresEstablish a baseline, build a small dataset, extract and normalize answers, count parser failures separately, apply verifiers with their limits, compute exact match and pass@k, and classify errors.12 min read
- Track 01Inference-Time Scaling: Sampling, Best-of-N, and Self-ConsistencyCompare repeated sampling, best-of-N, and self-consistency under shared token, latency, and selection budgets; show selection mechanisms, worked arithmetic, and failure cases.13 min read
- Track 01Self-Refinement: Critique, Revision, and Stopping PoliciesCompare generate–critique–revise using model self-feedback versus external tool checks, with worked cases, stopping policy, cost model, and a checkable exercise.13 min read
- Track 01Reinforcement Learning for Reasoning: Verifiable Rewards and GRPOTrain reasoning models with verifiable rewards using GRPO. Covers rollouts, group-relative advantages, the GRPO objective, regularisation choices, reward hacking, and a worked numerical example.14 min read
- Track 01Reasoning Distillation: Teacher Data, Filtering, and Student EvaluationBuild teacher-generated reasoning data, filter it for quality, train a student model, and evaluate capability against deployment cost.12 min read
- Track 01How LLM inference worksFollow a prompt through tokenization, prefill, and decode, then see why the second phase is sequential and often limited by memory bandwidth.9 min read
- Track 01Understanding the KV CacheWhat a language model stores between decode steps, why it grows with every token, and how to estimate the memory bill.12 min read
- Track 01Transformer Architecture Explained: Trace a Decoder-Only LLMFollow one small decoder-only transformer from token IDs to logits, with explicit tensor shapes and component boundaries.14 min read
- Track 01Tokenisation and Embeddings: From Text to TensorsFollow a short text string through GPT-2 byte-level BPE tokenisation and an embedding lookup, with explicit tensor-shape tracing from [B,T] to [B,T,d] to [B,T,V].12 min read
- Track 01Preparing Training Sequences: Shift, Pad, and PackOne small batch traced from token IDs through padded input IDs, labels, causal mask, logits, and loss. Covers padding and sequence packing conventions.15 min read
- Track 01Self-Attention: Queries, Keys, and ValuesBuild self-attention in PyTorch: Q/K/V projections, scaled dot-product scores, stable softmax, weighted value aggregation, verified against torch.nn.functional.scaled_dot_product_attention.14 min read
- Track 01Causal and Padding Masks: Implementation and TestsConstruct causal and key-padding masks, combine via broadcast AND, and verify future-token leakage. Covers causal mask, padding mask, combined formula, and fully masked rows.15 min read
- Track 01Multi-Head Attention: Splitting, Merging, and ProjectingBuild multi-head attention from scratch: combined in-projection, head splitting, parallel attention, head merging, output projection. Verified against PyTorch reference.14 min read
- Track 01Positional Information: Absolute Embeddings and RoPECompare learned absolute position embeddings with RoPE: trace shapes, verify position-offset extrapolation, and state the Staff/Principal decision framework.13 min read
- Track 01Inside the Transformer Block: Residuals and NormalisationTrace residuals through a decoder block with pre-norm RMSNorm/SwiGLU. Compare LayerNorm/RMSNorm, GELU/SwiGLU, pre-norm vs post-norm with shapes and a PyTorch block.12 min read
- Track 01Assembling a GPT-Style Language ModelBuild a small runnable GPT-style decoder-only model by stacking decoder blocks, adding final normalisation, tying or untying the output projection, and verifying forward shapes and parameter counts.15 min read
- Track 01Training and Debugging: Next-Token Loss and Optimiser StepsConstruct next-token targets with shifted logits, compute cross-entropy loss and perplexity, implement optimizer steps and gradient accumulation, and run a tiny-batch overfit run.14 min read
- Track 01Pretrained Weights and Text GenerationLearn how to adapt a Hugging Face checkpoint to your model architecture, then generate text using four decoding strategies with explicit traceable examples and stopping rules.12 min read
- Track 01Prefill and Decode: Measuring the Two PhasesGiven a prompt length, output length, and batch size, trace the two phases separately, measure them, and explain which latency metric each measurement informs.11 min read
- Track 01KV Cache Implementation and Memory AccountingUpdate the cache correctly across positions, verify cached and uncached logits match, and calculate memory at a stated precision.14 min read
- Track 01Context Windows and Their LimitsDistinguish the configured context limit from effective context, then test truncation, positional limits, and evidence-position sensitivity.11 min read
- Track 01MHA vs. MQA vs. GQA: Attention Architecture TradeoffsCompare MHA, MQA, and GQA under explicit constraints: query head sharing, KV-cache formula, ratios, and when architecture choice requires evaluation rather than unconditional ranking.14 min read
- Track 01Mixture of Experts: Routing, Dispatch, and Load BalancingTrace MoE routing: scores, top-k dispatch, normalised output. Separate active and stored params, compute, memory, latency; include minimal example, capacity/overflow case, brief dense-to-MoE note.15 min read
- Track 01Multi-Head Latent Attention: DeepSeek's KV-Cache DesignDeepSeek MLA: joint K/V compression, decoupled RoPE key, projection absorption during decode. A constrained matrix-order trick; cache ratios depend on stated toy dimensions.14 min read
- Track 01Multi-Token Prediction: Objectives and TradeoffsMTP predicts future training targets; DeepSeek uses sequential modules discardable at inference, Meta uses parallel heads. Training quality and inference speed must be evaluated separately.15 min read
- Track 01What Can You Remove From a Transformer?Compare structural removal (layers, MLP width, attention) from zeroing weights and quantization. Include compatibility, recovery training, an ablation plan, and the latency/parameter gap.14 min read
Keep going
Read a track in order.
Lessons are written as a sequence. Track 01 is live now, and the remaining tracks open as their chapters are published.