Skip to content
intai
Search lessonsCtrl K

Track 02

Live

Inference Optimisation & AI Infrastructure

Make models fast and affordable to serve: memory layout, batching, quantisation, and the GPU behaviour behind every latency number.

  • Advanced
  • 16 of 16 lessons live
  • 309 min live
Start with lesson 1View curriculum

What you'll be able to do

By the end of the core course.

  1. Separate prefill and decode costs when diagnosing latency
  2. Estimate KV cache memory for a model, context, and batch size
  3. Choose batching, paging, and quantisation strategies against a target
  4. Plan GPU capacity from a traffic profile and a latency budget

Inference Optimisation & AI Infrastructure

In progress · 4 modules live

Read the modules in order. Each card links to the module's first lesson and states the reader question it answers and the work product you produce.

Inference Optimisation & AI Infrastructure module roadmap4 modules in teaching order, left to right. Each node links to its module section and first lesson; the step cards below give the reader question, work product, and release status per module.M01Measure and Explain InferencePerformance3 lessons liveM02Schedule Requests andManage KV Memory4 lessons liveM03Quantize Without Losingthe Result4 lessons liveM04Build a ServingSystem5 lessons live
  1. Module 013 lessons live
    Measure and Explain Inference Performance

    Reader question: "Why is my first token or stream slow?"

    Work product: Latency and Memory Attribution Report.

    Start with: Serving Workloads and Performance Metrics

  2. Module 024 lessons live
    Schedule Requests and Manage KV Memory

    Reader question: "How do we keep the GPU saturated without running out of memory?"

    Work product: Continuous Batching and Chunk Budget Policy.

    Start with: Batching and throughput

  3. Module 034 lessons live
    Quantize Without Losing the Result

    Reader question: "Can we cut weight and KV memory without degrading accuracy?"

    Work product: Hardware-Calibrated Precision Gate.

    Start with: Quantization: Scale, Zero Point, and Granularity

  4. Module 045 lessons live
    Build a Serving System

    Reader question: "How many GPUs do we need and what happens when traffic spikes?"

    Work product: Production Capacity and Cost Decision Record.

    Start with: Speculative Decoding

Main subjects

The territory this track covers.

  1. 01FlashAttention
  2. 02KV cache
  3. 03Quantisation
  4. 04Batching
  5. 05GPU performance
  6. 06Local and distributed serving

Who this is for

Engineers who run models in production or on their own hardware and need to defend the cost and latency numbers.

Before you start

  • Read How LLM inference works, or know the difference between prefill and decode
  • Comfort with multiplication and powers of two
  • Some experience running a model or using a hosted inference API

Curriculum

16 of 16 lessons published

Core course

16 of 16 live

The main sequence, read in order.

Module 01

Measure and Explain Inference Performance

How to instrument and explain inference latency and memory before changing any system.

3 of 3 lessons live

  1. 01Serving Workloads and Performance MetricsDefine TTFT, inter-token latency, throughput, and tail latency, then map each to a reproducible measurement and the instrument that actually answers it.Intermediate12 min readRead
  2. 02Where Inference Time and Memory GoBreak down inference cost into weights, KV cache, compute, and memory bandwidth, and identify whether prefill or decode is the bottleneck for a stated workload.Intermediate13 min readRead
  3. 03FlashAttention and GPU Data MovementExplain why reducing HBM reads can matter more than reducing FLOPs, and trace where FlashAttention avoids materialising the full attention matrix.Advanced16 min readRead

Module 02

Schedule Requests and Manage KV Memory

How schedulers and cache managers keep the GPU busy without running out of memory.

4 of 4 lessons live

  1. 04Batching and ThroughputAnalyze how batching amortizes weight transfers in LLM decode, derive memory-versus-compute bounds with KV traffic, and evaluate queue and capacity tradeoffs.Intermediate26 min readRead
  2. 05PagedAttention: Virtual Memory for the KV CacheMap logical KV positions to physical blocks, trace copy-on-write sharing, and calculate capacity and tail waste without assuming memory savings guarantee throughput.Advanced35 min readRead
  3. 06Continuous Batching: Iteration-Level SchedulingTrace iteration-level scheduling, request state machines, packed tensor representations, roofline compute-memory dynamics, and KV memory budget limits in LLM serving engines.Advanced30 min readRead
  4. 07Prefill Optimisation: Prefix Caching, Chunked Prefill, and DisaggregationPrefix caching reuses KV blocks across queries to skip redundant prefill compute, while chunked prefill bounds per-iteration tokens to protect decode latency. We trace both mechanisms and tradeoffs.Advanced35 min readRead

Module 03

Quantize Without Losing the Result

How precision choices change memory, speed, and quality, and how to verify them.

4 of 4 lessons live

  1. 08Quantization: Scale, Zero Point, and GranularityChoose the quantization granularity (per-tensor, per-channel, per-group) and measure the rounding error it introduces on a weight matrix.Advanced15 min readRead
  2. 09Calibration Data and Quality RegressionPick representative calibration data, run post-training quantization, and decide whether the quality drop on a held-out set is acceptable before deploying.Advanced18 min readRead
  3. 10Low-bit LLM Weights and ActivationsSeparate what is quantized (weights, activations, KV cache) from the numeric format (INT8, FP8, 4-bit), then compare each combination against hardware support and quality tradeoffs.Advanced15 min readRead
  4. 11KV-Cache QuantizationMeasure memory saved by quantizing the KV cache, the quality impact on a target task, and the runtime cost of dequantisation.Advanced14 min readRead

Module 04

Build a Serving System

How to assemble precision, scheduling, and capacity into a defensible serving design.

5 of 5 lessons live

  1. 12Speculative DecodingExplain draft-and-verify, estimate when it reduces generation latency, and identify the conditions where it does not help.Advanced15 min readRead
  2. 13Choose a Serving Runtime and HardwareCompare CPU, single-GPU, and production serving paths using the same workload, and state the constraints that decide between them.Advanced15 min readRead
  3. 14Capacity Planning: From Traffic Profile to GPU CountTurn a latency target and a traffic profile into a GPU count you can defend, using memory bounds, queueing theory, and a load-sweep protocol.Advanced18 min readRead
  4. 15Scale Beyond One GPU: Replication, Tensor Parallelism, and Prefill/Decode DisaggregationChoose a multi-GPU serving topology—replicas, tensor parallelism, or prefill/decode disaggregation—given a model footprint, traffic target, latency SLOs, and GPU interconnect.Advanced16 min readRead
  5. 16Overload and Failure HandlingWhen a burst exceeds serving capacity, decide which requests to admit, reject, cancel, or route to a fallback while protecting latency, useful throughput, quality, and cost.Advanced16 min readRead

Lessons are published in reading order. A coming-soon entry has no page yet — it is not a broken link.