Debug and Reproduce a Training Run

Section 1: The Practical Outcome First

In production deep learning, the most expensive error is not a syntax crash that fails on line 1; it is the silent failure that runs for 18 hours across 8 GPUs, consumes $3,000 in compute, and produces an unusable model checkpoint.

Consider common failure modes observed in fine-tuning runs:

1. The Inverted Mask: A collator bug sets user prompt tokens to active and masks assistant completions with label = -100. The model trains smoothly, training loss drops, but at inference time the model has learned only to predict the user prompt back to itself.
2. The Masked EOS Disaster: The template tokenizer fails to append or supervise the termination token (<|im_end|> or <|eot_id|>). The model learns valid domain responses, but in production it generates endless hallucinated chatter until hitting max token limits.
3. The Exploding Gradient Crash: At step 4,200, an unclipped outlier token produces an activation spike, driving the gradient norm to Inf and turning all subsequent weights into NaN.
4. The Ghost Run: A team achieves breakthrough accuracy on a domain evaluation benchmark, but three weeks later cannot reproduce the weights because environment seeds, data shard permutations, and library versions were never recorded.

To eliminate these failures, Staff engineers enforce a 4-Stage Pre-Flight Protocol before spending any full compute budget.

[Pre-Flight Verification Protocol: 4 Deterministic Gates]
┌────────────────────────────────────────────────────────────────────────┐
│ Gate 1: Token & Loss Mask Alignment                                    │
│ Requirement: 0% prompt loss, 100% completion loss + EOS supervised     │
└───────────────────────────────────┬────────────────────────────────────┘
                                    │ PASS
                                    ▼
┌────────────────────────────────────────────────────────────────────────┐
│ Gate 2: Karpathy Tiny-Batch Overfit Gate                               │
│ Requirement: 8 samples converge to Loss < 0.05 within 35 steps         │
└───────────────────────────────────┬────────────────────────────────────┘
                                    │ PASS
                                    ▼
┌────────────────────────────────────────────────────────────────────────┐
│ Gate 3: Gradient Norm & Health Telemetry                               │
│ Requirement: L2-norm in [0.1, 5.0]; clip at max_norm = 1.0; zero NaN  │
└───────────────────────────────────┬────────────────────────────────────┘
                                    │ PASS
                                    ▼
┌────────────────────────────────────────────────────────────────────────┐
│ Gate 4: Deterministic Reproducibility Manifest                         │
│ Requirement: Immutable SHA-256 for code, data, seeds, and weights     │
└───────────────────────────────────┬────────────────────────────────────┘
                                    │ PASS
                                    ▼
               [AUTHORIZE FULL TRAINING COMPUTE BUDGET]

Passing this 5-minute pre-flight protocol guarantees that data formatting, loss calculation, gradient backpropagation, and execution determinism are verified before launching full-scale training.

---

Section 2: The 4-Stage Pre-Flight Protocol

The 4-stage pre-flight debugging and verification architecture

Read this diagram as text

Before committing GPU clusters to full training, the pipeline verifies token mask alignment, executes the Karpathy tiny-batch overfit check to force loss below 0.05, monitors gradient L2-norms with clipping, and records an immutable provenance manifest.

1. Stage 1: Mask & Loss Sanity Diagnostic

Before launching a distributed training loop, print the string representation of token IDs and their corresponding label values for batch 0:

[Token ID Inspection: Sample 0]
Token: "<|im_start|>"  -> Label: -100  (Masked)
Token: "user"           -> Label: -100  (Masked)
Token: "\n"             -> Label: -100  (Masked)
Token: "Cluster"        -> Label: -100  (Masked)
Token: "CRITICAL"       -> Label: -100  (Masked)
Token: "<|im_end|>"    -> Label: -100  (Masked)
Token: "<|im_start|>"  -> Label: -100  (Masked)
Token: "assistant"      -> Label: -100  (Masked)
Token: "\n"             -> Label: -100  (Masked)
Token: "{"              -> Label: 128   (SUPERVISED)
Token: "\"subsystem\""  -> Label: 4501  (SUPERVISED)
Token: "cache"          -> Label: 2910  (SUPERVISED)
Token: "}"              -> Label: 130   (SUPERVISED)
Token: "<|im_end|>"    -> Label: 151645 (SUPERVISED - CRITICAL!)

Verify that:

• Every prompt token has label = -100.
• The assistant opening prefix (assistant\n) has label = -100.
• The completion payload has label >= 0.
• The termination token has label >= 0. If the stop token is labeled -100, the model will never receive gradient pressure to terminate its generation.

2. Stage 2: The Karpathy Tiny-Batch Overfit Gate

Originally formulated by Andrej Karpathy (2019) in A Recipe for Training Neural Networks, the tiny-batch overfit check is the single most effective heuristic in machine learning:

The Karpathy Rule: Take a micro-batch of 8 to 16 training examples. Turn off data shuffling, disable weight decay, set a relatively high learning rate (e.g. 1e-3 or 5e-2), and train for 30 to 50 iterations.

If the loss does not collapse monotonically to near-zero (Loss < 0.05), stop immediately.

A model with tens of millions of trainable parameters that cannot memorize 8 training examples is suffering from a fundamental structural defect:

• Broken loss computation or invalid label offsets.
• Frozen trainable parameters (e.g. adapters not marked with requires_grad = True).
• Exploding or vanishing learning rate schedules.
• Incorrect tensor permutations during collation.

3. Stage 3: Gradient Norm Health Monitoring

During training, monitor the global L2-norm of the parameter gradients:

||g||_2 = sqrt( sum_i ||g_i||_2^2 )

Gradient dynamics indicate training health:

• Vanishing Gradients (||g||_2 < 1e-5): Signals frozen layers, dead activations, or an underflow condition.
• Exploding Gradients (||g||_2 > 10.0 or NaN): Signals an excessively high learning rate, numerical overflow in FP16, or corrupted batch inputs.
• Healthy Range: In stable transformer fine-tuning, gradient norms typically settle between 0.1 and 3.0.

Always enforce gradient clipping (torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)) to truncate destabilizing gradient spikes.

4. Stage 4: Deterministic Reproducibility Manifest

True engineering reproducibility requires pinning every layer of the training stack:

import random
import numpy as np
import torch

def set_deterministic_env(seed: int = 42):
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False
    torch.use_deterministic_algorithms(True)
    # CUDA deterministic algorithm (recommended for most production work)
    import os
    os.environ["CUBLAS_WORKSPACE_CONFIG"] = ":4096:8"

Serialize an immutable training_manifest.json recording:

• Git commit SHA
• Training dataset SHA-256 hash
• Base model weights SHA-256 hash
• Tokenizer version and vocabulary hash
• Hyperparameter JSON (learning rate, batch size, warmup steps, adapter rank r, alpha)
• CUDA version, driver version, PyTorch version, and GPU hardware device names.

---

Section 3: Diagnostic Decision Matrix for Training Failures

When training loss anomalies appear in logs or monitoring dashboards, use this diagnosis guide:

Observed Symptom	Root Cause Mechanism	Immediate Remediation Action
Loss = NaN / Inf at Step 1	Numerical overflow in FP16 precision or learning rate too high	Switch from FP16 to BF16; lower learning rate by 5×; verify inputs contain no NaN/Inf values
Loss remains completely flat	Zero gradient flow to trainable parameters	Inspect p.requires_grad for adapter parameters; verify loss mask is not masking 100% of tokens
Loss collapses to 0.0 at Step 1	Target label leakage in the prompt sequence	Check collator; ensure target completion tokens are not concatenated inside the prompt string
Loss oscillates erratically	Micro-batch size too small or missing learning rate warmup	Increase gradient accumulation steps; introduce linear learning rate warmup for first 5% of steps
Infinite repetition at inference	Assistant stop token (EOS delimiter) was masked with label -100	Unmask termination token in loss mask; supervise EOS during SFT

In production

The Warmup Imperative (You et al., 2019): Never train transformers without a learning rate warmup schedule. In the initial steps of fine-tuning, optimizer momentum and variance estimates are uncalibrated. Applying full learning rates on step 1 introduces massive, erratic gradient updates that permanently destabilize pretrained attention weights. Always ramp learning rate linearly from 0 over the first 50 to 100 steps.

---

Section 4: Executable Verification & Arithmetic Parity

To verify the Karpathy overfit gate, loss mask checks, and deterministic seed repeatability, run the standalone verification script [scripts/verify-tiny-overfit.py]:

scripts/verify-tiny-overfit.py

# Runs a deterministic sequence model over 8 training samples
# Evaluates loss mask alignment and verifies loss convergence < 0.05

Execution produces the exact, verified numerical outputs:

=== Running Topic 12 Pre-Flight Overfit & Reproducibility Verifier ===
[*] Micro-Batch Size: 8 samples
[*] Prompt Mask Check: 4/8 tokens masked (-100), 4/8 supervised (incl. EOS)
[*] Step 00 - Initial Loss: 1.5980 | Grad Norm: 0.5980
[*] Step 10 - Loss: 0.0000 | Grad Norm: 0.0001
[*] Step 20 - Loss: 0.0000 | Grad Norm: 0.0001
[*] Step 30 - Loss: 0.0000 | Grad Norm: 0.0000
[*] Step 35 - Loss: 0.0000 | Grad Norm: 0.0000

[*] Convergence Check: 1.5980 -> 0.0000
[+] Karpathy Tiny-Batch Overfit Gate: PASSED (loss < 0.05).
[+] Deterministic Seed Reproducibility: PASSED.
=== All Topic 12 Verifications Passed Cleanly ===

Module 3 Exit Lab Walkthrough (labs/track3/03-adapter-training/)

To complete the Module 3 Exit Artifact, execute the CPU toy LoRA tiny-batch overfit diagnostic in labs/track3/03-adapter-training/:

python3 labs/track3/03-adapter-training/tiny_overfit_diagnostic.py

This script verifies the Module 3 exit gates locally without requiring a cloud GPU:

1. Instantiates TinyLoRALanguageModel (vocab_size = 64, d_model = 32, rank = 16, alpha = 32) with frozen base weights (W_head_frozen, W_embed_frozen) and trainable low-rank matrices (lora_A initialized Gaussian, lora_B initialized zero).
2. Asserts that all prompt tokens are masked (label = -100) and the termination token (EOS = 63) is supervised.
3. Overfits an 8-sequence micro-batch from initial loss 4.2204 (Step 0) to 0.0000 (< 0.05 target at Step 34) while asserting that frozen base weights remain bit-identical.
4. Verifies bit-exact trajectory reproducibility across two independent runs under seed = 42, writing labs/track3/03-adapter-training/overfit_run_record.json and the analytical 8B QLoRA budget specification labs/track3/03-adapter-training/adapter_training_config.json (documented in labs/track3/03-adapter-training/ADAPTER_TRAINING_AUDIT.md).

---

Key takeaways

• Always execute a 4-stage pre-flight verification protocol before committing GPU clusters to full training runs.
• Inspect batch 0 to verify that exactly 0% of prompt tokens are supervised and that the assistant completion and EOS tokens are unmasked.
• The Karpathy Tiny-Batch Overfit check trains on 8-16 samples; failure to converge to Loss < 0.05 within 35 steps proves a fatal structural defect in masking or gradients.
• Monitor global gradient L2-norm (||g||_2) and clip at max_norm = 1.0 to prevent exploding gradient NaN crashes.
• Always implement a linear learning rate warmup schedule (first 5% of steps) to prevent uncalibrated AdamW moments from corrupting weights at step 1.
• Serialize a deterministic run manifest capturing seeds, git commit SHA, base model weight hash, and dataset SHA-256 for bit-exact reproduction.

Try it yourself

1. 01A fine-tuned model outputs correct domain incident classifications, but never halts generation, appending repetitive filler text until reaching max_tokens. What diagnostic check was omitted?

   Show a hint

   The pre-flight token mask check was omitted. The assistant end-of-sequence delimiter (<|im_end|> or <|eot_id|>) was labeled -100, leaving the model without supervision to terminate.

2. 02Calculate the global gradient L2-norm for a model with two parameter matrices having gradient Frobenius norms ||g_1||_F = 3.0 and ||g_2||_F = 4.0.

   Show a hint

   ||g||_2 = sqrt(||g_1||_F^2 + ||g_2||_F^2) = sqrt(3^2 + 4^2) = sqrt(9 + 16) = sqrt(25) = 5.0.

3. 03Why does training loss suddenly become NaN at step 2,500 when fine-tuning in FP16 precision without gradient clipping?

   Show a hint

   An outlier activation produced a gradient norm spike that exceeded the maximum representable dynamic range of FP16 (65,504), causing numerical overflow into Inf and NaN.

Interview questions

What is the Karpathy tiny-batch overfit test, and why is it mandatory before large-scale training?

The Karpathy overfit test isolates a micro-batch of 8 to 16 training examples, disables regularization, and trains with a high learning rate for 30 to 50 iterations. Because a neural network has immense capacity, it must memorize 8 examples to near-zero loss (L < 0.05). If it fails to converge, there is a fundamental bug in the loss function, tensor collation, parameter gradients, or masking that must be fixed before spending compute.

Why is gradient clipping essential when fine-tuning autoregressive transformers with AdamW?

During autoregressive sequence training, long sequences, unusual formatting, or rare token sequences can produce localized activation spikes in attention layers. These spikes produce massive gradient vectors that can destabilize AdamW momentum estimates or cause numerical overflow in 16-bit precision. Clipping truncates the global gradient norm to a maximum threshold (such as 1.0), preserving direction while preventing destabilization.

What specific environment parameters must be pinned in a reproducibility manifest to achieve deterministic training?

A deterministic manifest must record: (1) Random seeds for Python random, NumPy, and PyTorch (CPU and CUDA); (2) Flags setting cudnn.deterministic = True and cudnn.benchmark = False; (3) Git commit SHA of the training repository; (4) SHA-256 hashes of the dataset shards and base model checkpoint; and (5) Software versions for PyTorch, CUDA, Transformers, and PEFT.

Why is learning rate warmup required during the initial steps of transformer fine-tuning?

As demonstrated by You et al. (2019), at the start of training, the AdamW optimizer has zero prior history, meaning its first and second moment estimates (momentum and uncentered variance) are completely uncalibrated. Taking large step sizes with uncalibrated optimizer states can corrupt delicate pretrained attention representations. Ramping the learning rate linearly from zero gives the optimizer time to calibrate stable direction vectors.

Related lessons

• Fine-Tuning & Domain-Specific Models · Decide Whether to TrainDiagnose the Gap Before Changing Weights12 min
• Fine-Tuning & Domain-Specific Models · Decide Whether to TrainSelect a Base Model for the Task13 min
• Fine-Tuning & Domain-Specific Models · Decide Whether to TrainSet a Baseline and Success Threshold12 min
• Fine-Tuning & Domain-Specific Models · Make Training Data TrustworthyFormat Conversations and Mask the Loss13 min
• Fine-Tuning & Domain-Specific Models · Adapt Within a Compute BudgetChoose Continued Pretraining or Supervised Fine-Tuning14 min
• Fine-Tuning & Domain-Specific Models · Adapt Within a Compute BudgetLoRA: Add Trainable Low-Rank Adapters14 min
• Fine-Tuning & Domain-Specific Models · Adapt Within a Compute BudgetQLoRA: Quantize Base Weights to 4-Bit14 min
• Fine-Tuning & Domain-Specific Models · Adapt Within a Compute BudgetFull Fine-Tuning Versus Adapters14 min