Choose Continued Pretraining or Supervised Fine-Tuning

Section 1: The Practical Outcome First

When engineering specialized language models, teams often face a fundamental architectural choice: should you continue pretraining on hundreds of millions of raw domain tokens, or jump straight into Supervised Fine-Tuning (SFT) on a few thousand instruction pairs?

Consider a real enterprise scenario: Site Reliability Engineering (SRE) incident triage. Your infrastructure uses a proprietary logging format, over 2,000 custom error codes (such as ERR_AUTH_REDIS_CONN_TIMEOUT_42), and an internal Domain-Specific Language (DSL) for querying system metrics:

trace.filter(latency > 5000ms).group_by(cluster).top(5)

If you feed these raw logs into standard SFT on an 8B foundation model, the results are catastrophic:

1. Subword Fragmentation: The general-purpose tokenizer breaks proprietary error codes into four to six meaningless subword pieces, treating the critical identifier as high-entropy noise.
2. Grammar Fragility: The model memorizes exact DSL queries from the training prompt-completion pairs but fails to compose valid syntax on novel metric filters.
3. Severe Hallucination: Because the underlying base model was never pretrained on your private infrastructure documentation, forcing it via SFT to state “the root cause is redis primary node 10.240.12.8” causes it to hallucinate plausible-sounding fictional server names across unrelated queries.

Look at the pipeline decision matrix:

[Raw Engineering Corpora: 1.2 Billion Tokens of Logs, DSL queries, Internal RFCs]
          │
          ▼
   ┌──────────────────────────────────────────────────────────┐
   │ Phase 1: Continued Pretraining (CPT / DAPT)              │
   │ Objective: Next-token prediction across 100% of tokens   │
   │ Compute: 48.18 ExaFLOPs on 8B model                      │
   │ Result: Internalizes subword distributions & DSL grammar │
   └──────────────────────────────────────────────────────────┘
          │
          ▼
   ┌──────────────────────────────────────────────────────────┐
   │ Phase 2: Supervised Fine-Tuning (SFT)                    │
   │ Objective: Masked loss on assistant actions only         │
   │ Compute: 0.49 ExaFLOPs (10,000 curated demonstrations)   │
   │ Result: Teaches triage formatting, JSON schema, tone     │
   └──────────────────────────────────────────────────────────┘
          │
          ▼
   ┌──────────────────────────────────────────────────────────┐
   │ Phase 3: Runtime Retrieval-Augmented Generation (RAG)    │
   │ Objective: Real-time context injection                   │
   │ Result: Supplies dynamic IPs, topology, active alerts    │
   └──────────────────────────────────────────────────────────┘

CPT teaches the model the language, syntax, and grammar of the domain. SFT teaches the model how to act as an assistant. RAG provides ground-truth volatile facts. Conflating these three roles leads to bloated compute budgets or unstable, hallucinating models.

---

Section 2: Mathematical Objectives: CPT vs SFT Loss

The core technical difference between Continued Pretraining (also known as Domain-Adaptive Pretraining or DAPT) and Supervised Fine-Tuning lies in which tokens receive gradient updates during backpropagation.

Loss computation comparison: Continued Pretraining (CPT) vs Supervised Fine-Tuning (SFT)

Read this diagram as text

In Continued Pretraining, 100% of tokens in the domain sequence are supervised with standard cross-entropy loss. In Supervised Fine-Tuning, user prompt tokens are masked with label -100, and only assistant completion tokens generate loss gradients.

Continued Pretraining Objective

CPT uses standard autoregressive next-token prediction across the full sequence length T of uncurated domain text. Every token participates in loss calculation:

L_CPT = - (1 / T) * sum_{t=1}^T log P(x_t | x_{<t})

Because every token produces a loss signal, the model adjusts its internal representations for every vocabulary token, token co-occurrence distribution, and syntactic pattern. There is no concept of a “user prompt” or an “assistant response.” The mask ratio is exactly 0.0%.

Supervised Fine-Tuning Objective

In contrast, SFT formats demonstrations into multi-turn conversations and masks out the prompt tokens using PyTorch’s ignore_index = -100. Only the subset of tokens generated by the assistant incur loss:

L_SFT = - (1 / N_active) * sum_{t in A} log P(x_t | x_{<t})

In standard SFT datasets, prompts are typically long (system prompt, few-shot schema, telemetry inputs), while completions are short and structured (JSON action payloads). As measured in production incident routing datasets:

• Total tokens: 9
• Prompt masked tokens: 6 (66.7% mask ratio)
• Active assistant tokens: 3 (33.3% supervised)

SFT teaches conditional generation: given this input pattern, produce this structured output. It does not restructure the model’s fundamental world knowledge or subword distributions.

---

Section 3: The 6ND Compute Law & Resource Disparity

Understanding the vast compute disparity between CPT and SFT is essential for realistic capacity budgeting.

According to empirical scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022), the total floating-point operations (FLOPs) required to train an autoregressive transformer can be calculated using the standard formulation:

C ≈ 6 * N * D

Where:

• C is the total compute in FLOPs.
• N is the parameter count of the model (approximately N ≈ 8.03 × 10^9 for Llama 3.1 8B).
• D is the total number of training tokens processed.

Let us compare the compute requirements for adapting a Llama 3.1 8B model under both paradigms:

Training Paradigm	Dataset Token Scale (D)	Total Compute (C ≈ 6ND)	Multiplier vs SFT	Typical GPU Hours (8x H100)
Domain CPT	1.0 × 10^9 tokens (1B tokens)	48.18 ExaFLOPs (4.82 × 10^19 FLOPs)	97.7×	~24.0 hours
Supervised SFT	10,000 examples @ 1,024 tokens (1.024 × 10^7 tokens)	0.49 ExaFLOPs (4.93 × 10^17 FLOPs)	1.0× (Baseline)	~0.25 hours (15 mins)

CPT requires nearly two orders of magnitude more compute (97.7×) than a comprehensive SFT run. Furthermore, CPT demands strict learning rate scheduling:

1. Warmup & Cosine Decay: You cannot use a high initial learning rate. CPT typically uses 10% to 20% of the base model’s peak learning rate (e.g., 2e-5 instead of 3e-4), with a cosine decay to 10%.
2. Replay Buffers: Without mixing 10% to 20% generic pretraining data into your domain corpus, the model will suffer catastrophic forgetting of general reasoning and linguistic abilities.

In production

The Capacity Budgeting Rule: Only commit to Continued Pretraining if your domain vocabulary divergence exceeds tokenizer capacity (e.g., medical clinical notes, proprietary legal codes, or codebases with extensive internal DSLs) AND you have at least 500M tokens of clean text. If you have fewer than 100M tokens, CPT will quickly overfit or degenerate; use SFT with RAG instead.

---

Section 4: The Hallucination Pitfall: Gekhman et al. (2024)

A widespread misconception among engineering teams is that Supervised Fine-Tuning is the appropriate tool for teaching models “new internal company facts” (such as new employee directories, updated API endpoints, or recent infrastructure IP allocations).

Research by Gekhman et al. (2024) (Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?, arXiv:2405.05904) systematically dismantled this assumption.

When you fine-tune a model on factual associations that were never present in its pretraining corpus, the model experiences an alignment distortion:

1. Loss Forcing: The cross-entropy loss forces the model to assign high probability to specific factual tokens (e.g., "host": "10.240.12.8") despite having no underlying semantic prior in its weights.
2. Calibration Collapse: Instead of learning to output uncertainty when queried about unknown entities, the model internalizes a toxic prior: it is rewarded for asserting arbitrary plausible completions with high confidence.
3. Generalization Failure: Gekhman et al. proved that fine-tuning on unseen factual knowledge dramatically increases hallucination rates on completely unrelated queries across the evaluation suite.

[Knowledge Update Strategy Matrix]
┌────────────────────────┬────────────────────────────────────────────────────────┐
│ Target Knowledge Type  │ Correct Architectural Mechanism                        │
├────────────────────────┼────────────────────────────────────────────────────────┤
│ Domain Vocabulary/DSL  │ Continued Pretraining (CPT / DAPT)                     │
│ Behavior & Schema      │ Supervised Fine-Tuning (SFT) with Masked Loss          │
│ Volatile / Dynamic Data│ Retrieval-Augmented Generation (RAG) Context Injection │
│ Safety Policies        │ SFT + Direct Preference Optimization (DPO / RLHF)      │
└────────────────────────┴────────────────────────────────────────────────────────┘

Never fine-tune model weights to memorize information that changes more frequently than your quarterly retraining cadence. Ground volatile operational telemetry at runtime via RAG (Lewis et al., 2020).

---

Section 5: Executable Verification & Arithmetic Parity

To verify the loss masking ratios and FLOP compute calculations in your environment, run the standalone verification script [scripts/verify-cpt-vs-sft.py]:

scripts/verify-cpt-vs-sft.py

```python
# scripts/verify-cpt-vs-sft.py
def compute_training_flops(params_b: float, tokens_b: float) -> float:
    # Kaplan et al. (2020) formula: C ≈ 6 * N * D
    n = params_b * 1e9
    d = tokens_b * 1e9
    flops = 6 * n * d
    return round(flops / 1e18, 2)  # ExaFLOPs

# Llama 3.1 8B (N = 8.03B)
cpt_flops = compute_training_flops(8.03, 1.0)       # 1 Billion tokens -> 48.18 ExaFLOPs
sft_flops = compute_training_flops(8.03, 0.01024)   # 10.24M tokens -> 0.49 ExaFLOPs

Execution produces the exact, verified numerical outputs:

=== Running Topic 08: CPT vs SFT Verification ===
[*] CPT Supervised: 9/9 tokens (Mask ratio: 0.0%)
[*] SFT Supervised: 3/9 tokens (Mask ratio: 66.7%)
[*] CPT on 1B tokens (8B model): 48.18 ExaFLOPs
[*] SFT on 10K examples (10.24M tokens, 8B model): 0.49 ExaFLOPs
[*] Compute ratio CPT/SFT: 97.7x compute multiplier
=== All Topic 08 Verifications Passed Cleanly ===

---

Key takeaways

• Continued Pretraining (CPT) optimizes next-token loss across 100% of tokens; Supervised Fine-Tuning (SFT) masks user prompt tokens (label = -100).
• Training FLOPs follow C ≈ 6ND: CPT on 1B tokens requires 48.18 ExaFLOPs for an 8B model, 97.7x more compute than 10K SFT demonstrations (0.49 ExaFLOPs).
• Gekhman et al. (2024) proved that forcing models to learn unseen factual knowledge via SFT severely inflates hallucination rates across unrelated tasks.
• Use CPT for domain vocabulary and DSL grammar; use SFT for format, role adherence, and tone; use RAG for volatile, dynamic facts.
• Always incorporate a 10-20% general replay buffer during CPT to prevent catastrophic forgetting of general reasoning capabilities.

Try it yourself

1. 01Calculate the training FLOPs (using C ≈ 6ND) required to adapt a 7B parameter model across 500 Million tokens of domain telemetry during Continued Pretraining.

   Show a hint

   C ≈ 6 * (7 * 10^9) * (500 * 10^6) = 2.1 * 10^19 FLOPs = 21.0 ExaFLOPs.

2. 02Why does standard SFT with a prompt mask ratio of 75% require fewer backward-pass FLOPs per step than CPT on identical batch sizes?

   Show a hint

   Tokens with label = -100 bypass cross-entropy loss computation, and while the forward pass evaluates all tokens, gradient backpropagation targets only active assistant parameters.

3. 03An engineering team proposes fine-tuning an 8B model daily on the latest infrastructure IP inventory to eliminate RAG latency. What will happen?

   Show a hint

   The model will suffer severe hallucination inflation (Gekhman et al., 2024), catastrophic forgetting of general reasoning, and high continuous training costs without accuracy guarantees.

Interview questions

What is the fundamental difference between the loss formulations of Domain-Adaptive Pretraining (DAPT) and Supervised Fine-Tuning (SFT)?

DAPT uses unmasked autoregressive cross-entropy loss across 100% of tokens in the corpus, training the model to internalize the domain vocabulary, token frequencies, and grammar. SFT applies a mask (ignore_index = -100) to prompt tokens, restricting gradient computation exclusively to the assistant completion tokens to teach task execution and format adherence.

How does the 6ND rule inform the decision between CPT and SFT when planning GPU compute budgets?

Because CPT requires hundreds of millions or billions of tokens (D = 10^8 to 10^10) while SFT requires only thousands of curated demonstrations (D = 10^6 to 10^7), the 6ND formula reveals that CPT requires roughly 50x to 100x more FLOPs. A team must budget multi-node GPU clusters for days for CPT, whereas SFT can typically run on a single multi-GPU node in a few hours.

Why does fine-tuning on new factual information cause hallucination, and how should an architecture resolve this?

As established by Gekhman et al. (2024), forcing gradient updates on factual associations absent from pretraining teaches the model to confabulate plausible completions without epistemic calibration. This degrades performance broadly. The system design solution is to reserve fine-tuning for behavior and format alignment, while injecting factual knowledge dynamically via Retrieval-Augmented Generation (RAG).

What is catastrophic forgetting during Continued Pretraining, and how is it mitigated?

Catastrophic forgetting occurs when gradient updates on a specialized domain distribution overwrite the model weights that encode general world knowledge, math, and coding capabilities. It is mitigated by mixing a 10% to 20% general replay dataset (e.g. from the original pretraining distribution or diverse instruction corpora) into the domain training stream, and using small learning rates with warmup and cosine decay.

Related lessons

• Fine-Tuning & Domain-Specific Models · Decide Whether to TrainDiagnose the Gap Before Changing Weights12 min
• Fine-Tuning & Domain-Specific Models · Decide Whether to TrainSelect a Base Model for the Task13 min
• Fine-Tuning & Domain-Specific Models · Decide Whether to TrainSet a Baseline and Success Threshold12 min
• Fine-Tuning & Domain-Specific Models · Make Training Data TrustworthyFormat Conversations and Mask the Loss13 min
• Fine-Tuning & Domain-Specific Models · Make Training Data TrustworthyUse Synthetic and Teacher Data Carefully13 min