Format Conversations and Mask the Loss

Section 1: The Practical Outcome First

In pre-training, a language model calculates next-token loss across every single token in a text document. In supervised fine-tuning (SFT), computing loss across the entire sequence is an expensive mistake: the model expends gradient updates memorizing the wording of instructions, telemetry formatting, and user questions rather than learning how to reason and respond.

Prompt loss masking aligns input token IDs with target label IDs, assigning the PyTorch sentinel value -100 to all prompt tokens, delimiter tokens, and padding tokens. Loss and gradients are accumulated exclusively on the assistant’s response.

Look at the token-level transformation for an operational incident triage conversation:

[System Turn - 13 tokens]
Input:  <|im_start|>system\nYou are an SRE incident triage assistant.<|im_end|>\n
Labels: [-100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100] (100% MASKED)

[User Turn - 15 tokens]
Input:  <|im_start|>user\nPostgres connection pool exhausted on cluster-02.<|im_end|>\n
Labels: [-100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100] (100% MASKED)

[Assistant Turn - 18 tokens]
Input:  <|im_start|>assistant\n{"subsystem": "database", "severity": "P1"}<|im_end|>\n
Labels: [-100, -100, 24, 88, 12, 59, 14, 91, 3, -100]
         ^^^^ Prefix Delimiters MASKED (-100)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Target JSON SUPERVISED (active token IDs)
                                                 ^^^ <|im_end|> SUPERVISED (CRITICAL stop token)

In this 46-token example:

• 38 tokens (82.6%) are masked with label = -100. They provide causal context in the forward pass but generate zero gradient in the backward pass.
• 8 tokens (17.4%) are supervised. The model updates its weights strictly to predict the triage output and its terminating stop token.

In production

Why Assistant-Only Loss Matters: Training on unmasked prompts encourages the model to memorize system prompts and user queries. If your prompt includes dynamic telemetry, unmasked loss causes the model to develop spurious priors about specific timestamps, pod IDs, and IP addresses. Assistant-only loss ensures 100% of the optimization budget teaches response quality.

---

Section 2: Chat Templates and Boundary Tokens

A chat template converts structured conversation dictionaries ([{"role": "user", "content": "..."}]) into a single tokenizable string. Every foundation model architecture enforces unique reserved tokens to distinguish roles:

1. ChatML (Qwen 2.5, OpenHermes, Yi)

• Turn Prefix Delimiter: <|im_start|>role\n
• Turn Stop Token: <|im_end|>\n

<|im_start|>system
You are an SRE incident triage assistant.<|im_end|>
<|im_start|>user
Postgres connection pool exhausted.<|im_end|>
<|im_start|>assistant
{"subsystem": "database", "severity": "P1"}<|im_end|>

2. Llama 3 / 3.1 (Meta Llama 3 8B, 70B)

• Turn Prefix Delimiter: <|start_header_id|>role<|end_header_id|>\n\n
• Turn Stop Token: <|eot_id|>

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

You are an SRE incident triage assistant.<|eot_id|><|start_header_id|>user<|end_header_id|>

Postgres connection pool exhausted.<|eot_id|><|start_header_id|>assistant<|end_header_id|>

{"subsystem": "database", "severity": "P1"}<|eot_id|>

3. Gemma 2 (Google Gemma 2 2B, 9B, 27B)

• Turn Prefix Delimiter: <start_of_turn>role\n
• Turn Stop Token: <end_of_turn>\n

<bos><start_of_turn>user
Postgres connection pool exhausted.<end_of_turn>
<start_of_turn>model
{"subsystem": "database", "severity": "P1"}<end_of_turn>

4. Mistral v3 (Mistral 7B v0.3, Mixtral)

• Turn Prefix Delimiter: [INST]
• Turn Stop Token: [/INST] and </s>

<s>[INST] Postgres connection pool exhausted. [/INST] {"subsystem": "database", "severity": "P1"}</s>

The Tokenizer Trap

Never format conversation strings with raw string formatting (such as f"<|im_start|>{role}...").

Tokenizers map multi-character special tokens to specific reserved vocabulary IDs (for example, Llama 3 maps <|eot_id|> to token ID 128009). If you concatenate strings manually and tokenize with standard settings, the tokenizer may break the special sequence into multiple subword pieces (such as < + | + eot + _ + id + |>), destroying the architectural boundary markers.

Always use the Hugging Face tokenizer API:

formatted_prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=False,
)

---

Section 3: The Prompt Loss Masking Mechanism

Token Alignment & Loss Masking Architecture

INPUT IDS (X)

System Turn (13 tok)

User Telemetry (15 tok)

Prefix

Assistant JSON (7 tok)

EOS

TARGET LABELS (Y)

All -100 (Masked)

All -100 (Masked)

-100

Y_i = X_i (Supervised)

Active

Zero gradient contribution

Gradient updates active weights

Teaches model to stop!

Token alignment and loss masking architecture for a multi-turn conversation

Read this diagram as text

System and User turns have their target labels set to -100, excluding them from PyTorch CrossEntropyLoss. The Assistant turn masks the prefix header, supervises the JSON completion tokens, and crucially supervises the closing <|im_end|> stop token.

Mathematical Derivation of Masked Cross-Entropy

Given an input sequence of tokens X = (x_1, x_2, ..., x_L) and an aligned target label sequence Y = (y_1, y_2, ..., y_L) where y_i in {1, ..., V} U {-100}:

L(X, Y) = - (1 / N_active) * sum_{i=1}^L [ y_i != -100 ] * log P(x_i = y_i | x_1, ..., x_{i-1})

Where:

• N_active = sum_{i=1}^L [ y_i != -100 ] is the number of active supervised tokens.
• [ y_i != -100 ] is the indicator function evaluating to 1 when unmasked, and 0 when masked.
• In PyTorch, torch.nn.CrossEntropyLoss(ignore_index=-100, reduction='mean') implements this exact formula.

When computing gradients via backpropagation:

dL / d z_{i, k} = (1 / N_active) * (softmax(z_i)_k - delta_{k, y_i})    if y_i != -100
dL / d z_{i, k} = 0                                                     if y_i == -100

Where z_i represents the model logits at sequence position i. The gradient at every masked position is strictly zero.

---

Section 4: The Critical Termination Token Failure Mode

The single most frequent regression in custom fine-tuning pipelines is accidentally masking the assistant’s stop token (<|im_end|> in ChatML, <|eot_id|> in Llama 3, or </s> in Mistral).

Why the Infinite Loop Happens

During generation, decoding continues until the model samples the stop token or hits max_new_tokens.

If your data pipeline sets label = -100 on the turn suffix delimiter, the loss function never penalizes the model for failing to emit the stop token. The model learns to predict JSON keys and values, but the probability assigned to <|eot_id|> remains infinitesimal compared to whitespace or punctuation tokens.

[Inference Output with Masked Stop Token]
Model output:
{"subsystem": "database", "severity": "P1"}
\n\nUser: Can you also check Redis?\nAssistant: Yes, checking Redis cluster now...
\n\nUser: Any other issues?\nAssistant: None reported.
... [Generates until max_tokens=2048 is exhausted]

The model hallucinates phantom conversational turns, wasting inference latency and breaking downstream JSON parsers.

Common mistake

The Off-by-One Delimiter Trap: When building your data collator, ensure the slicing logic isolates the prefix delimiter from the suffix delimiter. Mask the prefix (<|start_header_id|>assistant<|end_header_id|>\n\n), but keep the suffix stop token (<|eot_id|>) strictly supervised (label = token_id).

---

Section 5: Sequence Packing and Cross-Contamination

In SFT workloads, example lengths vary wildly: a short health-check triage may take 60 tokens, while a multi-service cascaded failure log takes 1,800 tokens.

Padding Waste vs Packing Efficiency

If you pad batches to a fixed length of 2,048 tokens with batch size 8, padding tokens typically comprise 65% to 80% of all processed tokens. The GPU spends memory bandwidth and tensor core cycles calculating self-attention over useless padding masks.

Sequence Packing concatenates multiple short conversations into a single fixed-length sequence:

Unpacked Batch (Wasteful):
Sample 1: [Tokens (46)] [PAD ... PAD (2002)] -> 97.7% wasted
Sample 2: [Tokens (110)] [PAD ... PAD (1938)] -> 94.6% wasted

Packed Sequence (Efficient):
Buffer: [Doc 1 (46 tok)] [Doc 2 (110 tok)] [Doc 3 (85 tok)] ... [Doc K (120 tok)]
Zero padding tokens required. GPU utilization increases by 2.5x to 4x.

The Cross-Contamination Threat

However, naive sequence packing introduces a severe failure mode under standard causal attention:

Attention Masking in Packed Sequences

Standard Causal (LEAKAGE)

Doc 1 Attn

Doc 2 Attn

Cross-Bleed!

Doc 2 attends to Doc 1 context

Block-Diagonal (ISOLATED)

MASKED (-inf)

Doc 1 Only

Doc 2 Only

Zero cross-document leakage

Standard causal attention leakage versus block-diagonal document isolation in packed sequences

Read this diagram as text

Under standard causal attention, tokens in Document 2 attend to tokens in Document 1, causing cross-contamination. Block-diagonal masking restricts attention strictly to tokens within the same document ID using cu_seqlens.

Under standard lower-triangular causal attention, token i in Document 2 attends to all preceding tokens in Document 1. The model learns false statistical correlations across unrelated incidents (for example, assuming that if a Postgres alert occurred earlier, a subsequent Redis alert must also be caused by connection limits).

The Solution: Block-Diagonal Masks and cu_seqlens

To eliminate cross-contamination, attention must be restricted so tokens in document A can only attend to tokens with the identical document ID:

M_{i, j} = 0          if j <= i and doc_id(i) == doc_id(j)
M_{i, j} = -infinity  otherwise

In modern kernels like FlashAttention-2 (Dao, 2023), this is implemented natively without constructing sparse 2D attention matrices by passing cumulative sequence lengths (cu_seqlens):

# Two packed incident examples of length 46 and 46
cu_seqlens = torch.tensor([0, 46, 92], dtype=torch.int32)
# FlashAttention-2 computes attention strictly within offsets [0, 46) and [46, 92)

---

Section 6: Worked Code Example & Verifier Walkthrough

The companion script scripts/verify-loss-masking.py implements the complete formatting, loss masking, cross-entropy calculation, and sequence packing pipeline.

6.1 Tokenization and Label Masking

def build_masked_training_example(
    messages: List[Dict[str, str]], tokenizer: SimpleVocabularyTokenizer
) -> Tuple[List[int], List[int], List[str]]:
    input_ids = []
    labels = []
    token_roles = []

    for msg in messages:
        role = msg["role"]
        content = msg["content"]

        prefix_ids = tokenizer.encode(f"<|im_start|>{role}\n")
        content_ids = tokenizer.encode(content)
        suffix_ids = tokenizer.encode("<|im_end|>\n")

        turn_ids = prefix_ids + content_ids + suffix_ids
        input_ids.extend(turn_ids)

        if role == "assistant":
            # Mask prefix delimiter (<|im_start|>assistant\n)
            turn_labels = [-100] * len(prefix_ids)
            # Supervise assistant content
            turn_labels.extend(content_ids)
            # Supervise closing stop token (<|im_end|>)! Mask trailing newline.
            turn_labels.extend([suffix_ids[0]] + [-100] * (len(suffix_ids) - 1))
        else:
            # System and User turns are 100% masked
            turn_labels = [-100] * len(turn_ids)

        labels.extend(turn_labels)

    return input_ids, labels, token_roles

6.2 Masked Cross-Entropy Loss

def compute_masked_cross_entropy(
    logits: List[List[float]], labels: List[int]
) -> Tuple[float, int]:
    total_loss = 0.0
    active_tokens = 0

    for step_logits, target_label in zip(logits, labels):
        if target_label == -100:
            continue

        # Softmax over vocabulary
        max_logit = max(step_logits)
        exp_sum = sum(math.exp(l - max_logit) for l in step_logits)
        log_prob = (step_logits[target_label] - max_logit) - math.log(exp_sum)

        total_loss -= log_prob
        active_tokens += 1

    mean_loss = total_loss / active_tokens if active_tokens > 0 else 0.0
    return mean_loss, active_tokens

6.3 Running the Verifier

Run the verifier script to validate token counts and packing offsets:

python3 scripts/verify-loss-masking.py

Expected output:

=== Running Topic 05: Loss Masking & Formatting Verifier ===
[*] Total Tokens Encoded: 46
[*] Total Labels Generated: 46
[*] Masked Tokens (label=-100):   38 (82.6%)
[*] Supervised Tokens (assistant): 8 (17.4%)
[+] Prompt Loss Masking Assertion Passed: System and User turns strictly masked.
[*] Active Supervised Tokens for Loss: 8
[*] Cross-Entropy Loss: 0.0016
[+] Masked Cross-Entropy Loss Computation Passed.
[*] Sequence Packing Demo: Packed 2 conversations.
[*] cu_seqlens: [0, 46, 92]
[+] Sequence Packing & cu_seqlens Verification Passed.
=== All Topic 05 Verifications Passed Cleanly ===

---

Key takeaways

• Prompt loss masking assigns label=-100 to all system, user, and formatting tokens, accumulating loss exclusively on assistant completions.
• Training on unmasked prompts forces the model to memorize prompt syntax and user queries rather than learning domain target reasoning.
• The assistant stop token (<|im_end|> or <|eot_id|>) must be supervised — masking it causes runaway generations and infinite decoding loops.
• Always use tokenizer.apply_chat_template() rather than manual f-string concatenation to preserve special token IDs.
• Sequence packing concatenates examples to eliminate padding waste, but requires block-diagonal attention masks or FlashAttention cu_seqlens to prevent cross-document contamination.
• PyTorch CrossEntropyLoss(ignore_index=-100) automatically normalizes loss by the count of active unmasked tokens.

Try it yourself

1. 01A team fine-tunes Llama 3 on customer support chats and finds the model generates the correct answer but never stops, hallucinating follow-up user questions until reaching max_tokens. What is the root cause in their data collator?

   Show a hint

   Inspect how the turn suffix delimiter is sliced. If <|eot_id|> is set to label=-100, the model receives zero gradient signal for emitting the termination token.

2. 02Calculate the active token count and masked percentage for a conversation with 80 system tokens, 240 user tokens, 4 prefix delimiter tokens, 60 assistant tokens, and 1 stop token.

   Show a hint

   Total tokens = 80 + 240 + 4 + 60 + 1 = 385. Active tokens = 60 + 1 = 61. Masked tokens = 324 (84.2%).

3. 03Explain why standard lower-triangular causal attention causes hallucinations when training on packed sequences of multiple incident tickets.

   Show a hint

   In standard causal attention, tokens in document 2 attend to all tokens in document 1, learning spurious cross-ticket correlations. Block-diagonal masking isolates each document.

4. 04In PyTorch CrossEntropyLoss with reduction="mean", does the divisor N represent total sequence length L or the count of active tokens where target != -100?

   Show a hint

   PyTorch normalizes strictly by the count of non-ignored targets (N_active). If all tokens are -100, it returns 0.0.

Interview questions

What is prompt loss masking, and why is it standard practice in Supervised Fine-Tuning (SFT)?

Prompt loss masking sets the target labels for system and user tokens to -100 (PyTorch ignore_index), ensuring loss and gradients are only computed on assistant response tokens. This prevents the model from wasting parameter capacity memorizing user prompts or variable formatting, focusing 100% of weight updates on target completion behavior.

What happens if you accidentally set the label for the assistant EOS/EOT token to -100 during training?

The model receives zero loss penalty for failing to output the stop token. In production inference, the model generates the expected answer but continues generating indefinitely — hallucinating extra questions or repeating phrases — until it hits the hard max_tokens limit.

How does sequence packing improve training throughput, and what mechanism prevents cross-document data leakage?

Sequence packing concatenates multiple short variable-length examples into a single fixed context buffer, eliminating padding tokens that typically consume 60-80% of training batches. To prevent cross-document contamination where document B attends to document A, block-diagonal attention masking or FlashAttention-2 cu_seqlens are used to isolate attention within document boundaries.

Why should you use tokenizer.apply_chat_template() instead of Python string formatting when preparing conversational training data?

Foundation models use special reserved tokens (such as Llama 3 <|start_header_id|> or Qwen <|im_start|>) that map to specific single token IDs in the tokenizer vocabulary. Raw string concatenation can cause the tokenizer to break these special delimiters into multiple subword tokens, corrupting the prompt boundaries.

Related lessons

• Fine-Tuning & Domain-Specific Models · Decide Whether to TrainDiagnose the Gap Before Changing Weights12 min
• Fine-Tuning & Domain-Specific Models · Decide Whether to TrainSelect a Base Model for the Task13 min
• Fine-Tuning & Domain-Specific Models · Decide Whether to TrainSet a Baseline and Success Threshold12 min
• Fine-Tuning & Domain-Specific Models · Make Training Data TrustworthyTurn Real Tasks Into Training Examples14 min