Multi-Token Prediction: Objectives and Tradeoffs

Multi-Token Prediction: Objectives and Tradeoffs

Multi-token prediction (MTP) asks a model to predict more than the next token at each training position. That changes the training signal: the model must plan representations that are useful for multiple future positions, not just the immediate one. Whether keeping the extra prediction machinery for inference is worthwhile is a separate decision. This lesson keeps those two evaluations apart.

The one-sentence version: At a training position, MTP computes additional future-token cross-entropy terms through extra heads or sequential modules. The total training objective is a weighted combination of the main next-token loss and those extra terms. Keeping the extra machinery at inference is optional; DeepSeek-V3 discards it by default, and any speed claim requires an explicit verification procedure separate from the training loss.

What MTP is — and what it is not

Standard language modeling minimizes a single cross-entropy loss at each position:

L_1 = - Σ_t log P_θ(x_{t+1} | x_{1:t})

This is the next-token objective (Gloeckle et al., 2024, §2, eq. 1). MTP extends it by predicting n future tokens at once. In Meta’s formulation, that translates to n independent heads operating in parallel on the same trunk representation (Gloeckle et al., 2024, §2, eq. 2; Figure 1). DeepSeek-V3 uses sequential MTP modules instead: depth k predicts token t_{i+k+1} using a transformed representation that combines the previous depth’s output with the embedding of the future token (DeepSeek-V3, arXiv:2412.19437, §2.2, eqs. 21-23). These are different mechanisms and should not be treated as identical.

Common mistake

MTP is not speculative decoding. The extra predictions can be reused as a draft for speculative decoding, but speculative decoding also requires a verification step (accept/reject against the base model’s greedy output) and a separate acceptance algorithm. Training loss alone does not make decoding faster.

Two architectures, clearly separated

Feature	Meta (Gloeckle et al., 2024)	DeepSeek-V3 (§2.2)
Prediction direction	n independent heads in parallel	D sequential modules with full causal chain
Shared components	Shared trunk f_s; independent head layers f_{h_i}; shared unembedding f_u (§2)	Shared embedding Emb(·) and shared output head OutHead(·); separate TRM_k block and projection M_k per module (§2.2)
Causal chain	Independent: each head predicts from the same trunk representation	Sequential: h_i^{k} depends on h_i^{k-1} and Emb(t_{i+k}) (eq. 21)
Inference use	Additional heads can be discarded; only the first head is required for standard autoregressive generation (§2, Figure 1)	MTP modules can be discarded; the main model functions independently (§2.2: “MTP in Inference”)
Speed claim source	Self-speculative decoding measured on tested setups only (§3.2)	Repurposed for speculative decoding; measured quality in ablation is with modules discarded (§4.5.1, Table 4)

The distinction matters for implementation: Meta’s independent heads mean each extra prediction uses its own transformer layer on the same trunk output. DeepSeek-V3’s sequential modules mean the k-th module consumes the output of depth k-1 and the embedding of the future token at position i+k, maintaining the full causal chain (DeepSeek-V3 §2.2, eq. 21; Figure 3).

A labelled toy sequence: computing the MTP loss

Consider the toy sequence below. Positions are 1-based; vocabulary size V is small and arbitrary (V = 8). The values are chosen only so the arithmetic is checkable.

Position:  1    2    3    4    5
Token:   t_1  t_2  t_3  t_4  t_5

We evaluate MTP at position i = 2 with D = 2 additional depths (two extra future targets). The predictions and targets are:

Prediction module	Predicted token position	Ground-truth token	Stated probability P	Cross-entropy -log P
Main model (next-token)	i + 1 = 3 → t_3	t_3	0.70	-log(0.70) ≈ 0.357
MTP depth k = 1	i + 2 = 4 → t_4	t_4	0.60	-log(0.60) ≈ 0.511
MTP depth k = 2	i + 3 = 5 → t_5	t_5	0.50	-log(0.50) ≈ 0.693

The cross-entropy terms above are computed directly from the stated probabilities. Each term uses the same log-probability formula; the only difference is which module produces the distribution (OutHead applied to h_i^k for depth k, DeepSeek-V3 §2.2, eq. 23).

In production

The arithmetic is independent of any model weights. Changing the stated probabilities changes the terms proportionally. Verify by substituting the values into -log p: -log(0.7) ≈ 0.357, -log(0.6) ≈ 0.511, -log(0.5) ≈ 0.693. The total below is a direct sum of these three numbers, scaled by the defined weight.

Weighted total loss (explicit definition)

For this toy, define the total MTP-augmented loss at position i = 2 as:

L_main      = -log P(main)(t_3)
L_MTP_1     = -log P(1)(t_4)
L_MTP_2     = -log P(2)(t_5)
L_total(i)  = L_main + λ · (1/D) · (L_MTP_1 + L_MTP_2)

Using λ = 1.0 and D = 2:

L_main      ≈ 0.357
(L_MTP_1 + L_MTP_2) / 2  ≈ (0.511 + 0.693) / 2 = 0.602
L_total(i)  ≈ 0.357 + 1.0 · 0.602 = 0.959

This is a defined calculation, not a universal formula from either paper. DeepSeek-V3’s actual training objective averages across all sequence positions and all depths with its own λ (DeepSeek-V3 §2.2, eqs. 24-25). Meta’s formulation uses independent heads with its own weighting (Gloeckle et al., 2024, §2, eq. 2). The arithmetic above is provided so the reader can verify the mechanism independently.

Inference: a separate greedy draft-and-verify example

Keep the training objective (previous section) separate from inference. At inference, the main model produces greedy tokens. If MTP modules are retained, they can provide draft tokens; verification is then required.

Consider a greedy-decoding step at the current position. The base model proposes the next greedy token by selecting the highest-probability token from its output distribution. We use the additional MTP modules to propose three future tokens as a draft:

Draft position	Proposed token	Base model greedy check	Result
+1	A	Matches greedy A	Accepted
+2	B	Matches greedy B	Accepted
+3	C	Greedy output is D (not C)	Rejected

Under greedy verification (accept if proposed token equals the base model’s greedy next token; reject otherwise), the verification rule is:

Verification rule (greedy assumption): For each proposed token at offset k, compute the base model’s greedy token at that same sequence position. If they match exactly, accept; otherwise reject the proposal and all subsequent ones in the same verification step.

In this example, the base model accepts the first two proposed tokens (A and B) and rejects the third (C). The number of accepted draft tokens is 2.

The tradeoff

Accepted tokens per verification step are not a measured latency speedup. The number of accepted tokens depends on proposal accuracy, batch size, sequence length, and the verification algorithm. Meta’s reported speed results are from its tested self-speculative setup only (§3.2, Table S2/S3). DeepSeek-V3 reports ablation quality with MTP modules discarded (§4.5.1, Table 4); it does not report universal speed numbers for ordinary inference. Any speed claim requires measuring end-to-end latency under the exact serving conditions.

Also note: this verification procedure uses greedy decoding. Generalizing to sampling requires a separate acceptance algorithm (for example, computing acceptance ratios from the proposal and target distributions) that is not covered here. Without that algorithm, the verification rule above does not apply to sampling.

What the evidence actually supports

Keep training evidence and inference evidence in separate categories.

Training-quality evidence (what the papers measure)

• DeepSeek-V3 §2.2 defines the MTP mechanism (eqs. 21-25), the sequential causal chain, and the shared embedding/output head design.
• DeepSeek-V3 §2.2 states that MTP modules can be discarded at inference; the main model functions independently.
• DeepSeek-V3 §4.5.1 (Table 4) reports ablation results for MTP. The ablation compares configurations with and without the MTP objective, and the quality evaluation is performed with the MTP modules discarded for ordinary inference. The exact improvement depends on the benchmark and the model scale in that table; do not treat any single number as universal.
• DeepSeek-V3 §4.4.1 defines the evaluation benchmarks used in the paper (e.g., MMLU, MMLU-Pro, GPQA, code/math benchmarks). It should be used only for understanding which benchmarks are referenced, not for making universal quality claims.

Inference-speed evidence (separate from training quality)

• DeepSeek-V3 §2.2 states MTP modules can be “repurposed” for speculative decoding, but does not present universal speed measurements for ordinary inference.
• Meta (§3.2) reports measured self-speculative speedups on code and text for its tested 4-token prediction model (7B parameters, greedy decoding). The reported figures (e.g., ~3× on code, ~2.7× on text) are specific to that setup and batch-size configuration. They should not be generalized without reproducing the measurement conditions.
• Meta §3.7 reports that multi-token prediction benefits are more pronounced on generative code tasks than on standard multiple-choice natural-language benchmarks; the 4-token predictor regresses slightly on some standard NLP choice tasks.

Common misattributions to avoid

• Do not cite DeepSeek-V3 section 1.3 for the MTP mechanism. The mechanism is defined in §2.2 (eqs. 21-25). Section 1.3 does not contain the MTP mechanism.
• Do not claim that the MTP training loss alone makes decoding faster. The loss changes the learned representations; faster decoding requires an explicit verification procedure that is separate from the loss definition.
• Do not claim universal quality gains. DeepSeek-V3’s ablation (§4.5.1) measures quality for its own model family; Meta’s results (§3.1, Figure 3) show scale-dependent improvements. Both are measured results, not universal guarantees.
• Do not describe Meta’s independent heads and DeepSeek’s sequential modules as the same mechanism. They share the idea of predicting future tokens, but the architecture, causal chain, and inference behavior are different.

Tradeoffs and serving cost

Adding MTP machinery increases training compute and memory use (extra modules or heads, extra loss terms, extra embeddings combined per depth for DeepSeek-V3). At inference:

• DeepSeek-V3 can discard the MTP modules and serve only the main model (§2.2). That means no additional serving memory or compute overhead if the modules are not retained.
• If the modules are retained for speculative decoding, the serving system must run both the base model and the verification logic, and must implement the greedy-decoding assumption (or a sampling acceptance algorithm, which is not covered here). The accepted tokens count depends on proposal accuracy, which is measured in Meta’s setup (§3.2) but not guaranteed by the training loss.
• Meta’s independent heads can be discarded individually; only the first head is required for standard autoregressive generation (§2, Figure 1). Keeping additional heads adds serving cost that must be justified by measured speed gains under the target batch and sequence conditions.

Exit task: calculate and assess separately

The following exercise requires two independent steps: (a) a loss calculation using stated probabilities, and (b) an inference assessment using the verification rule defined above. Do not mix the arithmetic from (a) with the speed claim from (b).

Exercise: MTP loss arithmetic and draft verification

1. 01Using the toy sequence at position i = 2 (targets t_3, t_4, t_5) and stated probabilities P(main) = 0.70, P(1) = 0.60, P(2) = 0.50, compute L_main, L_MTP_1, L_MTP_2, and L_total with λ = 1.0, D = 2 using the formula in the lesson. Confirm the arithmetic independently.

2. 02In the greedy draft-and-verify example, the base model accepts A and B but rejects C. State how many tokens are accepted, restate the greedy verification rule, and explain why this count is not a measured latency speedup.

3. 03Identify one training-quality claim in DeepSeek-V3 that relies on §4.5.1 / Table 4 (measured with MTP modules discarded) and one benchmark-definition reference that relies on §4.4.1 only. Do not mix the two categories.

Related lessons

• Lesson 17 (multi-head-latent-attention): DeepSeek architecture, sequential modules, and inference optimization patterns.
• Lesson 3 (preparing-training-sequences): cross-entropy loss and training-target mechanics that MTP extends.
• Lesson 1 (transformer-architecture): base transformer representation and output projection used by both the main model and MTP heads.

Related lessons

• LLM Internals & Architecture · Modern LLM architecturesMulti-Head Latent Attention: DeepSeek's KV-Cache Design14 min
• LLM Internals & Architecture · Transformer architecture and model inputsPreparing Training Sequences: Shift, Pad, and Pack15 min
• LLM Internals & Architecture · Transformer architecture and model inputsTransformer Architecture Explained: Trace a Decoder-Only LLM14 min