Reasoning Distillation: Teacher Data, Filtering, and Student Evaluation

1. Introduction

Reasoning distillation transfers Chain-of-Thought (CoT) capabilities from a larger, stronger “teacher” model to a smaller “student” model via supervised fine-tuning (SFT). Unlike Reinforcement Learning from Verifiable Rewards (RLVR, covered in lesson 25), which uses outcome-based rewards to optimize policy via sampling, reasoning distillation uses imitation learning on curated reasoning traces. The goal is to give the student the teacher’s reasoning patterns at lower inference cost—though, as the research evidence shows, this comes with important trade-offs.

This lesson covers:

• How teacher reasoning data is generated and filtered
• The role of data selection strategies (difficulty, diversity, verbosity)
• The mixed System-1 / System-2 distillation paradigm
• Evaluation against deployment cost
• Limits of RLVR versus distillation, and where each method excels

---

2. How Teacher Reasoning Data is Built

The standard distillation pipeline proceeds in three phases:

2.1 Teacher Generation

A strong reasoning model (e.g., DeepSeek-R1, QwQ-32B, or a frontier LLM with long CoT capability) generates reasoning traces on a broad question pool. The teacher prompt typically requires the model to produce a chain-of-thought before the final answer, e.g.:

<think> intermediate reasoning here </think>
<answer> final answer here </answer>

The initial generations form a candidate pool. Annotation and selection decide which examples enter student training; a final-answer check alone would not validate every intermediate step.

2.2 Filtering and Selection

Not all generated traces are equally useful for distillation. Three primary selection strategies exist:

Strategy	What it filters	Typical use
Difficulty	Short traces, easy questions	Emphasizes hard problems; “Long” subset downweights short CoTs
Diversity	Redundant strategies / uniform topic coverage	Selects by reasoning strategies, question embeddings, or topic domains
Verbosity	Traces selected by an annotated verbosity score	Compares low, medium, and high subsets; results vary by benchmark

See §5.1 of the NaturalThoughts paper [Li et al., 2025] for systematic ablations.

2.3 Mixed System-1 / System-2 Distillation

A key recent advance is mixed distillation, where training data is a combination of:

• System-2: Full CoT trace (reasoning + answer)
• System-1: Only the final answer (no intermediate reasoning)

Two mixing approaches exist:

• Random mixing: Each training example is System-2 with probability p ∈ 0.6
• Difficulty-based mixing: System-2 for questions identified as “difficult” (e.g., via teacher-model disagreement); System-1 otherwise

At inference time, the student can operate in three modes:

Mode	Instruction	Typical response length
No-Think	“Answer directly without thinking.”	Very short (System-1 only)
Think	“Think carefully before answering. Use about K words.” + <think> token	Long (full System-2)
Adaptive-Think	“Think carefully before answering.” (no <think> forced)	Dynamic, adapts to question difficulty

In NaturalThoughts [Li et al., 2025, §5.3, Table 4], difficulty-based mixing used 36% System-2 training responses. In Think mode on GPQA-Diamond it scored 38.9%, 1.3 percentage points above pure System-2 training’s 37.6%; other inference modes have different accuracy and length tradeoffs.

---

3. Data Selection Strategies in Practice

3.1 Difficulty-Based Selection (Models Disagreement)

Method: Compare the teacher’s long CoT response with a non-reasoning model’s short answer; where they disagree, the question is flagged as “difficult.”

Empirical finding: The result depends on the student and training size. At 10K examples, Models Disagree beats Random on GPQA-Diamond for Llama-3.1-8B (39.7% versus 37.6%) but trails Random for Qwen-2.5-7B (40.5% versus 41.8%) in NaturalThoughts Tables 1–2.

Interpretation: Model disagreement is a proxy for difficulty, which may select more informative traces. It does not establish that every disagreement is difficult or useful.

3.2 Diversity-Based Selection

Three axes are commonly used:

1. Question topics: Uniform sampling across 12–13 top-level domains (Engineering, Medicine, Law, etc.)
2. Semantic embeddings: Question embeddings clustered; one sample per cluster
3. Reasoning strategies: Downsample examples with fewer than R_min or more than R_max unique strategies; R_min=4, R_max=8 is a typical range

Finding: Reasoning-strategy diversity often outperforms topic or embedding diversity, because the way the model reasons matters more than the subject area.

3.3 Verbosity-Based Selection

Using annotated verbosity scores (0 = very efficient, 10 = excessive rambling):

• Low verbosity: Too sparse; may omit useful intermediate steps
• Medium verbosity: Can help on some benchmarks, but is not uniformly best
• High verbosity: May add unnecessary tokens; its measured effect depends on the benchmark

---

4. Evaluation: Capability vs. Deployment Cost

4.1 Metrics

Metric	What it measures
pass@k (typically pass@1)	Probability at least one sampled output is correct
Inference length (average tokens generated)	Direct deployment cost proxy
Accuracy–efficiency tradeoff	Accuracy at a given response-length budget

4.2 Typical Results (at 10K training examples)

Student (Model)	Selection Method	Training Size	GPQA-Diamond (pass@1)	MATH-500 (pass@1)	MMLU-Pro (pass@1)	SuperGPQA (pass@1)	Notes
Llama-3.1-8B-Instruct	Random	10K NT	37.6±0.6	61.3±0.3	57.8	28.8	NT Table 1
Llama-3.1-8B-Instruct	Models Disagree	10K NT	39.7±0.4	61.9±0.3	59.1	29.7	NT Table 1 Difficulty
Llama-3.1-8B-Instruct	Long	10K NT	39.1±0.7	63.5±0.3	58.6	28.8	NT Table 1 Difficulty
Llama-3.1-8B-Instruct	Models Agree	10K NT	37.5±0.6	60.1±0.3	56.7	28.9	NT Table 1 Difficulty
Llama-3.1-8B-Instruct	Verbosity=Med	10K NT	37.2±0.6	62.1±0.4	57.5	28.9	NT Table 1 Difficulty
Llama-3.1-8B-Instruct	Reasoning Strategies	10K NT	38.8±0.5	63.5±0.4	58.2	28.8	NT Table 1 Diversity
Qwen-2.5-7B-Instruct	Random	10K NT	41.8±0.6	79.0±0.2	59.7	31.2	NT Table 2
Qwen-2.5-7B-Instruct	Models Disagree	10K NT	40.5±0.5	78.4±0.3	61.7	32.7	NT Table 2 Difficulty
Qwen-2.5-7B-Instruct	Long	10K NT	40.0±0.6	78.8±0.2	61.8	33.5	NT Table 2 Difficulty
Qwen-2.5-7B-Instruct	Reasoning Strategies	10K NT	42.2±0.6	78.6±0.2	61.5	32.1	NT Table 2 Diversity

At larger scales (reported rows only):

Student (Model)	Selection Method	Training Size	GPQA-Diamond (pass@1)	MATH-500 (pass@1)	MMLU-Pro (pass@1)	SuperGPQA (pass@1)	Notes
Llama-3.1-8B-Instruct	Models Disagree	100K NT	45.2±0.6	70.2±0.2	59.8	32.2	NT Table 1
Llama-3.1-8B-Instruct	Long	100K NT	43.1±0.7	69.0±0.3	61.2	32.2	NT Table 1
Llama-3.1-8B-Instruct	Reasoning Strategies	100K NT	44.1±0.6	67.7±0.3	61.2	31.8	NT Table 1
Llama-3.1-8B-Instruct	Random	500K NT	48.3±0.6	72.3±0.3	61.9	31.3	NT Table 1
Qwen-2.5-7B-Instruct	Models Disagree	100K NT	43.7±0.6	80.8±0.2	62.8	33.9	NT Table 2
Qwen-2.5-7B-Instruct	Long	100K NT	43.4±0.4	81.7±0.4	62.2	34.0	NT Table 2
Qwen-2.5-7B-Instruct	Reasoning Strategies	100K NT	43.5±0.5	80.9±0.2	62.2	33.1	NT Table 2
Qwen-2.5-7B-Instruct	Random	500K NT	48.6±0.6	83.1±0.2	62.3	35.2	NT Table 2
Llama-3.3-70B-Instruct	Random	1k NT	63.5±0.6	86.3±0.2	78.0	46.0	NT Table 3
Llama-3.3-70B-Instruct	Random	10k NT	65.6±0.6	87.4±0.3	78.4	48.3	NT Table 3
Llama-3.3-70B-Instruct	Random	100k NT	67.6±0.4	88.5±0.2	78.9	50.6	NT Table 3

The larger model benefits more from scale, but the selection strategy still matters.

4.3 Efficiency–Accuracy Tradeoff

Difficulty-based mixed distillation (System-2 for hard, System-1 for easy questions) achieves:

• +1.3 percentage points in Think mode on GPQA-Diamond over pure System-2 distillation [Li et al., 2025, §5.3, Table 4]
• 36% System-2 in training (vs. 60% in random mixing), yielding better accuracy-efficiency tradeoff than random mixing
• At inference, the student can adapt: No-Think for easy questions, Think for hard ones

The tested mixed model offered a wider range of response lengths across the three inference instructions; accuracy must be compared within the same mode.

The mixed distillation efficiency findings from NaturalThoughts [Li et al., 2025, §5.3, Table 4] are summarized below. Training with mixed System-1/System-2 data enables flexible test-time compute by interpolating between thinking fast and slow:

Training data	No-Think: accuracy / tokens	Adaptive-Think: accuracy / tokens	Think: accuracy / tokens
System-1 only	34.0±0.6 / 321	32.4±0.7 / 541	Not reported
System-2 only	36.3±0.5 / 5,134	37.3±0.6 / 7,817	37.6±0.6 / 8,741
Random mixing, 20% System-2	31.2±0.7 / 511	31.8±0.6 / 1,056	34.7±0.7 / 8,642
Random mixing, 40% System-2	31.8±0.5 / 847	33.2±0.5 / 1,643	36.7±0.6 / 9,155
Random mixing, 60% System-2	34.1±0.6 / 730	35.3±0.6 / 1,683	37.5±0.5 / 7,140
Difficulty-based mixing, 36% System-2	34.5±0.5 / 799	35.1±0.6 / 2,033	38.9±0.7 / 7,562

Key insight: In Think mode, difficulty-based mixing scored 38.9% at about 7,562 tokens versus 37.6% at about 8,741 tokens for System-2-only training. In No-Think mode, it produced much shorter answers but scored 34.5% versus 36.3% for System-2-only training. These are different operating points, not one universal winner.

Caveat: These values are specific to the NaturalThoughts study using DeepSeek-R1 as teacher and Qwen/Llama students. Results may vary with different teacher–student pairings or evaluation benchmarks.

---

5. RLVR vs. Distillation: Complementary Methods

A critical comparison, grounded in Yue et al. [2025] and the research evidence:

Aspect	RLVR (lesson 25)	Distillation
What it does	Optimizes policy via verifiable rewards; improves sampling efficiency	Imitates teacher CoT via SFT; transfers reasoning patterns
Boundary effect	Does not expand reasoning boundary; pass@k at large k converges to base model [Yue et al., 2025, §3–4.2]	Can expand boundary: distilled models outperform base on MATH-500, AIME [NaturalThoughts Li et al., 2025; Scale-or-Reason Boizard et al., 2025/2026]
Compute at inference	Same model, more samples → higher cost	Smaller model at same cost; or better accuracy at same model size
When it fails	Narrower coverage; RLVR-trained models fail on problems base model solves	Teacher must have genuine reasoning capability; filtering can discard useful traces
Typical data scale	Millions of rollouts; on-policy	1K–500K curated examples; off-policy [NaturalThoughts Li et al., 2025]
Key reference	Yue et al., “Does RL Really Incentivize Reasoning?” (arXiv:2504.13837)	NaturalThoughts [Li et al., 2025, §3–5]; Scale-or-Reason [Boizard et al., 2025/2026]

Takeaway: RLVR improves reliability (pass@1) but not the reasoning boundary. Distillation can expand the boundary by transferring new reasoning patterns from a stronger teacher—provided the teacher actually has them and the data is well-filtered.

---

6. Reward Hacking and Data-Quality Failure Modes

Even with verifiable rewards or careful distillation, failure modes exist:

Mode	Description	Mitigation
Overfitting to filtering criteria	The teacher/model overfits to the selection metric (e.g., “Models Disagree”) without genuine reasoning improvement	Cross-validate with held-out tasks; use multiple selection axes
Teacher reasoning flaws	Teacher itself has sub-optimal patterns (overthinking, underthinking)	Use multiple diverse teachers; cross-feedback between teachers
Verbosity extremes	Very low verbosity omits steps; very high verbosity adds noise	Target medium verbosity; annotate and analyze length distributions
Easy-to-guess answers	Final answer is predictable without reasoning	Include difficulty-based filtering; verify with pass@k, not just pass@1
Distribution shift	Training data domain ≠ deployment domain	Topic/domain-balanced selection; evaluate on held-out benchmarks

---

7. Exit Task: Evaluate a Distilled Student

Prompt: You are given a student model (Qwen2.5-7B-Instruct) that has been distilled from DeepSeek-R1 using 10,000 NaturalThoughts training examples. Two curriculum variants were used:

• Variant A: Random selection of 10K examples
• Variant B: Models-disagreement difficulty filtering, 10K examples

The student is evaluated on GPQA-Diamond (pass@1) and MATH-500 (pass@1) with temperature 0.7, top-p 1.0, max generation 16,384 tokens. The baseline (undistilled Qwen2.5-7B-Instruct) scores 34.1 on GPQA-Diamond and 69.7 on MATH-500. These baseline values are from the undistilled Qwen2.5-7B-Instruct checkpoint before any NaturalThoughts distillation.

Derivation from NaturalThoughts Tables:

Variant	GPQA-Diamond (pass@1)	MATH-500 (pass@1)	Derivation
Variant A (Random 10K NT)	41.8 ± 0.6 [NT Table 2]	79.0 ± 0.2 [NT Table 2]	Qwen-2.5-7B-Instruct, Random 10K NT (NT Table 2)
Variant B (Models-Disagreement 10K NT)	40.5 ± 0.5 [NT Table 2]	78.4 ± 0.3 [NT Table 2]	Qwen-2.5-7B-Instruct, Models Disagree 10K NT (NT Table 2 Difficulty)

Answers to exit task questions:

1. Rank variants by expected pass@1:

   • GPQA-Diamond: Variant A (41.8) > Variant B (40.5) [both from NT Table 2, Qwen-2.5-7B-Instruct]
   • MATH-500: Variant A (79.0) > Variant B (78.4) [both from NT Table 2, Qwen-2.5-7B-Instruct]
   • For Qwen-2.5-7B-Instruct at 10K, Random selection outperforms Models Disagree on both benchmarks (NT Table 2). The Models Disagree advantage appears at larger scales (100K+) for this model.

2. Approximate absolute pass@1 gains over undistilled baseline:

   • GPQA-Diamond: Variant A gain = 41.8 - 34.1 = +7.7 points [NT Table 2 vs. undistilled baseline]
   • MATH-500: Variant A gain = 79.0 - 69.7 = +9.3 points [NT Table 2 vs. undistilled baseline]
   • Note: For Qwen-2.5-7B-Instruct at 10K, the Random subset outperforms Models Disagree.

3. Will mixed System-1/System-2 distillation (as in NT §5.3) enable shorter inference-time responses?

   • Yes, with a quality tradeoff. In No-Think mode, the difficulty-mixed model averaged 799 tokens at 34.5% accuracy; the System-2-only model averaged 5,134 tokens at 36.3%. In Think mode, difficulty mixing averaged 7,562 tokens at 38.9% versus 8,741 tokens at 37.6% for System-2 only. These are like-for-like comparisons within each inference mode from NT Table 4.

Exit task: Distilled student evaluation

1. 01Rank Variant A and B by expected pass@1 on GPQA-Diamond and MATH-500 for Qwen-2.5-7B-Instruct at 10K, with evidence from the research.

   Show a hint

   Use NT Table 2: Random 10K scores 41.8/79.0, Models Disagree 10K scores 40.5/78.4.

2. 02Estimate the pass@1 gain for the best variant on each benchmark.

   Show a hint

   Compare against the undistilled baselines: GPQA-D +7.7, MATH-500 +9.3.

3. 03Will mixed System-1/System-2 distillation enable shorter inference-time responses? What accuracy tradeoff appears?

   Show a hint

   Compare training variants within the same inference mode in NT Table 4: No-Think 799 tokens / 34.5% versus 5,134 / 36.3%; Think 7,562 / 38.9% versus 8,741 / 37.6%.

8. Key Takeaways

Key takeaways

• Reasoning distillation transfers CoT capabilities from a teacher to a student via SFT, unlike RLVR which uses outcome-based policy optimization.
• Data selection strategies (difficulty, diversity, verbosity) significantly impact distillation quality; "Models Disagree" difficulty filtering and reasoning-strategy diversity are among the most effective.
• Mixed System-1 / System-2 distillation enables adaptive inference-time reasoning: the student can switch between short answers and full CoT depending on question difficulty.
• RLVR improves sampling reliability (pass@1) but does not expand the reasoning boundary; distillation can expand it by transferring new reasoning patterns from a stronger teacher.
• RLVR improves reliability (pass@1) but does not expand the reasoning boundary; distillation can expand it by transferring new reasoning patterns from a stronger teacher.
• The accuracy–efficiency tradeoff depends on inference mode: in the NaturalThoughts Think-mode GPQA-Diamond experiment, difficulty mixing improved accuracy by 1.3 percentage points over System-2-only training while using fewer tokens.
• Teacher quality is paramount: if the teacher lacks genuine reasoning patterns, distillation cannot transfer them. Filtering must preserve pedagogical value, not just discard "hard" examples.

9. Sources and Claim-to-Source Mapping

Source	Section	What it supports	Limits / gaps
Li et al., “NaturalThoughts” (arXiv:2507.01921), §3–5	3.2, 3.3, 5.1–5.3	Data selection strategies; mixed System-1/System-2 distillation; inference-time adaptive reasoning; quantitative accuracy-efficiency results on GPQA-D, MATH-500, MMLU-Pro, SuperGPQA	2025 preprint; may be updated; results specific to DeepSeek-R1 teacher + Llama/Qwen students
Boizard et al., “Scale or Reason?” (arXiv:2509.22193 / EMNLP 2026)	§1–7	RLVR vs. distillation compute efficiency; IFT Pareto frontier; curriculum mixing 25–50% reasoning data	2025/2026; compares single-teacher IFT vs. reasoning distillation; open-ended vs. MC tasks differ
Yue et al., “Does RL Really Incentivize Reasoning?” (arXiv:2504.13837), §3–4.2	§5 (RLVR vs. distillation comparison)	RLVR does not expand reasoning boundary; distillation can; pass@k at large k converges to base model	2025; specific to tested algorithms/model families; does not prove future RL cannot expand boundary
Tian et al., “Learning from Committee” (arXiv:2025.findings-acl.217)	Multi-teacher distillation	Step-level quality assessment; peer-review between teachers; CRD components	2025; two-teacher setup; may generalize differently with >2 teachers
Shao et al., “DeepSeekMath” (arXiv:2402.03300), §4.1	GRPO origin; group-relative advantages	Algorithm background for lesson 25 context	One lab’s algorithm; 7B model on GSM8K/MATH; G=4–16

Verification gaps stated:

• The NaturalThoughts results are specific to the DeepSeek-R1 → Llama/Qwen distillation pipeline; other teacher–student pairs may yield different gains.
• The “Models Disagree” difficulty signal depends on having a capable non-reasoning comparator model; without it, the signal is unavailable.
• The Scale-or-Reason findings are based on a single teacher toggling reasoning mode; real-world teacher generation may have additional variables.
• The RLVR-vs-distillation comparison is inferential; no single paper performs a head-to-head under identical conditions.
• Exit task assumptions: 10K examples, Qwen2.5-7B-Instruct student, DeepSeek-R1 teacher; other scales/model sizes will differ.
• No independent technical review has been performed on this lesson; passing build ≠ publication approval.

10. Related Lessons

Related lessons

• LLM Internals & Architecture · Reasoning models and test-time computationHow Reasoning Models Work: Training Paths, Traces, and Cost16 min
• LLM Internals & Architecture · Reasoning models and test-time computationReinforcement Learning for Reasoning: Verifiable Rewards and GRPO14 min
• LLM Internals & Architecture · Reasoning models and test-time computationEvaluating Reasoning: Baselines, Verifiers, and Parsing Failures12 min