Choose a Compact Student Model

Section 1: The Practical Outcome First

In enterprise deployments, the primary motivation for training or distilling a domain model is rarely to beat frontier models on generic benchmarks; it is to meet operational boundaries that frontier models cannot satisfy.

Consider a real-world edge deployment scenario:

• Deployment Target: 5,000 edge Kubernetes worker nodes running inside customer enterprise VPCs or retail stores.
• Hardware Profile: 4-core x86_64 or ARM64 CPU, 8 GB total system RAM.
• Container Allocation: Maximum 2 GB RAM allocated to the model triage pod, leaving 6 GB for production databases, message queues, and host telemetry.
• Latency SLO: p99 Time-to-First-Token (TTFT) under 50 ms to classify and route high-severity incident logs.
• Connectivity: Air-gapped or network-restricted environments where cloud API calls to 70B+ external models are strictly forbidden for compliance and cost reasons.

Scenario Continuity Bridge: After adapting an 8B specialist in Module 3 (Problem Diagnosis and Triaging, choose-compact-student-model.mdx:28-44), an edge deployment requirement arises where Kubernetes worker nodes have a strict 2.0 GB container RAM limit and no GPU. This Module 4 analysis explores how compact student distillation (1B–3B models) can bridge the gap between the 8B specialist capability and the hard operational constraints.

Deploying a 70B parameter model locally is impossible (requiring over 35 GB in INT4 and dedicated accelerators). An 8B parameter model requires 4.5 GB of weights in INT4 plus KV cache and engine memory, blowing through the 2 GB container budget and causing fatal Out-Of-Memory (OOM) pod evictions.

To solve this, Staff engineers size and select a compact student model (1B to 3B parameters).

[The Operational Sizing Funnel]
┌─────────────────────────────────────────────────────────────┐
│ 1. Hardware & Envelope Limits                               │
│    Memory Budget: <= 2.0 GB RAM total (Weights + KV + Engine)│
│    Latency: p99 TTFT < 50 ms on multi-threaded CPU          │
└──────────────────────────────┬──────────────────────────────┘
                               ▼
┌─────────────────────────────────────────────────────────────┐
│ 2. Distillation Architecture Compatibility                  │
│    Teacher: Llama 3.1 70B (128k vocabulary, BPE)            │
│    Tokenizer Match: Identical vocab enables direct KL loss  │
└──────────────────────────────┬──────────────────────────────┘
                               ▼
┌─────────────────────────────────────────────────────────────┐
│ 3. Selected Specialist Candidate                            │
│    Target: Llama 3.2 1B (INT4: 0.57 GB, KV: 32 KiB/token)    │
│    Total Pod Footprint: 1.14 GB (< 2 GB, 860 MB headroom)    │
└─────────────────────────────────────────────────────────────┘

Choosing the wrong student architecture before distillation begins creates failure modes that no amount of hyperparameter tuning can fix:

1. The Tokenizer Mismatch Trap: Selecting a student with a different vocabulary requires training intermediate projection layers or discarding token-level logit supervision, degrading distillation quality.
2. The KV-Cache Explosion: Selecting a student model with Multi-Head Attention (MHA) or high key-value head counts exhausts memory during multi-turn contexts, even when INT4 static weights fit.
3. The License Lockout: Investing weeks of distillation into a candidate whose community license prohibits internal enterprise deployment or specific revenue tiers.

---

Section 2: Candidate Architectures (1B–3B Parameter Landscape)

Modern small language models (SLMs) in the 1B–3B parameter class leverage advanced pretraining data curation (synthetic filtering, educational scoring) and modern architectural components (Grouped-Query Attention, RoPE scaling, SwiGLU activations).

The primary open-weights candidates for compact student deployment fall into three families:

Architecture	Parameters	Context Window	Attention Type (Q / KV Heads)	Vocab Size	License
Llama 3.2 1B	1.23B	128,000	GQA (32 / 8 heads)	128,256	Llama 3.2 Community
Llama 3.2 3B	3.21B	128,000	GQA (24 / 8 heads)	128,256	Llama 3.2 Community
Qwen 2.5 1.5B	1.54B	32,768	GQA (12 / 2 heads)	151,936	Apache 2.0
Qwen 2.5 3B	3.09B	32,768	GQA (16 / 2 heads)	151,936	Apache 2.0
SmolLM2 1.7B	1.71B	8,192	GQA (32 / 32 heads, MHA)	49,152	Apache 2.0

1. Llama 3.2 1B and 3B (Meta, 2024)

• Architectural Dimensions: Llama 3.2 1B features 16 transformer layers, hidden dimension 2,048, 32 query heads and 8 key-value heads with head dimension 64. Llama 3.2 3B features 24 query heads and 8 key-value heads with head dimension 128, and 28 layers.
• Tokenizer: Tiktoken-based BPE tokenizer with 128,256 tokens. Crucially, this vocabulary and tokenization scheme is identical to Llama 3.1 8B, 70B, and 405B.
• Context: Native 128k context support via RoPE base frequency theta = 500,000.
• License: Llama 3.2 Community License. Free for commercial and research use, subject to Meta’s acceptable use policy and a monthly active user ceiling of 700 million.

2. Qwen 2.5 1.5B and 3B (Alibaba, 2024)

• Architectural Dimensions: Qwen 2.5 1.5B features 28 transformer layers, hidden dimension 1,536, 12 query attention heads and only 2 key-value heads with head dimension 128. Qwen 2.5 3B features 36 layers, 16 query heads and 2 key-value heads with head dimension 128.
• Extreme KV Efficiency: By restricting the key-value heads to 2, Qwen 2.5 achieves low KV-cache compression per token.
• Context: Native 32k context support via RoPE base frequency theta = 1,000,000.
• License: Fully permissive Apache 2.0 license, free of commercial user-count covenants.

3. SmolLM2 1.7B (Hugging Face, 2025)

• Curated Pretraining: Trained on 11 trillion tokens curated specifically for reasoning density (FineWeb-Edu, synthetic math, code, and Python notebooks).
• Architectural Dimensions: 24 layers, hidden dimension 2,048, 32 query heads and 32 key-value heads (Multi-Head Attention, head_dim 64).
• Context: Conservative 8,192 context window.
• License: Apache 2.0.

---

Section 3: The Memory Equation: Weights, KV Cache, and Runtime Overhead

Before selecting a model, an engineer must accurately calculate the total serving memory footprint:

Memory_total = Memory_weights + Memory_KV_cache + Memory_runtime

1. Static Weight Memory

For a model with N parameters:

• FP16 (16-bit): Bytes = N * 2 (e.g., 2.46 GB for a 1.23B parameter model).
• INT4/Q4_K_M (4-bit GGUF): Bytes = N * 0.55 including 4.5-bit k-quant scales and GGUF metadata overhead (e.g., 0.68 GB for Llama 3.2 1B; 0.85 GB for Qwen 2.5 1.5B; 0.94 GB for SmolLM2 1.7B depending on packing efficiency).

2. KV-Cache Memory Per Token

In a transformer with Grouped-Query Attention (GQA), the KV cache stores keys and values for each layer and each KV head:

Bytes per token = 2 * n_layers * n_kv_heads * d_head * bytes_per_element

Where:

• The factor of 2 accounts for storing both Key (K) and Value (V) tensors.
• n_layers is the number of transformer blocks.
• n_kv_heads is the number of key-value heads.
• d_head is the dimension of each head.
• bytes_per_element is 2 bytes for FP16 or BF16 KV cache.

Let us compute the KV-cache footprint for the candidate models:

1. Llama 3.2 1B:

   Bytes/tok = 2 * 16 layers * 8 kv_heads * 64 head_dim * 2 bytes = 32,768 bytes = 32 KiB/token

   For a 2,048-token context:

   32 KiB * 2,048 = 65,536 KiB = 64 MiB

2. Qwen 2.5 1.5B:

   Bytes/tok = 2 * 28 layers * 2 kv_heads * 128 head_dim * 2 bytes = 28,672 bytes = 28 KiB/token

   For a 2,048-token context:

   28 KiB * 2,048 = 57,344 KiB = 56 MiB

3. Llama 3.2 3B:

   Bytes/tok = 2 * 28 layers * 8 kv_heads * 128 head_dim * 2 bytes = 114,688 bytes = 112 KiB/token

   For a 2,048-token context:

   112 KiB * 2,048 = 229,376 KiB = 224 MiB

4. SmolLM2 1.7B (MHA):

   Bytes/tok = 2 * 24 layers * 32 kv_heads * 64 head_dim * 2 bytes = 393,216 bytes = 384 KiB/token

   For a 2,048-token context:

   384 KiB * 2,048 = 786,432 KiB = 768 MiB

3. Total Serving Footprint Under INT4 Precision

Assuming a lightweight edge inference runtime (llama.cpp or ONNX Runtime) requiring 500 MB base overhead:

Candidate Serving Footprint Comparison (INT4 Weights + 2048 Context KV Cache):
┌───────────────────┬──────────────┬──────────────┬──────────────┬──────────────────┐
│ Model             │ Weights INT4 │ KV (2k tok)  │ Runtime Base │ Total Pod Memory │
├───────────────────┼──────────────┼──────────────┼──────────────┼──────────────────┤
│ Llama 3.2 1B      │ 0.68 GB      │ 0.064 GB     │ 0.50 GB      │ 1.25 GB [FIT]    │
│ Qwen 2.5 1.5B     │ 0.85 GB      │ 0.056 GB     │ 0.50 GB      │ 1.41 GB [FIT]    │
│ SmolLM2 1.7B      │ 0.94 GB      │ 0.768 GB     │ 0.50 GB      │ 2.21 GB [MARGINAL]│
│ Qwen 2.5 3B       │ 1.70 GB      │ 0.072 GB     │ 0.50 GB      │ 2.27 GB [EXCEEDS]│
│ Llama 3.2 3B      │ 1.77 GB      │ 0.229 GB     │ 0.50 GB      │ 2.50 GB [EXCEEDS]│
└───────────────────┴──────────────┴──────────────┴──────────────┴──────────────────┘

Notice that under a strict 2.0 GB container limit, 3B models either exceed the ceiling or leave less than 10 MB of safety margin, making them prone to OOM crashes during request spikes. The 1B–1.5B models fit comfortably, leaving 0.59 GB (Qwen 2.5 1.5B), 0.17 GB (Llama 3.2 1B), and 0.49 GB (Qwen 2.5 1.5B with fallback) of headroom for OS buffers and multi-session concurrency. SmolLM2 1.7B exceeds the 2.0 GB limit by ~230 MB and is not recommended for this edge deployment.

Context Length Multipliers in Production

While a 2,048-token context requires only 64 MB of KV cache on Llama 3.2 1B, scaling to its full 128,000-token window requires 32 KiB * 128,000 = 4.096 GB of KV cache alone! When sizing student models for edge nodes, always cap max_model_len in your serving engine config to the maximum required by your domain task (e.g., 4,096 tokens for incident triage).

---

Section 4: Tokenizer Alignment and Distillation Mechanics

A major trap when selecting a student model is ignoring vocabulary alignment with the teacher model.

When distilling knowledge from a teacher (T) to a student (S), two primary paradigms exist:

1. Sequence-Level Distillation (Black-Box): The teacher generates synthetic prompt-completion pairs, and the student is trained via standard Supervised Fine-Tuning cross-entropy loss.
2. Token-Level Logit Distillation (White-Box): The student minimizes the Kullback-Leibler (KL) divergence between its output logit distribution and the teacher’s probability distribution over the entire vocabulary:

L_distill = D_KL( P_teacher(y_t | x, y_<t) || P_student(y_t | x, y_<t) )

Case A: Aligned Tokenizers (Llama 3.1 70B Teacher -> Llama 3.2 1B Student)
Teacher Vocab (128,256) ───► [Logit Vector (128,256)]
                                     │  Direct KL Divergence
Student Vocab (128,256) ───► [Logit Vector (128,256)]
Outcome: Zero information loss. Every token index matches identically.

Case B: Mismatched Tokenizers (Llama 3.1 70B Teacher -> Qwen 2.5 1.5B Student)
Teacher Vocab (128,256) ───► [Logit Vector (128,256)]
                                     ▼
                      Requires Learned Linear Projection Matrix
                             W_proj in R^(128,256 x 151,643)
                                     ▼
Student Vocab (151,643) ───► [Logit Vector (151,643)]
Outcome: Adds 19.4M parameters just for projection; introduces noise and alignment artifacts.

The Cost of Tokenizer Mismatch

If you distill a Llama 3.1 70B teacher into a Qwen 2.5 1.5B student using logit-level KL divergence:

1. Different Token Boundaries: The teacher may tokenize the word Unrecoverable as ['Un', 'recover', 'able'] (3 tokens), while the student tokenizer splits it as ['Unrecover', 'able'] (2 tokens). Sequence positions no longer map 1-to-1, preventing standard step-wise KL loss.
2. Vocabulary Dimension Mismatch: The teacher outputs vectors of size 128,256; the student expects 151,643. Comparing logits requires training an auxiliary cross-vocabulary projection matrix W_proj, which introduces optimization instability and memory overhead during training. A dense projection from 128,256 × 151,643 has 19,486,703,616 parameters (~19.45B, ~38.97 GB in BF16), which is larger than the 1.5B student itself. This is why cross-tokenizer logit distillation is usually replaced by sequence-level KD or token alignment.
3. Sequence Distillation Fallback: To avoid projection layers, teams with mismatched tokenizers are forced to fall back to sequence-level distillation (training only on the hard argmax tokens produced by the teacher). This discards the rich “dark knowledge” contained in the teacher’s soft probability distribution (e.g., semantic relationships between second- and third-ranked alternative actions).

Therefore, when distilling from a Llama 3.1 teacher, Llama 3.2 1B provides a decisive engineering advantage: direct, zero-loss token-level distillation without auxiliary projection layers.

---

Section 5: Worked Example: SRE Edge Node Sizing Decision

Let us evaluate our domain scenario: deploying an automated Incident Triage and Routing specialist to 5,000 edge nodes.

Constraint Specifications

• Memory Ceiling: <= 2.0 GB pod RAM limit.
• Latency SLO: p99 TTFT < 50 ms, decode speed > 30 tokens/sec on 4 CPU threads.
• Task Input/Output: Input context ~1,500 tokens (log stack traces and metadata); generation length ~60 tokens (JSON triage payload with severity, root cause category, and target team).
• Teacher Model: Centrally hosted Llama 3.1 70B.

Verification Code: Sizing and Selecting the Candidate

Run the automated verification suite to calculate exact memory footprints and test budget compliance across candidates:

# Save as scripts/verify-student-selection.py and run via python3
def evaluate_student_models():
    # Model parameters, layer counts, KV head counts, and head dimensions
    models = {
        "Llama-3.2-1B": {
            "params": 1.23e9, "layers": 16, "q_heads": 32, "kv_heads": 8, "head_dim": 64,
            "vocab": 128256, "license": "Llama 3.2 Community", "teacher_match": True
        },
        "Llama-3.2-3B": {
            "params": 3.21e9, "layers": 28, "q_heads": 24, "kv_heads": 8, "head_dim": 128,
            "vocab": 128256, "license": "Llama 3.2 Community", "teacher_match": True
        },
        "Qwen-2.5-1.5B": {
            "params": 1.54e9, "layers": 28, "q_heads": 12, "kv_heads": 2, "head_dim": 128,
            "vocab": 151936, "license": "Apache-2.0", "teacher_match": False
        },
        "Qwen-2.5-3B": {
            "params": 3.09e9, "layers": 36, "q_heads": 16, "kv_heads": 2, "head_dim": 128,
            "vocab": 151936, "license": "Apache-2.0", "teacher_match": False
        },
        "SmolLM2-1.7B": {
            "params": 1.71e9, "layers": 24, "q_heads": 32, "kv_heads": 32, "head_dim": 64,
            "vocab": 49152, "license": "Apache-2.0", "teacher_match": False
        }
    }

    context_len = 2048
    runtime_overhead_gb = 0.50 # 500 MB base engine buffer
    budget_limit_gb = 2.0

    print("=== Candidate Sizing Audit ===")
    for name, m in models.items():
        # INT4 weights (GGUF Q4_K_M: 0.55 bytes/param includes quantization metadata)
        int4_weights_gb = round((m["params"] * 0.55) / 1e9, 2)

        # KV cache: 2 * layers * kv_heads * head_dim * 2 bytes per token
        bytes_per_tok = 2 * m["layers"] * m["kv_heads"] * m["head_dim"] * 2
        kv_cache_gb = round((bytes_per_tok * context_len) / 1e9, 3)

        total_serving_gb = round(int4_weights_gb + kv_cache_gb + runtime_overhead_gb, 2)
        fits_budget = total_serving_gb <= budget_limit_gb

        print(f"[*] {name}: Total={total_serving_gb} GB | Fits: {fits_budget} | Tokenizer Match: {m['teacher_match']}")

evaluate_student_models()

Execution Results and Architecture Trade-Off Matrix

=== Candidate Sizing Audit ===
[*] Llama-3.2-1B: Total=1.25 GB | Fits: True  | Tokenizer Match: True
[*] Llama-3.2-3B: Total=2.50 GB | Fits: False | Tokenizer Match: True
[*] Qwen-2.5-1.5B: Total=2.27 GB | Fits: True  | Tokenizer Match: False
[*] Qwen-2.5-3B: Total=2.68 GB | Fits: False | Tokenizer Match: False
[*] SmolLM2-1.7B: Total=2.21 GB | Fits: False | Tokenizer Match: False

The Architecture Selection Decision

1. Primary Recommendation: Llama 3.2 1B.
   • Total serving memory under INT4 quantization is 1.25 GB, leaving 0.75 GB of safe headroom under the 2.0 GB ceiling.
   • Native tokenizer alignment with Llama 3.1 70B teacher enables direct KL-divergence logit distillation, capturing soft probability boundaries without projection layer complexity.
   • Analytical TTFT estimate on 4 CPU threads with llama.cpp: 0.68 GB / 35 GB/s ≈ 19.4 ms/token ≈ 51.5 tokens/sec (upper bound, reduced to ~30-40 tokens/sec after CPU cache/thread overhead).
2. Fallback Alternative: Qwen 2.5 1.5B.
   • If enterprise legal policy prohibits the Llama 3.2 Community License due to user-count thresholds, Qwen 2.5 1.5B is the approved Apache 2.0 fallback.
   • Serving footprint is 1.41 GB (viable, but requires tokenizer mismatch handling).
   • Required engineering adaptation: Distillation must utilize sequence-level synthetic data or an auxiliary 19.45B parameter vocabulary projection head.

---

Section 6: Advanced Engineering: CPU Memory Bandwidth and TTFT

On edge CPUs lacking dedicated HBM (High Bandwidth Memory), inference latency is strictly memory-bandwidth bound during generation (decode phase) and compute-bound during prompt processing (prefill phase).

1. The Decode Latency Formula

During decode, the CPU must stream the entire active weight matrix from system DDR memory into cache once for every generated token:

Generation Time Per Token = Model Weight Memory (Bytes) / DDR Memory Bandwidth (Bytes/sec)

Consider an edge server equipped with dual-channel DDR4 memory providing an effective bandwidth of 35 GB/sec:

• For Llama 3.2 3B (INT4 = 1.77 GB):

  Decode Time = 1.77 GB / 35 GB/sec = 0.0505 sec = 50.5 ms/token  (19.8 tokens/sec)

• For Llama 3.2 1B (INT4 = 0.68 GB):

  Decode Time = 0.68 GB / 35 GB/sec = 0.0194 sec = 19.4 ms/token  (51.5 tokens/sec, upper bound)

• For SmolLM2 1.7B (INT4 = 0.94 GB):

  Decode Time = 0.94 GB / 35 GB/sec = 0.0686 sec = 68.6 ms/token  (14.7 tokens/sec)

By selecting the 1B student over the 3B student, decode throughput increases by ~2.6x on identical CPU hardware, ensuring that the 60-token triage response finishes in under 1.5 seconds total end-to-end latency (adjusted for corrected 19.4 ms/token vs original 16.3 ms/token claims).

---

Key takeaways

• Size for Operational Envelopes First: Sizing a student model is dictated by RAM ceilings, CPU memory bandwidth, and container budgets rather than arbitrary benchmark rankings.
• The 2 GB Boundary: At INT4 precision, 1B–1.5B models require 1.25 GB to 1.41 GB total serving memory with 2k context, leaving safe headroom. 3B models require >2.0 GB, causing OOM risks on 2 GB edge pods.
• KV Cache Scaling with GQA: Grouped-Query Attention drastically reduces KV cache growth. Llama 3.2 1B consumes 32 KiB/tok; Qwen 2.5 1.5B with 2 KV heads consumes only 28 KiB/tok.
• Tokenizer Alignment Dictates Distillation Strategy: Identical tokenizers (Llama 3.1 70B teacher to Llama 3.2 1B student) enable direct token-level logit KL distillation. Mismatched tokenizers force fallback to sequence distillation or projection layers.
• CPU Decode is Bandwidth-Bound: Decode speed on CPU scales linearly with smaller weight footprints; moving from 3B to 1B yields a 2.6x throughput increase on DDR4 memory.

Try it yourself

1. 01You are deploying a bedside clinical assistant on an embedded ARM64 tablet with 4 GB total RAM and a 1.5 GB pod limit. The application requires an 8,192-token context window. Calculate the KV cache memory for Llama 3.2 1B (32 KiB/token) and determine whether total memory (with 0.68 GB INT4 weights and 400 MB runtime) fits the pod limit.

   Show a hint

   KV Cache = 32 KiB * 8192 = 262,144 KiB = 256 MiB = 0.268 GB. Total memory = 0.68 + 0.268 + 0.400 = 1.348 GB. Since 1.348 GB <= 1.5 GB, the configuration fits with ~152 MB headroom.

2. 02A development team wants to distill domain capability from an internal Mistral Large 2 (123B) teacher into Qwen 2.5 1.5B or SmolLM2 1.7B. Why does neither student have tokenizer alignment with Mistral, and what parameter-free workaround allows distillation?

   Show a hint

   Mistral Large 2 uses a 32,768-token BPE tokenizer, while Qwen uses 151,643 tokens and SmolLM2 uses 49,152 tokens. Sequence-level distillation (generating synthetic teacher completions and training the student via autoregressive cross-entropy) avoids auxiliary projection layers.

3. 03Calculate the theoretical decode throughput (tokens/sec) for Qwen 2.5 1.5B (0.85 GB INT4 weights) on an edge processor with 28 GB/s effective memory bandwidth.

   Show a hint

   Decode Time Per Token = 0.85 GB / 28 GB/s = 0.0304 sec = 30.4 ms/token. Throughput = 1 / 0.0304 = 32.9 tokens/sec.

Interview questions

How do you choose between an 8B, 3B, and 1B model when designing an on-device AI system?

Follow a physical constraint hierarchy: (1) Resource Budget: calculate weights, KV cache, and runtime overhead. Under a 2 GB container budget, 8B and 3B models cause immediate OOM crashes; (2) Latency SLO: CPU memory bandwidth bounds generation speed (T = Weights / Bandwidth). Achieving >50 tok/s on DDR4 requires <= 0.7 GB weights (1B class); (3) Task Complexity: for structured routing or extraction, a 1B model distilled from a 70B teacher regularly matches an un-tuned 8B base model.

Why does Grouped-Query Attention (GQA) have such a dramatic effect on edge inference memory?

In Multi-Head Attention, every query head requires a dedicated key and value head. GQA shares key-value heads across multiple query heads (e.g., 32 Q heads to 8 KV heads in Llama 3.2, or 12 Q heads to 2 KV heads in Qwen 2.5). This cuts KV-cache memory consumption by 4x to 6x per token, allowing memory-constrained edge devices to maintain multi-turn contexts without running out of RAM.

Why does tokenizer alignment matter when distilling a teacher model into a compact student?

Direct token-level logit distillation minimizes Kullback-Leibler divergence over the vocabulary distribution at each token step. If teacher and student tokenizers differ in vocabulary size or BPE merging boundaries, sequence positions no longer align 1-to-1, and logit vectors cannot be directly compared. The team must either train an auxiliary vocabulary projection matrix (adding parameters and complexity) or fall back to sequence-level distillation (which loses soft logit information).

Related lessons

• Fine-Tuning & Domain-Specific Models · Decide Whether to TrainDiagnose the Gap Before Changing Weights12 min
• Fine-Tuning & Domain-Specific Models · Adapt Within a Compute BudgetFull Fine-Tuning Versus Adapters14 min
• Fine-Tuning & Domain-Specific Models · Adapt Within a Compute BudgetDebug and Reproduce a Training Run14 min
• Fine-Tuning & Domain-Specific Models · Adapt Within a Compute BudgetChoose Continued Pretraining or Supervised Fine-Tuning14 min