Select a Base Model for the Task

The one-sentence version

Selecting a base model is an irreversible architectural commitment: you must choose the smallest, most permissively licensed model whose tokenizer fertility minimizes KV cache overhead on your domain data, whose parameter count fits your serving GPU VRAM budget, and whose pretraining alignment matches your task.

Over-provisioning parameter count or accepting restrictive licenses creates severe operational drag without proportional quality gains.

---

The opening decision: four candidate models, one incident triage specialist

In Lesson 1 (diagnose-gap-before-training), our four-gate decision engine revealed that our production incident triage service suffers from multi-hop reasoning and domain policy failures that zero-shot and few-shot prompting cannot bridge. Having determined that weight adaptation is necessary, we must now select the starting base model.

Our task requires:

1. Parsing nested JSON telemetry containing stack traces, Kubernetes metrics, and database health indicators.
2. Emitting strict JSON triage decisions (subsystem, severity, recommended_action, requires_human_escalation).
3. Serving predictions with a tight P99 latency budget (< 500 ms) on an existing pool of single 24 GB GPUs (NVIDIA RTX 4090 / A10G / L4).
4. Unrestricted commercial deployment across global corporate infrastructure without legal tracking overhead.

Four prominent modern open-weights candidates present themselves in the 7B–9B parameter class:

Model Candidate	Parameter Count	Context Window	Vocab Size	Architecture	License
Llama 3.1 8B	8.03B	128k	128,256	GQA (8 KV heads, 32 Q heads)	Llama 3.1 Community (700M MAU limit)
Qwen 2.5 7B	7.61B	32k native (128k via YaRN)	151,643	GQA (4 KV heads, 28 Q heads)	Apache 2.0 (Permissive)
Gemma 2 9B	9.24B	8k native	256,000	Sliding Window + Full (GQA)	Gemma Terms of Use (Attribution notice required)
Mistral 7B v0.3	7.25B	32k	32,768	GQA (8 KV heads, 32 Q heads)	Apache 2.0 (Permissive)

Naively choosing the model with the highest broad academic benchmark score often leads to deployment failure. In production AI engineering, four critical dimensions dominate the selection process: tokenizer fertility, memory footprint, checkpoint alignment, and licensing.

---

1. Tokenizer architecture and fertility on structured data

Every token generated or attended to consumes GPU High Bandwidth Memory (HBM) in the Key-Value (KV) cache and incurs memory bandwidth transfer overhead during autoregressive decoding.

What is Tokenizer Fertility?

Tokenizer fertility measures the average number of subword tokens produced per unit of text (per word or per character):

Fertility_words = Total Tokens / Total Words
Fertility_chars = Total Tokens / Total Characters

Lower fertility means the tokenizer compresses the domain vocabulary more efficiently into fewer tokens. A model with 25% lower fertility on your domain data effectively provides:

• 25% shorter sequence lengths for the identical input text.
• 25% less KV cache memory allocated per request.
• 25% faster time-to-first-token (TTFT) during prompt prefill.
• Higher effective context capacity within a fixed physical context window limit.

Empirical Measurement on Structured JSON Telemetry

Consider our incident triage JSON payload from production systems:

{
  "incident_id": "INC-2026-004",
  "service": "distributed-kv-store",
  "subsystem": "consensus_engine",
  "anomaly_signature": "raft_leader_heartbeat_timeout_ms > 5000",
  "metrics": {
    "network_partition_detected": true,
    "uncommitted_log_entries": 1420,
    "disk_io_latency_p99_ms": 42.8
  },
  "recommended_action": "isolate_split_brain_partition"
}

When encoded across candidate tokenizers, the architectural differences in vocabulary design produce striking disparities:

Tokenizer Family	Vocab Size	Token Count	Fertility (Tokens/Word)	Compression (Chars/Token)	Delta vs Mistral (Baseline)
Mistral 7B v0.3 (SentencePiece)	32,768	157 tokens	6.83 tok/word	2.31 chars/tok	Baseline (+46.7% vs Llama)
Gemma 2 9B (SentencePiece)	256,000	143 tokens	6.22 tok/word	2.53 chars/tok	-8.9% tokens
Qwen 2.5 7B (BPE)	151,643	116 tokens	5.04 tok/word	3.12 chars/tok	-26.1% tokens
Llama 3.1 8B (Tiktoken BPE)	128,256	107 tokens	4.65 tok/word	3.38 chars/tok	-31.8% tokens (Compression Leader)

(Measurements verified using our reproducible benchmark script scripts/verify-base-model-selection.py via real Hugging Face tokenizers).

Why Gemma 2 has higher fertility despite a 256k vocabulary

A larger vocabulary does not automatically guarantee superior compression for code or JSON. Gemma 2’s 256,000-token vocabulary dedicates massive subword coverage to global natural language scripts, but its tokenizer rules split underscores (_), whitespace indentations, and camelCase syntax more aggressively than Qwen 2.5 or Llama 3.1. Qwen 2.5 and Llama 3.1 were specifically trained with dedicated tokens for whitespace sequences and common programming syntax, resulting in superior compression for structured workloads.

The Visual Tradeoff: Fertility Impact on Context and Cache

Tokenizer Fertility Impact on KV Cache and Usable Context

Read this diagram as text

Comparison of tokenizer fertility across 32k Mistral, 152k Qwen 2.5, and 128k Llama 3.1 vocabularies on structured JSON telemetry, illustrating sequence compression and KV cache reduction.

---

2. Memory footprint: training budgets versus serving limits

Choosing a base model requires verifying two distinct memory regimes:

1. Training VRAM: Can we adapt this model using LoRA or QLoRA on our development hardware?
2. Serving VRAM: Can we host the adapted specialist model within our production SLA and GPU memory limit?

2.1 The Training Memory Equation

When fine-tuning with Low-Rank Adaptation (LoRA), memory consumption decomposes into five primary components:

VRAM_train = M_base + M_lora_weights + M_gradients + M_optimizer + M_activations + M_cuda_overhead

Where:

• M_base: Frozen base model weights. For 16-bit (BF16), M_base = P × 2 bytes. For 4-bit NormalFloat (NF4 in QLoRA), M_base = P × 0.5 bytes.
• M_lora_weights: Trainable adapter matrices A and B across targeted layers. When targeting all linear attention and MLP projections (q, k, v, o, gate, up, down) with rank r = 16, trainable parameters are approximately ~0.5% of base parameters (P_trainable ≈ 0.005 × P, ~40M–42M parameters on an 8B model). When targeting only attention query and value projections (q, v), trainable parameters reduce to ~0.2% (P_trainable ≈ 0.002 × P, ~16M parameters). At 16-bit, M_lora_weights = P_trainable × 2 bytes.
• M_gradients: Gradients stored only for trainable adapter parameters: M_gradients = P_trainable × 2 bytes.
• M_optimizer: AdamW maintains two 32-bit floating point states (momentum and variance) for each trainable parameter: M_optimizer = P_trainable × 8 bytes (or P_trainable × 2 bytes if using 8-bit Paged AdamW). Total adapter overhead for weights, gradients, and FP32 AdamW is P_trainable × (2 + 2 + 8) = P_trainable × 12 bytes.
• M_activations: Intermediate activations retained for backpropagation. For an 8B model with Batch = 2, SeqLen = 2048, HiddenDim = 4096, NumLayers = 32, uncheckpointed activation memory scales as 2 × Batch × SeqLen × HiddenDim × NumLayers × 2 bytes ≈ 2.15 GB. With activation checkpointing (recomputing intermediate tensors during the backward pass), memory footprint is stabilized at approximately 1.5 GB – 2.0 GB.

2.2 Empirical Training Memory Fit Across GPU Tiers

Using verified component formulas from our calibration script (scripts/verify-base-model-selection.py):

Base Model	Param Count	16-bit LoRA (r=16) VRAM	Fits 24 GB GPU?	4-bit QLoRA (r=16) VRAM	Fits 24 GB GPU?
Qwen 2.5 7B	7.61B	18.90 GB	YES (tight)	7.49 GB	YES (abundant headroom)
Llama 3.1 8B	8.03B	19.75 GB	YES (tight)	7.71 GB	YES (abundant headroom)
Gemma 2 9B	9.24B	22.20 GB	YES (very tight)	8.34 GB	YES
Qwen 2.5 14B	14.7B	33.25 GB	NO (requires 40/80 GB)	11.20 GB	YES
Llama 3.1 70B	70.6B	146.39 GB	NO (requires 2× 80 GB)	40.49 GB	NO (requires 48/80 GB)

16-bit LoRA vs. 4-bit QLoRA on a 24 GB GPU

While 16-bit LoRA for an 8B model mathematically fits within 24 GB VRAM (~19.75 GB), it leaves only ~4 GB of headroom. If sequence length spikes or batch size increases, CUDA memory fragmentation triggers an immediate Out-Of-Memory (OOM) error. In contrast, 4-bit QLoRA consumes only ~7.7 GB, allowing larger batch sizes, longer context packing (up to 8,192 tokens), and training on inexpensive consumer GPUs (RTX 3090/4090) or cloud instances (A10G/L4 at $1.00–$1.85/hr).

2.3 The Serving Memory Equation

At inference time, optimizer states and gradient tensors do not exist. Serving memory is strictly governed by:

VRAM_serve = W_bytes + KV_cache_bytes + CUDA_context

Where:

• W_bytes = P × bytes_per_weight (FP16 = 2 bytes, AWQ/GPTQ INT4 = 0.5 bytes).
• KV_cache_bytes: Autoregressive key-value history:

  KV_cache_bytes = 2 × NumLayers × NumKVHeads × HeadDim × SeqLen × Batch × bytes_per_elem

For Grouped-Query Attention (GQA) models like Llama 3.1 8B (NumKVHeads = 8, HeadDim = 128, NumLayers = 32), 16-bit KV cache requires:

Bytes per token per sequence = 2 × 32 × 8 × 128 × 2 bytes = 131,072 bytes (128 KiB)

At a concurrency batch of 4 requests with an 8,192-token context window:

KV_cache_bytes = 4 × 8192 × 131,072 bytes ≈ 4.29 GB

Adding INT4 quantized model weights (8.03B × 0.5 bytes = 4.02 GB) and CUDA overhead (1.5 GB):

VRAM_serve = 4.02 GB + 4.29 GB + 1.50 GB = 9.81 GB

This serves comfortably on a single 16 GB or 24 GB GPU, delivering high throughput.

---

3. Base versus Instruct checkpoints: the starting point tradeoff

A frequent dilemma when starting a fine-tuning project is whether to adapt the raw Base (pretrained) checkpoint or the vendor’s aligned Instruct/Chat checkpoint.

Architectural Dimension	Starting from Base Checkpoint	Starting from Instruct Checkpoint
Pretraining Objective	Self-supervised next-token prediction across trillions of tokens.	Supervised Instruction Tuning (SFT) + RLHF / DPO preference optimization.
Instruction Following	❌ None. Emits continuous document text unless explicitly taught conversational turn-taking.	✅ Pre-aligned. Natively recognizes system, user, and assistant boundaries.
Catastrophic Forgetting	✅ None. Custom domain format is learned directly as the primary output structure.	⚠️ High risk. Fine-tuning on a narrow task can degrade chat formatting, reasoning, and multi-turn stability.
Safety Guardrail Clashes	✅ No baked-in refusal patterns. Unrestricted adaptation for internal security tasks.	⚠️ Frequent friction. Built-in RLHF safety filters may trigger refusals on error logs, malware signatures, or exploit traces.
Training Data Requirement	High (5,000–50,000+ examples to establish prompt-response structure and domain skills).	Low (250–2,000 high-quality examples sufficient for specialized downstream tasks).
Convergence Speed	Slower (requires more training steps to align representation to output format).	Fast (adapters quickly steer pre-aligned conversational representations).

The Engineering Decision Rule: Base vs. Instruct

IF Task == "Domain Knowledge Adaptation" (e.g. Continued Pretraining on raw manuals/codebases):
    --> START FROM BASE CHECKPOINT.
    (Prevents catastrophic forgetting of instruction templates; maximizes representation learning.)

ELSE IF Task == "Specialized Task Following" (e.g. Structured JSON triage, Classification, Tool Calling):
    IF Task Domain contains security logs, exploit traces, or triggers safety refusals:
        --> START FROM BASE CHECKPOINT (or uncensored base variant).
    ELSE:
        --> START FROM INSTRUCT CHECKPOINT.
        (Leverages pre-existing instruction alignment; converges in 1-3 epochs with minimal data.)

For our incident triage specialist, the domain consists of operational telemetry and runbook routing without harmful exploit generation. Starting from Instruct enables rapid convergence on JSON schema adherence within a few hundred training pairs.

---

4. Licensing constraints and enterprise legality

A technically superior model is useless if its license prohibits your commercial deployment model or creates compliance risk. Modern open weights fall under three primary legal regimes:

Open Weights Licensing Continuum:
[Permissive Open Source]  <------------------------->  [Proprietary Commercial Restriction]
    Apache 2.0 / MIT             Gemma Terms of Use         Llama 3.1 Community License
(Qwen 2.5, Mistral v0.3)             (Google)                         (Meta)

4.1 Permissive Open Source: Apache 2.0 (Qwen 2.5, Mistral v0.3)

• Permissions: Unrestricted commercial use, modification, private closed-source deployment, and model weight redistribution.
• Patent Grant: Express reciprocal patent license protects users against infringement claims by contributors.
• Restrictions: Minimal. Requires attribution and preserving copyright notices in source files.
• Enterprise Suitability: Gold Standard. Permitted in virtually all commercial enterprise environments without legal review friction.

4.2 Open-Access with Acceptable Use: Gemma Terms of Use (Gemma 2)

• Permissions: Commercial deployment permitted worldwide without royalty.
• Patent Grant: ❌ No express patent license grant included.
• Restrictions: Users must adhere to Google’s Prohibited Use Policy (restricting surveillance, automated legal/medical determinations, and defense applications). Redistribution requires including a copy of the Gemma Terms of Use notice text file and passing terms to downstream recipients.
• Model Derivatives & Distillation: Gemma terms explicitly govern model derivatives, including synthetic output distillation: model outputs cannot be used to generate synthetic training data to develop competitive models that circumvent safety filters or violate acceptable use.

4.3 Commercial License with User Threshold: Llama 3.1 Community License

• Permissions: Commercial use permitted for applications under 700 million monthly active users.
• The 700M MAU Trigger: The threshold is evaluated as of the Llama 3.1 version release date (July 23, 2024) across the licensee and licensee’s affiliates. If monthly active users in the preceding calendar month exceeded 700 million, the licensee must formally request an explicit commercial license from Meta.
• Patent Provisions: If you initiate patent infringement litigation against Meta, your license to Llama 3.1 immediately terminates.
• Attribution & Derivative Naming: Products using the model must prominently display “Built with Llama”. Furthermore, under Section 1.b.i, fine-tuning or distilling another model on Llama outputs is permitted provided that any downstream model derivative includes “Llama” at the beginning of its AI model name.

The Corporate Compliance Trap: The 700M MAU Clause

Even if your startup currently serves only 10,000 users, large enterprise legal teams frequently reject internal tooling built on Llama models if the parent corporation or conglomerate has global consumer products exceeding the 700M MAU threshold. For enterprise software sold to Fortune 50 companies, Apache 2.0 (Qwen 2.5) completely bypasses legal audits.

---

5. Architectural synthesis: the five-dimension decision tree

To systematically evaluate base models for any production deployment, trace this five-gate decision hierarchy:

Five-Dimension Base Model Selection Decision Architecture

Read this diagram as text

Five-gate decision tree evaluating licensing (Apache 2.0), hardware VRAM budgets (24GB), checkpoint alignment (Base vs Instruct), tokenizer fertility on structured data, and context window requirements.

---

The worked verdict: selecting the base model for incident triage

Applying our five-dimension framework to our incident triage domain:

1. Licensing: Enterprise deployment across multi-cloud corporate environments requires indemnification and zero compliance drag. Qwen 2.5 7B (Apache 2.0) passes unconditionally with express patent grants. Llama 3.1 8B is permissible for organizations below the 700M release-date MAU threshold.
2. Tokenizer Fertility: On our JSON telemetry fixture, Llama 3.1 8B (107 tokens) and Qwen 2.5 7B (116 tokens) compress the prompt 31.8% and 26.1% more efficiently than Mistral 7B (157 tokens), saving substantial KV cache memory and sequence length.
3. Hardware Capacity: Training on single 24 GB GPUs requires 4-bit QLoRA (~7.5 GB VRAM), fitting comfortably with batch size = 2 and sequence length = 4,096. Serving fits within 10 GB under INT4 weights and 16-bit KV cache.
4. Checkpoint Starting Point: Task requires strict JSON schema compliance for triage actions without safety filter interference. We choose the Instruct checkpoint as our starting point to exploit pre-trained dialogue formatting.
5. Context Window: Native 32k context on Qwen 2.5 7B comfortably covers the ~4k–8k token incident history buffers without requiring YaRN position interpolation.

The Selected Base Model: Qwen/Qwen2.5-7B-Instruct (Alternative: meta-llama/Llama-3.1-8B-Instruct if Llama licensing is pre-approved).

---

Key takeaways

• Tokenizer Fertility Matters: Tokenizer vocabulary design directly dictates KV cache memory consumption and effective context capacity on structured JSON/code data.
• VRAM Equation Split: Training memory includes optimizer states, gradients, and activations; serving memory consists strictly of weights, KV cache, and CUDA overhead.
• QLoRA Enables 24 GB Training: 4-bit NormalFloat QLoRA fits 7B-9B parameter models into ~7.5 GB VRAM, leaving ample room for context packing on commodity GPUs.
• Base vs. Instruct Alignment: Use Base checkpoints for continued pretraining and domain adaptation; use Instruct checkpoints for fast convergence on downstream instruction/triage tasks.
• License Before Code: Apache 2.0 provides unconditional commercial freedom; Llama 3.1 requires release-date MAU tracking and derivative naming; Gemma requires distribution notices and restricts distillation.

---

Try it yourself

1. 01Run scripts/verify-base-model-selection.py. Inspect the calculated fertility across tokenizers on the incident JSON sample, and calculate the KV cache memory difference in megabytes for a batch of 8 requests at 4,096 tokens sequence length comparing Mistral 7B against Qwen 2.5 7B.

   Show a hint

   Mistral 7B uses 128 KiB/token (32 layers × 8 KV heads × 128 × 2 bytes), while Qwen 2.5 7B uses 56 KiB/token (28 layers × 4 KV heads × 128 × 2 bytes: 57,344 bytes ≈ 56 KiB). Compute the memory footprint for both models using their respective KV cache constants.

2. 02Calculate the minimum VRAM required to train an unquantized 16-bit LoRA adapter (r=16, targeting all linear projections) on Qwen 2.5 14B with batch size 1 and sequence length 2048. Does it fit on a 24 GB GPU?

   Show a hint

   Base weights require ~29.4 GB in BF16 alone (14.7B params × 2 bytes), which already exceeds the 24 GB hardware limit before accounting for adapters or optimizer states. Training Qwen 2.5 14B on a single 24 GB GPU requires 4-bit QLoRA (~11.2 GB).

3. 03Review the Llama 3.1 Community License Section 2. Identify the exact wording regarding the 700 million Monthly Active User threshold and determine what action an enterprise must take if they cross it.

   Show a hint

   Look for Section 2(b) concerning commercial usage rights and Meta licensing requests.

---

Interview questions

Why can a model with a larger vocabulary (e.g. Gemma 2 with 256k tokens) exhibit worse tokenizer fertility on code and JSON than a model with a smaller vocabulary (e.g. Llama 3.1 with 128k tokens)?

Vocabulary size measures total tokens, but vocabulary allocation determines fertility. Gemma 2 allocates large token capacity to diverse global languages, but retains subword splitting rules that divide whitespace, punctuation, and identifiers (like snake_case or camelCase). Llama 3.1 and Qwen 2.5 explicitly train dedicated merge tokens for multi-space indentation and common code keywords, achieving higher compression per character on structured data despite having fewer total tokens.

When fine-tuning an 8B model with LoRA (rank 16) in 16-bit precision, why does optimizer memory scale with the trainable parameters rather than the base model parameters?

During LoRA adaptation, the 8B base model weights are strictly frozen: no gradients are computed for them, and no optimizer momentum or variance statistics are tracked. The AdamW optimizer is instantiated only for the low-rank adapter matrices A and B (typically ~16 million parameters or ~0.2% for attention q/v targeting, or ~40M–42M parameters / ~0.5% when targeting all linear projections). As a result, optimizer states consume only ~128 MB to ~336 MB (at 8 bytes per trainable parameter in FP32) rather than the ~64 GB that full fine-tuning of an 8B model would demand.

Under what technical and organizational conditions would you recommend fine-tuning a raw Base checkpoint rather than starting from an Instruct checkpoint?

You should start from a Base checkpoint when: (1) Performing continued pre-training on a large corpus of unstructured domain literature or proprietary code; (2) Adapting to a domain where vendor Instruct safety alignment causes false-positive refusals (e.g. analyzing vulnerability signatures, malware binaries, or raw medical incident logs); or (3) Inventing an entirely custom interaction format or non-dialogue sequence protocol where existing chat turn templates (like ChatML or Llama-3 headers) would cause interference.

How does Grouped-Query Attention (GQA) change the serving memory profile of a base model compared to Multi-Head Attention (MHA)?

In Multi-Head Attention, every query head has a dedicated key and value head (e.g. 32 Q heads, 32 KV heads). In GQA, multiple query heads share a single key-value head (e.g. 32 Q heads grouped across 8 KV heads in Llama 3.1 8B, a 4:1 ratio; or 28 Q heads across 4 KV heads in Qwen 2.5 7B, a 7:1 ratio). This reduces the physical size of the KV cache by 4x to 7x per token, dramatically lowering serving VRAM and enabling larger batch sizes and longer context windows on a single GPU.

---

Related lessons

• Fine-Tuning & Domain-Specific Models · Decide Whether to TrainDiagnose the Gap Before Changing Weights12 min
• LLM Internals & Architecture · Reasoning models and test-time computationEvaluating Reasoning: Baselines, Verifiers, and Parsing Failures12 min

---

Sources and scope

• Dubey et al. (2024), The Llama 3 Herd of Models, arXiv:2407.21783 — Architectural specifications, 128k Tiktoken tokenizer design, GQA parameters, and Llama Community License terms.
• Yang et al. (2024), Qwen2 Technical Report, arXiv:2407.10671 — 152k vocabulary BPE tokenizer, architectural scaling, and Apache 2.0 open-source release parameters.
• Google et al. (2024), Gemma 2: Improving Open Language Models at a Practical Size, arXiv:2408.00118 — 256k SentencePiece tokenizer, sliding window attention, and Gemma Terms of Use.
• Jiang et al. (2023), Mistral 7B, arXiv:2310.06825 — 32k SentencePiece tokenizer, sliding window attention, and Apache 2.0 release.
• Dettmers et al. (2023), QLoRA: Efficient Finetuning of Quantized LLMs, arXiv:2305.14314 — 4-bit NormalFloat (NF4) quantization, double quantization, paged optimizers, and VRAM memory accounting.
• Hu et al. (2021), LoRA: Low-Rank Adaptation of Large Language Models, arXiv:2106.09685 — Low-rank parameter decomposition, rank scaling, and trainable parameter ratio derivations.
• Su et al. (2024), RoFormer: Enhanced Transformer with Rotary Position Embedding, Neurocomputing — Rotary position embeddings and context window scaling interaction with tokenization density.

Illustrative Data Notice: The JSON payload and tokenization measurements represent reproducible calculations executed via Python tokenizers and the test script scripts/verify-base-model-selection.py. GPU hosting rates and cloud instance prices reflect standard AWS/GCP public pricing at time of writing. No synthetic benchmark claims or simulated GPU hardware metrics are invented.