Full Fine-Tuning Versus Adapters

Section 1: The Practical Outcome First

When engineering specialized domain models, every engineering leader eventually confronts an architectural debate: should you update 100% of the model parameters via Full Fine-Tuning (FFT), or freeze the base checkpoint and train lightweight adapters (LoRA / QLoRA)?

Consider an enterprise incident triage platform serving 10 distinct engineering microservices (such as authentication, database clusters, payments, and networking). Each service demands specialized routing logic, unique runbook IDs, and domain-specific JSON completion formats.

Look at the operational infrastructure comparison when serving 10 specialized variants of an 8B model:

Operational Metric	Full Fine-Tuning (10 Models)	LoRA / Multi-Tenant (10 Adapters)	Architectural Advantage
Trainable Parameters per Model	8.03 Billion (100%)	41.9 Million (0.522%)	-99.4% Trainable Footprint
Static Training Memory (8B)	119.66 GB (Requires multi-GPU)	15.58 GB (Single 24GB GPU)	Fits Single Workstation
Disk Storage for 10 Models	149.6 GB (10 × 14.96 GB)	15.74 GB (1 Base + 10 Adapters)	89.5% Disk Savings
Serving VRAM for 10 Models	149.6 GB (Requires multiple 80GB GPUs)	15.74 GB (Fits single 24GB GPU)	Single-GPU Multi-Tenant
Adapter Swap Latency	~30 seconds (full model reload)	<100 ms (dynamic memory page)	Zero-Downtime Hot-Swapping

Full Fine-Tuning locks your infrastructure into multi-node GPU clusters, massive checkpoint storage, and rigid deployment topologies. Adapters decouple base language capabilities from task specialization, enabling high-density multi-tenant serving on a fraction of the hardware.

---

Section 2: The Scientific Trade-Off: Capability Ceiling vs Forgetting

The choice between FFT and LoRA is not merely an engineering convenience; it represents a fundamental statistical tradeoff between adaptation ceiling and capability retention.

The statistical and systems comparison between Full Fine-Tuning and Low-Rank Adapters

Read this diagram as text

Full Fine-Tuning updates every weight across the entire parameter space, achieving the highest ceiling on radical distribution shifts but risking catastrophic forgetting and requiring ZeRO/FSDP sharding. LoRA constrains updates to a low-rank subspace, acting as a regularizer that preserves base model general reasoning and fits single-GPU training.

The Empirical Finding of Biderman et al. (2024)

In LoRA Learns Less and Forgets Less (Biderman et al., 2024, arXiv:2405.09673), the researchers conducted the largest comparative study to date across model families and downstream benchmarks. Their empirical findings define modern best practices:

1. LoRA Acts as an Implicit Regularizer: By constraining parameter updates to a low-rank subspace (such as rank r = 16 or r = 64), LoRA prevents the model from overwriting the complex multi-head attention and representation manifolds learned during pretraining. It retains out-of-domain reasoning, calibration, and factual memory far better than full fine-tuning.
2. The Ceiling on Radical Distribution Shifts: When a target task requires learning fundamentally new token semantics (such as compiling raw binary bytecode, translating an unrepresented low-resource language, or adapting a general model to complex chemical SMILES notation), LoRA’s rank constraint can bottleneck adaptation. In these extreme shifts, full fine-tuning achieves a measurably higher in-domain accuracy ceiling.
3. The Practical Rule: For domain instruction-following, task formatting, conversational style, API tool use, and operational classification, all-linear LoRA matches full fine-tuning while preventing catastrophic model collapse.

---

Section 3: Training Memory Breakdown & The Distributed Sharding Wall

Why does Full Fine-Tuning demand so much memory?

When fine-tuning an 8B parameter model using the standard AdamW optimizer in 16-bit precision (BF16), the static memory requirements are:

M_static_FFT = M_weights + M_gradients + M_optimizer

Where:

• M_weights: 8.03 * 10^9 * 2 bytes = 16.06 GB (14.96 GiB) in BF16.
• M_gradients: 8.03 * 10^9 * 2 bytes = 16.06 GB (14.96 GiB) in BF16.
• M_optimizer: AdamW maintains three FP32 states per parameter (FP32 master weight copy = 4 bytes, 1st momentum vector = 4 bytes, 2nd variance vector = 4 bytes). In total, AdamW requires 12 bytes per parameter:

  8.03 * 10^9 * 12 bytes = 96.36 GB (89.75 GiB)

• Total Static FFT Memory: 14.96 + 14.96 + 89.75 = 119.66 GB.

Because static memory alone (119.66 GB) dwarfs any single GPU (whether a 24 GB RTX 4090 or an 80 GB H100), full fine-tuning is impossible without distributed memory sharding:

Distributed Sharding Strategies (Rajbhandari et al., 2020)

Sharding Architecture	Sharded Components	Memory Savings	Minimum Infrastructure (8B Model)
ZeRO-1	Optimizer States partitioned across GPUs	~4× reduction in optimizer	4× 40GB GPUs or 2× 80GB GPUs
ZeRO-2	Optimizer States + Gradients partitioned	~8× reduction	2× 40GB GPUs or 1× 80GB GPU
ZeRO-3 / PyTorch FSDP	Optimizer + Gradients + Model Parameters	Linear with GPU count (N_GPUs)	Multi-node or 4× 24GB GPUs

With PyTorch Fully Sharded Data Parallel (FSDP) or DeepSpeed ZeRO-3, model parameters are gathered across GPUs on-the-fly right before each layer’s forward pass, and released immediately afterward. While this enables full fine-tuning across commodity GPUs, it introduces heavy inter-GPU communication overhead over PCIe or NVLink.

In contrast, all-linear LoRA (r=16) updates only 41.9M parameters:

• Frozen Base: 14.96 GB
• Adapter Weights: 80.0 MB
• AdamW Optimizer States: 480.0 MB
• Total Static Memory: 15.58 GB (Comfortably fits on a single 24GB GPU).

---

Section 4: Multi-Tenant Serving Runtimes & Dynamic Routing

The operational superiority of adapters becomes decisive in production serving architectures.

The Dedicated Instance Anti-Pattern (FFT)

If you train 12 specialized models via Full Fine-Tuning:

• You must serialize 12 distinct 16 GB checkpoint files (192 GB storage).
• In production, serving 12 models concurrently requires hosting 12 independent model replicas. At 16 GB per replica, weights consume 192 GB of VRAM, requiring multiple dedicated 80GB GPU nodes running 24/7—costing thousands of dollars per month regardless of query traffic.
• If traffic to an obscure microservice spikes, you must spin up an entire new 16 GB model instance with a 30-to-60-second cold-start latency.

High-Throughput Adapter Routing (S-LoRA & vLLM)

Using systems like S-LoRA (Sheng et al., 2023, arXiv:2311.03285) or vLLM multi-LoRA:

1. Shared Base Memory: The engine loads a single 8B base model into 16 GB of GPU VRAM.
2. Unified Paging: Dynamic adapters (80 MB each) reside in CPU system RAM or fast PCIe-attached SSDs.
3. Batched Low-Rank GEMM: The serving engine batches requests for different adapters together into a single GPU execution pass. As tokens arrive, custom CUDA kernels apply the requested adapter’s down-projection A and up-projection B dynamically without reloading base weights.
4. Instant Hot-Swapping: Introducing a new service adapter requires copying an 80 MB file into memory cache in milliseconds.

In production

The Release Candidate Strategy: Default to all-linear LoRA / QLoRA for domain adaptation unless empirical evaluation on held-out tasks demonstrates a clear failure to capture domain mechanics that only full parameter updates can resolve.

---

Section 5: Executable Verification & Arithmetic Parity

To verify the training memory breakdown and multi-tenant serving calculations in your environment, run the standalone verification script [scripts/verify-fft-vs-lora.py]:

scripts/verify-fft-vs-lora.py

# Calculates exact static training memory for Llama 3.1 8B (8.03B params)
fft_weights = 14.96  # GB in BF16
fft_grads   = 14.96  # GB in BF16
fft_opt     = 89.75  # GB in FP32 AdamW (12 bytes/param)
static_fft  = fft_weights + fft_grads + fft_opt  # 119.66 GB

Execution produces the exact, verified numerical outputs:

=== Running Topic 11 FFT vs LoRA Verifier ===

[*] Training Static Memory Comparison for Llama-3.1-8B:
    Full Fine-Tuning (FFT):
      - Weights (BF16):   14.96 GB
      - Gradients (BF16): 14.96 GB
      - AdamW Optimizer:  89.75 GB
      - Total Static:     119.66 GB (Requires ZeRO/FSDP: True)
    Low-Rank Adaptation (LoRA, all-linear r=16):
      - Frozen Base:      14.96 GB
      - Adapter Weights:  80.0 MB
      - AdamW Optimizer:  480.0 MB
      - Total Static:     15.58 GB (Fits 24GB GPU: True)

[*] Multi-Tenant Serving (10 domain tasks):
    FFT Disk Storage:    149.6 GB
    LoRA Disk Storage:   15.74 GB (Savings: 89.5%)
    FFT Serving VRAM:    149.6 GB (Requires multiple 80GB GPUs)
    LoRA Serving VRAM:   15.74 GB (Fits single 24GB GPU)

[+] Topic 11 FFT vs LoRA Memory & Serving Math Verified Cleanly.

---

Key takeaways

• Full Fine-Tuning updates 100% of parameters, requiring 119.66 GB of static memory for an 8B model (14.96 GB weights + 14.96 GB grads + 89.75 GB AdamW), mandating distributed ZeRO/FSDP sharding.
• LoRA (all-linear, r=16) updates only 41.9M parameters, requiring just 15.58 GB of static memory and fitting comfortably on a single 24GB GPU.
• Biderman et al. (2024) established that LoRA acts as an implicit regularizer, preventing catastrophic forgetting and out-of-domain degradation compared to FFT.
• FFT is justified primarily on radical distribution shifts (novel alphabets, symbolic code-to-spec formalisms) where rank bottlenecks restrict representation learning.
• Serving 10 specialized FFT models requires 149.6 GB VRAM and disk; serving 10 LoRA adapters requires just 15.74 GB (89.5% savings) and enables sub-100ms adapter hot-swapping.

Try it yourself

1. 01Calculate the AdamW optimizer memory (in GB) required during full fine-tuning of a 70B parameter model in standard 16-bit precision.

   Show a hint

   AdamW requires 12 bytes per parameter (4 bytes master weight + 4 bytes momentum + 4 bytes variance). M_opt = (70 * 10^9 * 12) / (1024^3) ≈ 840 * 10^9 / 1,073,741,824 ≈ 782.3 GB.

2. 02Why does ZeRO-3 (PyTorch FSDP) incur higher network communication overhead than ZeRO-1 during multi-GPU training?

   Show a hint

   ZeRO-1 only shards optimizer states and communicates gradients during the backward pass. ZeRO-3 shards model parameters, requiring an all-gather communication step before every forward and backward layer.

3. 03Explain why an autonomous code generation model fine-tuned on an obscure internal programming language might justify Full Fine-Tuning over LoRA.

   Show a hint

   A completely novel programming language represents a radical distribution shift with unfamiliar token transitions and syntactic rules that may exceed the capacity of low-rank adapter projections.

Interview questions

What is the difference between ZeRO-1, ZeRO-2, and ZeRO-3 distributed training architectures?

ZeRO-1 partitions only the optimizer states across data-parallel GPUs (4x memory reduction). ZeRO-2 partitions optimizer states and gradients (8x reduction). ZeRO-3 (implemented in PyTorch FSDP) partitions optimizer states, gradients, and model parameters, gathering parameters on-the-fly during forward and backward execution to eliminate memory scaling limits at the cost of increased inter-GPU communication.

Why did Biderman et al. (2024) conclude that LoRA learns less but forgets less than Full Fine-Tuning?

Because LoRA restricts gradient updates to a low-rank subspace (such as rank 16 or 64), it acts as a regularizer that prevents the model from overwriting the high-dimensional representation manifold learned during pretraining. While full fine-tuning can achieve higher in-domain accuracy on radical shifts, it aggressively erases general world knowledge and reasoning capabilities.

How does S-LoRA achieve high-throughput multi-tenant serving for hundreds of concurrent LoRA adapters?

S-LoRA stores a single base model in GPU VRAM and manages thousands of dynamic adapter weights in CPU host RAM. It uses unified paging to allocate memory dynamically for adapter weights and KV caches, and employs custom batched low-rank matrix multiplication kernels to process requests for different adapters simultaneously within the same forward pass.

When should an engineering team strictly choose Full Fine-Tuning over LoRA?

Full fine-tuning is justified when the adaptation task involves a radical distribution shift that requires reconfiguring foundational token representations—such as training on a completely new language family, symbolic biological sequences, or when empirical evaluations demonstrate that increasing LoRA rank r fails to capture task complexity.

Related lessons

• Fine-Tuning & Domain-Specific Models · Decide Whether to TrainDiagnose the Gap Before Changing Weights12 min
• Fine-Tuning & Domain-Specific Models · Decide Whether to TrainSelect a Base Model for the Task13 min
• Fine-Tuning & Domain-Specific Models · Decide Whether to TrainSet a Baseline and Success Threshold12 min
• Fine-Tuning & Domain-Specific Models · Make Training Data TrustworthyFormat Conversations and Mask the Loss13 min
• Fine-Tuning & Domain-Specific Models · Adapt Within a Compute BudgetChoose Continued Pretraining or Supervised Fine-Tuning14 min
• Fine-Tuning & Domain-Specific Models · Adapt Within a Compute BudgetLoRA: Add Trainable Low-Rank Adapters14 min
• Fine-Tuning & Domain-Specific Models · Adapt Within a Compute BudgetQLoRA: Quantize Base Weights to 4-Bit14 min