LoRA: Add Trainable Low-Rank Adapters

Section 1: The Practical Outcome First

Full fine-tuning of modern foundation models requires immense GPU memory. When updating every parameter of an 8B model in 16-bit precision, you must allocate memory not just for the model weights (16 GB), but also for backward-pass gradients (16 GB), optimizer momentum states (32 GB), and optimizer variance states (32 GB). In total, static memory alone exceeds 96 GB before counting dynamic activation buffers, requiring multi-GPU tensor parallelism or deep parameter sharding (FSDP/ZeRO-3).

Low-Rank Adaptation (LoRA) (Hu et al., 2021) bypasses this memory wall by making a profound mathematical observation: the weight updates delta-W accumulated during task-specific adaptation have a very low intrinsic rank.

Instead of updating the full frozen base weight matrix W_0, LoRA freezes W_0 and injects two small, trainable low-rank decomposition matrices A and B in a parallel forward path:

[Input Vector x: Dimension 4096]
       │
       ├───► [Frozen Base Matrix W_0: 4096 x 4096] ──────────┐
       │     (No gradients, FP16/BF16, 16M params)           │
       │                                                     ▼
       └───► [Matrix A: 16 x 4096] ──► [Matrix B: 4096 x 16] ──► (+) ──► Output h
             (Trainable, Gaussian)     (Trainable, Zeros)     Scaled by (alpha / r)

Look at the empirical impact when fine-tuning Llama 3.1 8B for domain incident triage:

Adaptation Strategy	Trainable Parameters	% of Base Checkpoint	FP16 Disk Storage	Fits Single 24GB GPU?
Full Fine-Tuning (FFT)	8,030,261,248	100.0%	16.0 GB	No (>119 GB static VRAM)
LoRA (Attention-only, r=16)	13,631,488	0.170%	26.0 MB	Yes (~16.8 GB peak VRAM)
LoRA (All-linear, r=16)	41,943,040	0.522%	80.0 MB	Yes (~17.5 GB peak VRAM)

With LoRA, your trainable parameter footprint shrinks by over 99.4%, your adapter artifact compresses to an 80 MB file that fits on an email attachment, and training runs comfortably on a single consumer or cloud workstation GPU (such as an RTX 4090 or A10G).

---

Section 2: Mathematical Formulation & Stabilization Scaling

For a given pre-trained weight matrix W_0 of dimension d_out x d_in, full fine-tuning directly optimizes the modified weight W = W_0 + ΔW.

LoRA parameterizes the update matrix ΔW as the matrix product of two low-rank matrices B and A:

ΔW = (alpha / r) * (B @ A)

Where:

• A has dimension r x d_in (down-projection).
• B has dimension d_out x r (up-projection).
• r is the rank hyperparameter, where r << min(d_in, d_out). Typically, r is set between 8 and 64.
• alpha is a constant scaling hyperparameter.

The modified forward pass for an input token representation x is:

h = W_0 @ x + (alpha / r) * (B @ (A @ x))

The LoRA forward pass architecture with parallel low-rank decomposition branches

Read this diagram as text

The input token vector x is simultaneously processed by the frozen pre-trained base weight matrix W_0 and the parallel LoRA adapter branch. Matrix A projects x down to rank r, followed by matrix B projecting back up to d_out. The adapter output is scaled by alpha / r and added to the frozen base output.

Step 0 Identity Initialization

To ensure training begins smoothly without disrupting pretrained capabilities, LoRA enforces an asymmetric initialization:

• Matrix A: Initialized with a random zero-mean Gaussian distribution A ~ N(0, σ^2).
• Matrix B: Initialized to exact zeros (B = 0).

At training step 0, before gradient descent begins:

ΔW = (alpha / r) * (0 @ A) = 0

Because ΔW = 0, the forward pass output h = W_0 @ x + 0 = W_0 @ x is mathematically identical to the untouched foundation model. The model begins training from its known baseline state, eliminating the initial instability common in random adapter configurations.

The Scaling Constant alpha / r

The hyperparameter alpha acts as a constant learning rate multiplier for the adapter update. When you adjust the rank r (for example, sweeping r = 8, 16, 32, 64 to find optimal model capacity), dividing by r normalizes the magnitude of the gradient updates:

• If you increase r, each individual rank contributes a smaller fraction of the update, preventing the adapter branch from overpowering the base weights.
• Setting alpha = 2 * r (or keeping a fixed alpha = 16) ensures you do not need to radically retune your learning rate when experimenting with different rank sizes.

---

Section 3: Parameter Counting & Target Modules: Attention vs All-Linear

In the original LoRA paper (Hu et al., 2021), the authors applied adapters exclusively to attention projection layers (W_q and W_v). However, modern architectural research has fundamentally refined this practice.

Modern GQA Attention Projection Math

Modern foundation models like Llama 3.1 8B and Qwen 2.5 7B use Grouped Query Attention (GQA). In Llama 3.1 8B, the hidden dimension is d_in = 4096, but while queries have 32 heads (d_q = 4096), keys and values have only 8 heads (d_k = d_v = 1024).

For rank r = 16, the trainable parameter count per layer is:

• W_q: r * (d_in + d_q) = 16 * (4096 + 4096) = 131,072
• W_k: r * (d_in + d_k) = 16 * (4096 + 1024) = 81,920
• W_v: r * (d_in + d_v) = 16 * (4096 + 1024) = 81,920
• W_o: r * (d_q + d_o) = 16 * (4096 + 4096) = 131,072
• Total Attention LoRA per layer: 425,984
• Across 32 layers: 425,984 * 32 = 13,631,488 parameters (0.170% of base, 26.0 MB FP16 storage).

MLP Feedforward Projection Math

In SwiGLU transformer architectures, the feedforward network consists of three dense projections: W_gate, W_up, and W_down, where the intermediate dimension is d_mlp = 14336.

For rank r = 16:

• W_gate: r * (d_in + d_mlp) = 16 * (4096 + 14336) = 294,912
• W_up: r * (d_in + d_mlp) = 16 * (4096 + 14336) = 294,912
• W_down: r * (d_mlp + d_in) = 16 * (14336 + 4096) = 294,912
• Total MLP LoRA per layer: 884,736
• Across 32 layers: 884,736 * 32 = 28,311,552 parameters.

Why You Must Target All Linear Layers

Summing attention and MLP projections yields the All-Linear LoRA configuration:

• 13,631,488 + 28,311,552 = 41,943,040 parameters (0.522% of base, 80.0 MB FP16 storage).

In production

The All-Linear Imperative (Dettmers et al., 2023): Dettmers et al. demonstrated in the QLoRA study that applying LoRA strictly to attention projections fails to match full fine-tuning performance on complex reasoning and code tasks. The MLP feedforward layers store the vast majority of factual associations and heuristic representations in transformers. Targeting all linear layers (q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj) requires only 0.52% of base parameters while matching full fine-tuning quality across standard benchmarks.

Capability Retention and Catastrophic Forgetting

In addition to parameter efficiency, low-rank adaptation offers a critical safety property for domain models. In LoRA Learns Less and Forgets Less (Biderman et al., 2024, arXiv:2405.09673), the authors demonstrated that because LoRA restricts gradient updates to a low-dimensional subspace, it fundamentally mitigates catastrophic forgetting.

While full fine-tuning aggressively modifies representations across all dimensions and risks degrading general world knowledge and programming ability, LoRA preserves the pretrained representation manifold. For domain models (such as incident triage routers) that must preserve general conversational ability alongside specialized runbook parsing, all-linear LoRA delivers the optimal compromise between domain specialization and general safety.

---

Section 4: Multi-Tenant Serving & Adapter Merging

LoRA offers two massive operational advantages in production inference systems:

1. Zero-Latency Weight Merging

When deploying a single specialized model to production, you can fold the low-rank adapter directly into the base weights before exporting the checkpoint:

W_serving = W_0 + (alpha / r) * (B @ A)

Because matrix addition is computed offline during model serialization, the deployed model has the exact same architecture, parameter count, and tensor shapes as the base model. There is zero inference latency overhead and zero architectural modification required in high-throughput inference engines like vLLM or TensorRT-LLM.

2. High-Density Multi-Tenant Routing

In multi-tenant SaaS platforms (for example, hosting custom triage models for 50 enterprise customers), deploying 50 distinct full-weight checkpoints would require 50 instances of an 8B model (800 GB of VRAM), incurring tens of thousands of dollars in monthly cloud GPU infrastructure.

With LoRA, the serving architecture changes dramatically:

1. Load one shared 8B base model into GPU memory (16 GB VRAM).
2. Store 50 separate LoRA adapter weights (80 MB each = 4 GB total) in CPU host RAM or fast NVMe storage.
3. At inference time, dynamic serving runtimes (such as S-LoRA or vLLM multi-LoRA) load the specific adapter requested in the HTTP header into GPU cache and compute the low-rank delta in parallel with base matrix multiplication.

This allows a single 24GB GPU to serve dozens of specialized customer models concurrently.

---

Section 5: Executable Verification & Arithmetic Parity

To verify the exact parameter counts and memory calculations in your local environment, run the standalone verification script [scripts/verify-lora-math.py]:

scripts/verify-lora-math.py

from dataclasses import dataclass

@dataclass
class ModelConfig:
    name: str
    d_model: int
    num_layers: int
    num_q_heads: int
    num_kv_heads: int
    d_head: int
    d_mlp: int
    total_base_params: int

LLAMA_3_1_8B = ModelConfig(
    name="Llama-3.1-8B",
    d_model=4096,
    num_layers=32,
    num_q_heads=32,
    num_kv_heads=8,
    d_head=128,
    d_mlp=14336,
    total_base_params=8030261248,
)

Execution verifies the exact parameter counts and memory storage requirements:

=== Running Topic 09 LoRA Parameter Verification ===

[*] Llama-3.1-8B (Rank 16):
    Attention-only LoRA params: 13,631,488 (0.170%) | Storage: 26.00 MB
    All-linear LoRA params:     41,943,040 (0.522%) | Storage: 80.00 MB

[*] Qwen-2.5-7B (Rank 16):
    Attention-only LoRA params: 10,092,544 (0.133%) | Storage: 19.25 MB
    All-linear LoRA params:     40,370,176 (0.530%) | Storage: 77.00 MB

[+] LoRA Parameter Calculations Verified Cleanly.

---

Key takeaways

• LoRA decomposes weight updates into low-rank matrices B and A (ΔW = (α/r) * B @ A), reducing trainable parameters by >99% while freezing base weights.
• Step-0 initialization sets A ~ N(0, σ^2) and B = 0, guaranteeing that ΔW = 0 and preserving baseline capabilities at the start of training.
• The scaling factor alpha/r stabilizes gradient updates across varying rank choices, allowing rank sweeps without major learning rate retuning.
• Modern GQA architectures (Llama 3.1 8B) yield 13.6M trainable params (0.17%) for attention-only LoRA (r=16) and 41.9M params (0.52%) for all-linear LoRA.
• Dettmers et al. (2023) proved that targeting all linear layers (attention + MLP) is critical to matching full fine-tuning performance on complex tasks.
• LoRA adapters can be merged directly into base weights offline (W = W_0 + ΔW) for zero latency overhead during production serving.

Try it yourself

1. 01Calculate the trainable parameter count for a rank-32 LoRA adapter targeting only the query and value projections (W_q and W_v) of an 8B model with d_model = 4096, 32 layers, and standard MHA (d_q = d_v = 4096).

   Show a hint

   Per layer: W_q has 32 * (4096 + 4096) = 262,144; W_v has 32 * (4096 + 4096) = 262,144. Total per layer = 524,288. Across 32 layers = 16,777,216 params (16.78M).

2. 02Why would initializing both Matrix A and Matrix B to exact zeros be problematic for gradient descent backpropagation?

   Show a hint

   If both A and B are zero, their gradients with respect to loss would also be zero at step 0 for symmetry reasons, preventing the adapter weights from receiving initial learning signals.

3. 03Explain why an enterprise serving 20 distinct task adapters saves substantial GPU costs by keeping base weights frozen in shared memory rather than hosting 20 full model instances.

   Show a hint

   One base model requires ~16GB VRAM, while 20 adapters require only 20 * 80MB = 1.6GB VRAM. Hosting 20 full models requires 20 * 16GB = 320GB VRAM, demanding multiple high-cost GPU nodes.

Interview questions

How does Low-Rank Adaptation (LoRA) reduce the memory required for LLM fine-tuning?

LoRA freezes the base model weights and only trains small decomposition matrices A and B. Because optimizer states (such as AdamW momentum and variance) and gradients scale with the number of trainable parameters, reducing trainable parameters from 8B to ~42M reduces optimizer memory from ~96GB to under 500MB, allowing the training process to fit on a single 24GB GPU.

Why is matrix B initialized to zeros while matrix A is initialized with random Gaussian values?

Initializing B to zero ensures that delta-W = B @ A is identically zero at step 0, meaning the model output exactly matches the pretrained checkpoint without degradation. Initializing A with random Gaussian values breaks symmetry so that backpropagation can compute non-zero gradients for both matrices once training commences.

What is the function of the alpha/r scaling factor in the LoRA forward pass equation?

The scaling factor alpha/r normalizes the magnitude of the adapter output across different rank hyperparameters r. When tuning rank r, this normalization ensures the learning rate schedule does not need to be heavily readjusted, stabilizing gradient magnitudes during hyperparameter sweeps.

Why do modern fine-tuning practitioners recommend targeting all linear layers rather than attention projections only?

As established by Dettmers et al. (2023) in the QLoRA study, transformer MLP feedforward layers store the majority of knowledge and relational reasoning. Adapting only attention projections leaves MLP weights frozen, causing the model to underfit complex reasoning tasks. All-linear adaptation increases trainable parameters slightly (from 0.17% to 0.52% of base), but fully recovers the performance of full fine-tuning.

Related lessons

• Fine-Tuning & Domain-Specific Models · Decide Whether to TrainDiagnose the Gap Before Changing Weights12 min
• Fine-Tuning & Domain-Specific Models · Decide Whether to TrainSelect a Base Model for the Task13 min
• Fine-Tuning & Domain-Specific Models · Decide Whether to TrainSet a Baseline and Success Threshold12 min
• Fine-Tuning & Domain-Specific Models · Make Training Data TrustworthyFormat Conversations and Mask the Loss13 min
• Fine-Tuning & Domain-Specific Models · Make Training Data TrustworthyUse Synthetic and Teacher Data Carefully13 min
• Fine-Tuning & Domain-Specific Models · Adapt Within a Compute BudgetChoose Continued Pretraining or Supervised Fine-Tuning14 min