LoRA — Low-Rank Adaptation

The Fine-Tuning Problem

You have a pre-trained LLM with 7 billion parameters. It speaks English fluently, knows trivia, and can write code. But you need it to specialize — to answer medical questions, generate legal contracts, or write in a specific brand voice. This is fine-tuning: adjusting the model's weights on a smaller, task-specific dataset.

The problem is scale. Full fine-tuning updates every parameter in the model. For a 7B model, that means storing gradients, momentum, and optimizer states for all 7 billion parameters. You need multiple high-end GPUs just to hold the model in memory, let alone train it. For a 70B model, the cost becomes prohibitive for most organizations. This is the fine-tuning bottleneck — and it is exactly the problem LoRA solves.

What Is LoRA?

LoRA (Low-Rank Adaptation) was proposed by Hu et al. in 2021. The key insight is simple but powerful: when fine-tuning a model, the weight updates do not need to be full-rank. Instead of learning a complete update matrix ΔW of size d×k, LoRA decomposes it into two small matrices: B (size d×r) and A (size r×k), where r is much smaller than both d and k.

The original weights W stay completely frozen — they are never updated. Only the small A and B matrices are trained. During inference, you simply add the product BA to the original weights: W' = W + BA. This means LoRA adds zero inference latency if you merge the weights after training. You get the benefits of fine-tuning with a fraction of the cost.

The Math of LoRA

Let us make this precise. Suppose a pre-trained weight matrix W has dimensions d×k (for example, 4096×4096 in a typical Transformer layer). Full fine-tuning learns an update ΔW of the same size: 4096×4096 = 16.7 million parameters per weight matrix.

LoRA decomposes ΔW into BA, where B is d×r and A is r×k. With rank r=8, B is 4096×8 (32,768 params) and A is 8×4096 (32,768 params). Total: 65,536 parameters — a 256× reduction. The matrix A is initialized with small random values; B is initialized to zero, so BA starts as zero and the model behaves exactly like the original at the start of training. A scaling factor α/r controls the magnitude of the update.

LoRA Architecture: The Bypass

The diagram below shows how LoRA wraps around an existing weight matrix. The input x flows through both the frozen pre-trained weight W and the trainable LoRA path (B·A). The outputs are summed: Wx + BAx = (W + BA)x. Think of it as a bypass lane alongside a highway — the main road (frozen W) stays the same, while a small side lane (LoRA) carries the learned adaptation.

Frozen weight W (grey) with trainable LoRA bypass B·A (cyan). The outputs merge at the ⊕ node.

Why Low Rank Works

You might wonder: how can a rank-8 matrix possibly capture enough information to fine-tune a model? The answer lies in the intrinsic dimension hypothesis. Research has shown that the learned weights of over-parameterized models actually reside in a low-dimensional subspace. The model has far more parameters than it needs for any single task — the extra capacity helps during pre-training but is unnecessary for fine-tuning.

Empirically, LoRA at rank 4 to 16 matches or exceeds the quality of full fine-tuning on most benchmarks. The original LoRA paper tested on GPT-3 (175B parameters) and found that rank 4 or 8 was sufficient for most tasks. Higher rank helps for tasks that require learning entirely new knowledge rather than just recombining existing capabilities, but the gains diminish quickly.

A large weight update matrix (left) decomposes into two small low-rank matrices (right) — same expressive power, far fewer parameters

Visualizing Rank: Full Fine-Tuning vs LoRA

The 3D visualization below shows two weight matrices side by side. On the left, full fine-tuning lights up every cell — all parameters are updated. On the right, LoRA only trains a thin slice of columns (the low-rank decomposition). Rotate the scene to appreciate the contrast in parameter count.

Left: full fine-tuning updates ALL cells (coral). Right: LoRA only trains a thin low-rank slice (cyan).

LoRA Hyperparameters

LoRA has a small but important set of hyperparameters that control its behavior. Understanding these is essential for getting good results.

Key LoRA Parameters

Rank (r): the dimension of the low-rank decomposition. Typical values: 4, 8, 16, 32, 64. Higher rank = more capacity but more parameters. Alpha (α): a scaling factor that controls the overall magnitude of the LoRA update. The effective scaling is α/r. Common practice: set α = 2r. Target Modules: which weight matrices to apply LoRA to. The most impactful targets are the attention Q and V projections, though applying LoRA to all linear layers (Q, K, V, O, FFN up, FFN down) often yields better results. Dropout: optional dropout applied to the LoRA path, typically 0.05-0.1, to prevent overfitting on small datasets.

A common starting configuration for a 7B model is: r=16, α=32, target all linear layers, no dropout. This trains roughly 20-40 million parameters — less than 1% of the total model.

Parameter Count: Full Fine-Tuning vs LoRA

The chart below puts the parameter savings in concrete terms. For a single weight matrix of size 4096×4096 (common in 7B-class models), full fine-tuning updates all 16.7 million parameters. LoRA with rank 4 trains only 32,768 — a 512× reduction. Even at rank 16, LoRA trains only 0.78% of the parameters.

Bar lengths are proportional — the LoRA bars are so small they are nearly invisible compared to full fine-tuning

LoRA in Practice

Since its introduction, LoRA has become the standard approach for fine-tuning large models. Several important extensions and practical patterns have emerged.

QLoRA: LoRA on a Quantized Model

QLoRA (Dettmers et al., 2023) combines LoRA with 4-bit quantization of the base model. The frozen weights are stored in 4-bit precision (instead of 16-bit), reducing memory usage by 4×. The LoRA adapters are still trained in 16-bit precision for numerical stability. This makes it possible to fine-tune a 65B model on a single 48GB GPU — a task that previously required 8+ GPUs. QLoRA is the most popular way to fine-tune open-source models today.

LoRA Merging: Zero-Cost Inference

Because LoRA weights are additive (W' = W + BA), you can merge them into the base model after training by simply adding BA to W. The merged model has the exact same architecture and size as the original — no extra inference cost. You can also merge multiple LoRA adapters (trained on different tasks) into one model by weighted addition, or swap between adapters at runtime without reloading the base model.

Multi-LoRA Serving

In production, you often need multiple fine-tuned variants of the same base model — one for summarization, one for translation, one for code. With LoRA, you load the base model once and swap only the tiny adapter weights (a few MB) for each request. This is called multi-LoRA serving and is used by platforms like vLLM and LoRAX to serve hundreds of fine-tuned models from a single GPU.

Other PEFT Methods

LoRA is the most popular Parameter-Efficient Fine-Tuning (PEFT) method, but it is not the only one. The comparison table below shows how it stacks up against alternatives. Each method makes a different trade-off between parameter count, inference cost, and quality.

LoRA uniquely combines zero inference overhead (after merging) with near-full-fine-tuning quality

Key Takeaways

Explore Related Topics

Continue your journey through LLM fundamentals:

• Training — how models learn from data
• RLHF — aligning models with human preferences