Parameter-Efficient Fine-Tuning: LoRA Mathematics | AI & Machine Learning Engineering

Large Language Models contain tens to hundreds of billions of parameters. Updating all parameters during fine-tuning (Full Fine-Tuning) requires caching optimizer states (momentum and variance vectors in Adam), necessitating 16 to 24 bytes of VRAM per parameter and rendering adaptation computationally prohibitive.

Low-Rank Adaptation (LoRA) (Hu et al., 2021) leverages the Intrinsic Rank Hypothesis, freezing pre-trained weights and injecting low-rank decomposed matrices into attention projections.

Trainable ParametersPEFT

> 99% Reduction

LoRA shrinks trainable parameters from d * k down to r * (d + k) with zero degradation in adaptation quality.

1. The Intrinsic Rank Hypothesis#

Aghajanyan et al. (2020) demonstrated that pre-trained language models reside on low-dimensional manifolds: the effective change in weight space needed to adapt a foundation model to a downstream domain has a very small intrinsic rank $r \ll \min(d, k)$ .

Architecture Diagram

Rendering diagram...

2. Mathematical Formulation of LoRA#

Consider a pre-trained weight matrix $W_0 \in \mathbb{R}^{d \times k}$ (typically the query $W_Q$ or value $W_V$ projection matrices in self-attention).

During full fine-tuning, the parameter update is modeled as $W = W_0 + \Delta W$ , where $\Delta W \in \mathbb{R}^{d \times k}$ has full rank.

In LoRA, $W_0$ is completely frozen and does not receive gradient updates. The parameter update $\Delta W$ is decomposed into the matrix product of two low-rank matrices:

\Delta W = B A, \quad \text{where } B \in \mathbb{R}^{d \times r}, \quad A \in \mathbb{R}^{r \times k}

With rank $r \ll \min(d, k)$ (typically $r \in \{4, 8, 16\}$ ).

The modified forward pass computes:

h = W_0 x + \Delta W x = W_0 x + \frac{\alpha}{r} B A x

Where $\alpha$ is a constant hyperparameter. The scaling factor $\frac{\alpha}{r}$ ensures that when experimenting with different values of rank $r$ , the magnitude of the adapter update remains stable without requiring retuning of the learning rate.

Step 1: Exact Zero Initial State

If both $A$ and $B$ were initialized with random Gaussian noise, $\Delta W = B A \ne 0$ , immediately disrupting the high-quality pre-trained representations before training begins.

LoRA enforces an asymmetrical initialization:

Matrix $A$ is initialized from a Gaussian normal distribution: $A \sim \mathcal{N}(0, \sigma^2)$
Matrix $B$ is initialized strictly to zero: $B = 0$

Evaluating $\Delta W$ at step $t = 0$ :

\Delta W^{(0)} = B^{(0)} A^{(0)} = 0 \cdot A^{(0)} = 0

Thus, training begins with the adapted layer performing identically to the pre-trained baseline.

Step 2: Parameter Savings Derivation

For a standard hidden dimension $d = k = 4096$ and rank $r = 16$ :

Full Fine-Tuning: $4096 \times 4096 = 16,777,216$ parameters per matrix.
LoRA: $r(d + k) = 16 \times (4096 + 4096) = 131,072$ parameters.

\text{Parameter Reduction} = 1 - \frac{131,072}{16,777,216} = 1 - 0.0078125 \approx 99.22\%

3. Zero-Latency Inference Deployment#

In production systems, executing two separate branches ( $W_0 x$ and $B A x$ ) introduces kernel launch overhead and GPU memory bandwidth contention.

Because matrix multiplication is linear and associative, we can explicitly compute the low-rank delta matrix and fold it into the base weights once training completes:

W_{\text{deployed}} = W_0 + \frac{\alpha}{r} B A

During online inference, the engine performs a single standard matrix multiplication $h = W_{\text{deployed}} x$ , achieving zero added latency overhead.

4. From-Scratch Implementation: Zero Dependencies#

The following implementations construct a complete LoRA Linear Layer from scratch in pure Python and pure Rust, showing parameter factorization, the forward pass, and final inference weight merging.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
import math
import random
from typing import List

class LoRALinearLayerFromScratch:
    def __init__(self, in_features: int, out_features: int, rank: int = 4, alpha: float = 8.0):
        self.in_features = in_features
        self.out_features = out_features
        self.rank = rank
        self.scaling = alpha / rank

        # 1. Base weights W_0 (frozen pre-trained weights)
        random.seed(42)
        scale = 1.0 / math.sqrt(in_features)
        self.W0: List[List[float]] = [
            [random.uniform(-scale, scale) for _ in range(in_features)]
            for _ in range(out_features)
        ]

        # 2. LoRA Matrix A: Gaussian initialization N(0, 1/r)
        a_std = 1.0 / math.sqrt(rank)
        self.A: List[List[float]] = [
            [random.gauss(0.0, a_std) for _ in range(in_features)]
            for _ in range(rank)
        ]

        # 3. LoRA Matrix B: strictly initialized to zeros
        self.B: List[List[float]] = [
            [0.0 for _ in range(rank)]
            for _ in range(out_features)
        ]

    @staticmethod
    def _mat_vec_mul(W: List[List[float]], x: List[float]) -> List[float]:
        return [sum(W[i][j] * x[j] for j in range(len(x))) for i in range(len(W))]

    def forward(self, x: List[float]) -> List[float]:
        # Path 1: Base pre-trained forward pass: W_0 * x
        base_out = self._mat_vec_mul(self.W0, x)

        # Path 2: LoRA adapter forward pass: (alpha / r) * B * (A * x)
        # First compute intermediate: a_out = A * x  (in R^r)
        a_out = self._mat_vec_mul(self.A, x)
        # Then compute: b_out = B * a_out (in R^out_features)
        lora_out = self._mat_vec_mul(self.B, a_out)

        # Combine: h = W_0 x + (alpha / r) * B * A * x
        return [base_out[i] + self.scaling * lora_out[i] for i in range(self.out_features)]

    def merge_weights_for_inference(self) -> List[List[float]]:
        # Compute W_merged = W_0 + (alpha / r) * (B * A)
        merged_W = [[0.0] * self.in_features for _ in range(self.out_features)]
        for i in range(self.out_features):
            for j in range(self.in_features):
                # Inner product across rank dimension r
                delta_w_ij = sum(self.B[i][k] * self.A[k][j] for k in range(self.rank))
                merged_W[i][j] = self.W0[i][j] + self.scaling * delta_w_ij
        return merged_W

if __name__ == "__main__":
    layer = LoRALinearLayerFromScratch(in_features=4, out_features=3, rank=2, alpha=4.0)
    x_test = [1.0, 0.5, -0.5, 2.0]

    # At step 0, B=0, so output equals base weights exactly
    out_init = layer.forward(x_test)
    base_only = layer._mat_vec_mul(layer.W0, x_test)
    print("Step 0 verification (diff should be 0.0):")
    print("  Output:", [round(v, 4) for v in out_init])
    print("  Base:  ", [round(v, 4) for v in base_only])

    # Simulate adapted weights in B
    layer.B[0][0] = 0.5
    layer.B[1][1] = -0.5
    out_adapted = layer.forward(x_test)
    print("\nAdapted Output:", [round(v, 4) for v in out_adapted])

    # Merge weights for production deployment
    W_deploy = layer.merge_weights_for_inference()
    deploy_out = layer._mat_vec_mul(W_deploy, x_test)
    print("Deployed Merged Matrix Output:", [round(v, 4) for v in deploy_out])

5. Active Knowledge Checks#

Concept check

Why is matrix B in LoRA (Delta W = B * A) initialized strictly to zero at the start of fine-tuning?

Concept check

How does LoRA achieve zero added latency during online inference deployment compared to full fine-tuning?