Understanding Parameter-Efficient Fine-Tuning
When adapting foundation models to domain-specific university benchmarks, full-parameter fine-tuning requires updating billions of weights. Low-Rank Adaptation (LoRA) freezes the pre-trained model weights $W_0 in mathbb{R}^{d times k}$ and injects trainable rank decomposition matrices:
W = W_0 + Delta W = W_0 + frac{alpha}{r} (B times A)
Where $B in mathbb{R}^{d times r}$ and $A in mathbb{R}^{r times k}$ with intrinsic rank $r ll min(d, k)$. A is initialized with Gaussian random values, and B is initialized to zero, ensuring $Delta W = 0$ at the onset of training.