What is LoRA? Fine-tuning that leaves the model as it is and trains only small add-on tables of numbers

Fine-tuning that leaves the model as it is and trains only small add-on tables of numbers.

LoRA is a way to fine-tune a big model. The original weights stay frozen; two small tables are attached beside each table being changed, and only those are trained. Microsoft researchers published it in 2021.

Instead of rewriting a big table, LoRA writes the change as the product of two long, thin tables. For a 4,096-by-4,096 table, two 4,096-by-8 tables hold about 65,000 numbers instead of 16.8 million. The change can later be merged in, so answers come no slower.

In May 2023, QLoRA attached LoRA to a model squeezed to 4 bits and fine-tuned a 65-billion-parameter model on one 48 GB GPU. But a 2024 study teaching extra coding and maths found LoRA learned less than whole-model fine-tuning.

Think of it like this

It's like adding sticky notes to a textbook instead of writing over the pages. Peel the notes off and the book is back to how it was, and you can swap in a different stack of notes for each subject.

For example

With image AIs, people sometimes keep the base model file as it is and download a LoRA file of a few hundred MB, holding only the newly taught part, to load on top.

Common mix-ups

Mix-upA LoRA file on its own is enough to run an AI.
ActuallyA LoRA file holds only the small add-on tables, not the base model. When it's saved with Hugging Face's PEFT library, a settings file alongside names the base model it was made for, and that model has to be loaded first. On a model whose layers are a different size, the tables won't even fit.
Mix-upTuning with LoRA makes the model forget what it was good at.
ActuallyIn the same 2024 study, LoRA learned less of the new job but kept its skills outside that area better than full fine-tuning did. The paper's title says it all: LoRA learns less and forgets less. Which to pick depends on what matters more to you.

Try it in class

For teachers at the start of a lesson — or for students working on their own.

Question: If you only add things on and never touch the original model, what do you gain and what do you give up?
Activity: Work out how many cells a 4,096-by-4,096 table has and how many two 4,096-by-8 tables have, and find how many times smaller the second is. Then read the LoRA (E0126) and QLoRA (E0229) records, write one line each on what they cut down, and use the sticky-note comparison to sort the good points from the drawbacks.