In this guide
A general-purpose LLM knows a little about everything and a lot about nothing in particular. It does not know your college's lab manuals, your project's domain vocabulary, or the tone your chatbot needs. Full fine-tuning would fix that — but updating all 7 billion parameters needs serious GPU memory and risks wrecking what the model already knows.
LoRA (Low-Rank Adaptation) is the technique that made fine-tuning accessible: instead of retraining the model, you train a tiny set of adapter weights — often under 1% of the parameters — while the base model stays frozen. A 7B model can be fine-tuned on a single consumer GPU in hours. This guide explains how LoRA works, how to prepare your data, how to train, and how to avoid the mistakes that waste GPU time.
Why full fine-tuning is painful
Fine-tuning updates a model's weights on your data so it behaves the way you want. The naive approach — update every parameter — has three costs:
- Memory. Training a 7B model in full precision needs the weights (~28 GB in fp32, ~14 GB in fp16) plus gradients plus optimizer states. Adam stores two extra states per parameter, roughly tripling the footprint. A 7B full fine-tune can demand 60+ GB of VRAM.
- Catastrophic forgetting. Aggressively updating all weights can erase capabilities the base model had — your medical Q&A model suddenly forgets basic grammar.
- Storage and switching. Every fine-tuned variant is a full 14 GB copy. Want five task variants? Five copies.
Parameter-efficient fine-tuning (PEFT) methods address all three by training a small number of new parameters and leaving the base model frozen.
How LoRA works: the low-rank trick
A weight matrix W in the transformer has shape d×d (say 4096×4096 — about 16.7M parameters). LoRA hypothesizes that the change to W needed for your task lives in a low-dimensional subspace. Instead of learning a full ΔW, it learns two small matrices:
ΔW = B · A, where B is d×r and A is r×d, with rank r much smaller than d.
With r = 8 and d = 4096: B·A has 2 × 4096 × 8 = 65,536 parameters instead of 16.7M — roughly 0.4%. The forward pass becomes:
h = Wx + (B·A)x = Wx + B(Ax)
W stays frozen; only A and B are trained. A is typically initialized with small random values and B with zeros, so ΔW starts at zero — training begins from exactly the base model's behavior and deviates gradually. That zero-initialization is a quiet piece of genius: your model is never worse than the base model at step zero.
LoRA is usually applied to the attention projections (query and value matrices, sometimes all four) and optionally the MLP layers. At inference, B·A can be merged into W (W′ = W + BA), which means zero added latency — the deployed model is just a normal model with updated weights.
Note: LoRA adapters are tiny — tens of megabytes. You can keep one base model on disk and swap task-specific adapters in seconds. This is how teams serve dozens of fine-tuned variants cheaply.
QLoRA: fine-tuning on a single GPU
QLoRA (Dettmers et al., 2023) pushes efficiency further: the frozen base model is loaded in 4-bit quantization, and the LoRA adapters train in higher precision on top. A 7B model that needs ~14 GB in fp16 fits in ~5 GB in 4-bit, leaving room for adapters, activations, and optimizer states on a 16 GB GPU — or even a free Colab T4 with careful settings.
The key components:
| Component | What it does |
|---|---|
| 4-bit NormalFloat (NF4) | A quantization format tuned for the distribution of neural network weights |
| Double quantization | Quantizes the quantization constants themselves, saving more memory |
| Paged optimizers | Handles occasional memory spikes by paging optimizer states to CPU |
Practical implication: with QLoRA, a student with one RTX 3060 (12 GB) or a Colab T4 can fine-tune a 7B model. Without it, you would need an A100.
Preparing your data: the part that actually matters
LoRA cannot fix bad data. Your dataset is a set of training examples in the format your task needs. For instruction tuning, the standard is prompt → response pairs:
{
"instruction": "Explain the difference between TCP and UDP.",
"input": "",
"output": "TCP is connection-oriented and guarantees delivery through acknowledgments and retransmission, while UDP is connectionless and sends datagrams without delivery guarantees, trading reliability for lower latency..."
}
Data guidelines that matter more than hyperparameters:
- Quality over quantity. A few hundred excellent examples beat ten thousand mediocre ones. Every example teaches the model what "good" looks like.
- Match the deployment format. If users will ask short questions, train on short questions — not essays. Format mismatch between training and inference is a silent killer.
- Diversity within the task. Cover the range of inputs the model will see: edge cases, different phrasings, different lengths.
- Consistent style. If you want concise answers, every training output should be concise. The model learns the style distribution of your data.
- Clean ruthlessly. Duplicates, truncated outputs, and wrong answers teach the model to reproduce them. Review a random sample by hand.
A realistic student-scale dataset is 500–5,000 examples. Below ~200, you are mostly adjusting style rather than knowledge; that is fine if style is the goal.
Training with Hugging Face PEFT
The standard stack is transformers + peft + trl. The core configuration:
from peft import LoraConfig, get_peft_model
lora_config = LoraConfig(
r=16, # rank: capacity of the adapter
lora_alpha=32, # scaling factor (often 2x rank)
target_modules=["q_proj", "v_proj"], # which layers to adapt
lora_dropout=0.05,
task_type="CAUSAL_LM",
)
model = get_peft_model(base_model, lora_config)
model.print_trainable_parameters()
# trainable params: ~0.5% of total
Sensible starting hyperparameters for a 7B model on a single GPU:
| Hyperparameter | Starting value | Notes |
|---|---|---|
| Rank r | 8–16 | Higher rank = more capacity, more overfitting risk on small data |
| lora_alpha | 2 × r | Scales the adapter's contribution |
| Learning rate | 1e-4 to 2e-4 | Higher than full fine-tuning; adapters are small |
| Epochs | 2–4 | More epochs on small data = memorization |
| Batch size | As large as fits | Use gradient accumulation to simulate larger batches |
| Target modules | q_proj, v_proj | Add k_proj, o_proj, MLP for harder tasks |
Watch the training loss, but do not trust it alone — loss always goes down. Evaluate on a held-out set of prompts you never trained on, and read the outputs yourself.
Evaluating your fine-tune honestly
Fine-tuning evaluation is where most student projects are weakest. "The loss decreased" is not evidence the model got better at your task. Build a small evaluation set — 50–100 prompts representative of real use — and check:
- Task success: does it do the thing? (Correct format, correct domain behavior.)
- Regression: does it still do what the base model did? Ask general questions and confirm the answers did not degrade.
- Consistency: same prompt twice — is the behavior stable?
- Failure modes: deliberately try inputs designed to break it (contradictory instructions, out-of-domain questions).
Keep a frozen copy of the base model and compare side by side on the same prompts. If you cannot tell the fine-tuned model apart from the base model on your eval set, the fine-tune did not work — do not ship it.
Note: Never report accuracy-style claims from your training loss. If you need a number, define a task-specific pass/fail rubric on your held-out eval set and report that, with the rubric published alongside.
Common mistakes
- Too little data, too many epochs. Training 200 examples for 10 epochs memorizes them. The model will parrot training outputs verbatim on similar prompts.
- Wrong chat template. Instruction models expect a specific prompt format (system/user/assistant tokens). Training without the template, then inferring with it — or vice versa — silently breaks behavior. Use the model's tokenizer chat template everywhere.
- Training on the eval set. If your "test prompts" overlap your training data, your evaluation is fiction.
- Rank too high for the data. r=64 on 500 examples gives the adapter enough capacity to memorize. Start low.
- Forgetting the base model comparison. A fine-tune that is no better than prompting the base model well is wasted GPU time. Always try few-shot prompting first — it is free.
- Expecting new knowledge. LoRA is excellent at style, format, and task behavior. It is poor at injecting large amounts of new factual knowledge — that is what RAG is for.
LoRA vs alternatives: when to use what
| Approach | Use when | Cost |
|---|---|---|
| Prompt engineering | Task is within base model capability | Free |
| Few-shot prompting | A few examples steer behavior enough | Free |
| LoRA / QLoRA | Need consistent style, format, or domain behavior | One GPU, hours |
| Full fine-tuning | Pretraining-scale data and multi-GPU budget | Expensive, rarely needed |
| RAG | Need factual knowledge the model lacks | Retrieval infra instead of training |
The decision tree: prompt first, few-shot second, LoRA when behavior must be baked in, RAG when knowledge must be fresh. Most student projects that reach for fine-tuning actually need prompting or RAG.
Where to go from here
If your model needs external knowledge rather than behavior change, read the RAG project guide and then how to evaluate RAG. To run your fine-tuned model on modest hardware, see model quantization. For tracking training runs properly, the MLflow guide covers experiment tracking. More LLM topics in the AI & Machine Learning branch hub.