Fine-Tuning in Practice
What you’ll learn. When fine-tuning is the right tool (and when prompting or RAG is better); full fine-tuning versus parameter-efficient methods; the mathematics and memory accounting of LoRA and QLoRA; how to format and clean training data (chat templates, loss masking); a complete PEFT + TRL training script; merging and serving adapters; catastrophic forgetting and how to mitigate it; and a debugging checklist.
Should you fine-tune at all?
Section titled “Should you fine-tune at all?”Fine-tuning changes behavior (style, format, domain vocabulary, tool-calling conventions, refusal policy) more reliably than it adds knowledge. Order of attempts:
- Prompt better with examples and clear instructions (Prompting).
- Retrieve fresh or private facts at inference (RAG). Fine-tuning to inject facts is unreliable, hard to update and prone to hallucination.
- Fine-tune when you need: consistent output format or tone at low latency, a small cheap model matching a big one on a narrow task (distillation), behavior that doesn’t fit in the prompt budget, or specialization to a domain’s language.
Build an evaluation set first (Evaluating LLMs); without it you cannot tell whether fine-tuning helped or quietly broke something.
Full fine-tuning versus PEFT
Section titled “Full fine-tuning versus PEFT”Full fine-tuning updates all parameters. Memory with mixed-precision Adam is about 16 bytes/param (Pretraining): 7B needs about 112 GB plus activations. Highest ceiling, highest cost, and each task yields a full-size checkpoint.
Parameter-efficient fine-tuning (PEFT) freezes the base model and trains a tiny number of added parameters (typically 0.1 to 2% of ). Benefits: optimizer states only for trainable parameters, small artifacts (tens of MB) that you can swap per task or customer, and less forgetting because the base weights are untouched. The cost is some capacity limit, which matters for tasks requiring large behavior shifts or substantial new knowledge (continued pretraining on a new language is an example where full or high-rank training is advisable).
Families
Section titled “Families”- Adapters (Houlsby et al., 2019): small bottleneck MLPs inserted after sublayers. Adds inference latency (extra layers).
- Prefix/prompt tuning: learn continuous vectors prepended to inputs or per-layer keys/values. Cheap but less stable and weaker on smaller models.
- LoRA (Hu et al., 2021): low-rank updates to weight matrices that can be merged away, so zero inference overhead. The dominant method.
- Variants: DoRA (decomposes weight into magnitude and direction), rsLoRA (rank-stabilized scaling), LoRA+ (different learning rates for A and B), PiSSA/other SVD-based initializations, and IA3 (learned scaling vectors).
LoRA in detail
Section titled “LoRA in detail”For a frozen weight , LoRA learns an additive update of rank :
is initialized randomly (Gaussian) and to zero, so training starts exactly at the base model. Only get gradients. Trainable parameters per matrix: instead of . For , : versus M, about 0.8%.
Why it works. The hypothesis (Aghajanyan et al., 2020, on intrinsic dimensionality) is that weight updates during adaptation have low effective rank. Empirically LoRA matches full fine-tuning on many instruction-tuning and classification tasks, and is weaker where much new capability must be learned (large-scale math or code skill, new languages), a gap quantified in the paper “LoRA Learns Less and Forgets Less” (Biderman et al., 2024).
Hyperparameters.
- Rank : 8 to 64 is typical; larger for harder tasks. Returns diminish quickly.
- : scaling; commonly or . Effective step size scales with .
- Target modules: originally only attention . Applying LoRA to all linear layers (attention projections and MLP) generally works better at the same total rank budget.
- Learning rate: much higher than full fine-tuning, about to .
- Dropout 0 to 0.1.
A hand-written LoRA layer makes the mechanism concrete:
import math, torch, torch.nn as nn
class LoRALinear(nn.Module): def __init__(self, base: nn.Linear, r=16, alpha=32, dropout=0.0): super().__init__() self.base = base for p in self.base.parameters(): p.requires_grad = False self.A = nn.Parameter(torch.empty(r, base.in_features)) self.B = nn.Parameter(torch.zeros(base.out_features, r)) nn.init.kaiming_uniform_(self.A, a=math.sqrt(5)) self.scale = alpha / r self.drop = nn.Dropout(dropout)
def forward(self, x): return self.base(x) + self.scale * (self.drop(x) @ self.A.T) @ self.B.T
@torch.no_grad() def merge(self): # fold the update into the base weight self.base.weight += self.scale * (self.B @ self.A) return self.baseAfter merge(), inference cost equals the base model’s.
QLoRA (Dettmers et al., 2023) fine-tunes LoRA adapters on top of a 4-bit-quantized frozen base model, bringing a 65B model within a single 48 GB GPU in the original paper. Ingredients:
- NF4 (4-bit NormalFloat): a data type whose levels are placed at quantiles of a normal distribution, near-optimal for roughly Gaussian weights.
- Double quantization: quantize the per-block quantization constants too, saving about 0.4 bits/param.
- Paged optimizers: offload optimizer state spikes to CPU memory.
- Weights are dequantized to BF16 on the fly for each matmul; gradients flow through to the (BF16) LoRA matrices.
Memory for a 7B model: about 3.5 to 4 GB for the 4-bit base plus adapters, activations and optimizer states for the adapters, so it fits a consumer 12-24 GB GPU. Quality is typically close to 16-bit LoRA, with a slowdown from dequantization. Memory also depends heavily on sequence length and batch size; use gradient checkpointing to trade compute for activation memory.
Data: the part that decides success
Section titled “Data: the part that decides success”Formatting
Section titled “Formatting”Use the target model’s chat template exactly (Tokenization). A typical record:
{"messages": [ {"role": "system", "content": "You are a support agent for Acme."}, {"role": "user", "content": "How do I reset my password?"}, {"role": "assistant", "content": "Go to Settings > Security > Reset password..."}]}Rules of thumb:
- Train on assistant tokens only (completion-only loss). Training on prompts wastes capacity and can teach the model to generate user turns.
- Include the end-of-turn/EOS token in the labels so the model learns to stop.
- For tool calling, serialize calls and results in the same format used at inference time (Agents).
- Keep the distribution of prompts matched to production: real queries, real noise, real length.
- Deduplicate, remove contradictory labels, strip PII, and hold out an evaluation set (split by source/customer to avoid leakage).
Quantity and quality
Section titled “Quantity and quality”Hundreds to a few thousand high-quality examples often suffice to fix format and style. More helps for harder tasks, but noisy data hurts more than small data. A useful workflow: fine-tune on 200 examples, inspect failures, fix data, repeat. Synthetic data from a stronger model works well for distillation but inherits its errors and license terms; verify with filters or unit tests where possible.
A complete PEFT + TRL script
Section titled “A complete PEFT + TRL script”import torchfrom datasets import load_datasetfrom transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfigfrom peft import LoraConfigfrom trl import SFTConfig, SFTTrainer
model_id = "your-org/your-base-or-instruct-model" # any causal LM with a chat templatetok = AutoTokenizer.from_pretrained(model_id)
bnb = BitsAndBytesConfig( # QLoRA: 4-bit NF4 base load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=torch.bfloat16,)model = AutoModelForCausalLM.from_pretrained( model_id, quantization_config=bnb, torch_dtype=torch.bfloat16, device_map="auto")
peft_cfg = LoraConfig( r=16, lora_alpha=32, lora_dropout=0.05, task_type="CAUSAL_LM", target_modules="all-linear", # attention + MLP projections)
ds = load_dataset("json", data_files={"train": "train.jsonl", "eval": "eval.jsonl"})
cfg = SFTConfig( output_dir="out", num_train_epochs=2, per_device_train_batch_size=4, gradient_accumulation_steps=8, # effective batch 32 learning_rate=2e-4, lr_scheduler_type="cosine", warmup_ratio=0.03, bf16=True, gradient_checkpointing=True, logging_steps=10, eval_strategy="steps", eval_steps=100, save_strategy="steps", save_steps=100, max_length=2048, assistant_only_loss=True, # loss on assistant tokens (needs a template that marks them))
trainer = SFTTrainer( model=model, args=cfg, train_dataset=ds["train"], eval_dataset=ds["eval"], peft_config=peft_cfg, processing_class=tok,)trainer.train()trainer.save_model("out/adapter")TRL and PEFT APIs evolve (argument names such as max_length versus max_seq_length, and how assistant-only loss is specified, have changed between versions). Check the installed version’s documentation; the structure above is stable.
Merge and serve:
from peft import AutoPeftModelForCausalLMm = AutoPeftModelForCausalLM.from_pretrained("out/adapter", torch_dtype="bfloat16")m = m.merge_and_unload() # plain model, no adapter overheadm.save_pretrained("out/merged"); tok.save_pretrained("out/merged")Merge into a 16-bit base (not the 4-bit one) for best fidelity. Alternatively, serve many adapters on one base model with multi-LoRA support in inference servers such as vLLM (Inference & Serving), switching per request.
Catastrophic forgetting
Section titled “Catastrophic forgetting”Fine-tuning on a narrow dataset can erase general abilities: the model improves on your task while losing instruction following, multilingual ability, safety behavior or reasoning. Causes: large updates to weights important for old tasks, and distribution shift in data.
Mitigations, roughly in order of effectiveness:
- Use PEFT with small rank and learning rate (LoRA constrains the update).
- Mix in general data (replay): 10-50% instruction/chat data resembling the base model’s post-training distribution; self-generated responses from the base model on general prompts are an excellent replay source.
- Fewer epochs, lower learning rate, early stopping on a mixed evaluation that includes general benchmarks.
- Regularization toward base weights (L2-SP, EWC; rarely needed in practice) or model merging (interpolate fine-tuned and base weights, e.g. linearly with weight 0.3-0.7) which often recovers general ability at small cost to task performance.
- Evaluate for regressions explicitly, including safety: fine-tuning on benign data has been shown to weaken refusal behavior.
Other practical notes
Section titled “Other practical notes”- Continued pretraining (raw domain text, next-token loss) differs from SFT: use it to teach a domain’s language/knowledge or a new language, followed by SFT; needs far more data and often higher-rank adapters or full training.
- Packing and padding: pack short examples for throughput; sort by length or use bucketed batching.
- Evaluate generations, not just loss. Eval loss can keep decreasing while outputs overfit style.
- Overfitting signs: verbatim memorization of training examples, loss collapse toward zero, degraded generality.
- Cost reasoning: LoRA on a 7B model with a few thousand examples typically takes minutes to a few hours on one GPU.
- Hosted fine-tuning APIs exist from major providers; they expose a subset of knobs, but hide infrastructure. Check data-retention terms.
- Distillation: fine-tune a small model on outputs of a large one for a narrow task; often the highest-ROI use of fine-tuning.
Debugging checklist
Section titled “Debugging checklist”- Loss doesn’t go down: LR too low/high, labels all masked, template mismatch, bad padding side.
- Model never stops talking: EOS missing from labels, or wrong EOS ID.
- Outputs ignore training format at inference: inference template differs from the training one.
- Great training loss, poor generalization: leakage, too many epochs, duplicates between train and eval.
- General regressions: raise replay fraction, lower rank/LR, merge-with-base.
Key takeaways
Section titled “Key takeaways”- Fine-tune for behavior and format; use RAG for knowledge; always have an eval set first.
- LoRA freezes and learns ; it is cheap, mergeable (no inference overhead), and works best applied to all linear layers with a higher LR than full FT.
- QLoRA adds a 4-bit NF4 frozen base to fit large models on one GPU with modest quality loss.
- Data formatting (exact chat template, assistant-only loss, EOS) causes more failures than hyperparameters.
- Fight catastrophic forgetting with small updates, replay data, early stopping, and weight-space merging; evaluate regressions including safety.
Exercises
Section titled “Exercises”- Conceptual. Compute trainable parameters for LoRA with on all four attention projections and three MLP matrices of a layer with , (square attention projections, SwiGLU MLP). What fraction of that layer’s weights?
- Hands-on. Wrap the
LoRALinearclass around the linear layers of the minimal GPT from the architecture lecture, train only adapters on a small new text style, then merge and verify outputs match pre-merge. - Hands-on. Run the TRL script on a few hundred examples for a format-conversion task (for example, free text to JSON). Compare base few-shot prompting versus the fine-tuned model on 100 held-out items.
- Conceptual. Why initialize at zero rather than both matrices randomly?
- Hands-on. Measure forgetting: evaluate a general benchmark subset before and after fine-tuning, with and without 20% replay data.
Solutions (sketch)
- Each adapted matrix has parameters. Attention (4 matrices at ): . MLP (three matrices of size ): . Total about M versus about M base weights: roughly 0.58%.
- With the initial update is exactly zero, so the model begins identical to the base (no random perturbation), while gradients with respect to are nonzero immediately.
Further reading
Section titled “Further reading”- Hu et al., “LoRA: Low-Rank Adaptation of Large Language Models” (2021).
- Dettmers et al., “QLoRA: Efficient Finetuning of Quantized LLMs” (2023).
- Houlsby et al., “Parameter-Efficient Transfer Learning for NLP” (2019); Liu et al., “DoRA” (2024); Liu et al., “IA3” (2022).
- Biderman et al., “LoRA Learns Less and Forgets Less” (2024).
- Qi et al., “Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To” (2023).
- Wortsman et al., “Model soups” (2022), and work on weight-space model merging.
- Hugging Face documentation for
peft,trl,transformersandbitsandbytes.