Skip to content

Fine-Tuning in Practice

What you’ll learn. When fine-tuning is the right tool (and when prompting or RAG is better); full fine-tuning versus parameter-efficient methods; the mathematics and memory accounting of LoRA and QLoRA; how to format and clean training data (chat templates, loss masking); a complete PEFT + TRL training script; merging and serving adapters; catastrophic forgetting and how to mitigate it; and a debugging checklist.

Fine-tuning changes behavior (style, format, domain vocabulary, tool-calling conventions, refusal policy) more reliably than it adds knowledge. Order of attempts:

  1. Prompt better with examples and clear instructions (Prompting).
  2. Retrieve fresh or private facts at inference (RAG). Fine-tuning to inject facts is unreliable, hard to update and prone to hallucination.
  3. Fine-tune when you need: consistent output format or tone at low latency, a small cheap model matching a big one on a narrow task (distillation), behavior that doesn’t fit in the prompt budget, or specialization to a domain’s language.

Build an evaluation set first (Evaluating LLMs); without it you cannot tell whether fine-tuning helped or quietly broke something.

Full fine-tuning updates all NN parameters. Memory with mixed-precision Adam is about 16 bytes/param (Pretraining): 7B needs about 112 GB plus activations. Highest ceiling, highest cost, and each task yields a full-size checkpoint.

Parameter-efficient fine-tuning (PEFT) freezes the base model and trains a tiny number of added parameters (typically 0.1 to 2% of NN). Benefits: optimizer states only for trainable parameters, small artifacts (tens of MB) that you can swap per task or customer, and less forgetting because the base weights are untouched. The cost is some capacity limit, which matters for tasks requiring large behavior shifts or substantial new knowledge (continued pretraining on a new language is an example where full or high-rank training is advisable).

  • Adapters (Houlsby et al., 2019): small bottleneck MLPs inserted after sublayers. Adds inference latency (extra layers).
  • Prefix/prompt tuning: learn continuous vectors prepended to inputs or per-layer keys/values. Cheap but less stable and weaker on smaller models.
  • LoRA (Hu et al., 2021): low-rank updates to weight matrices that can be merged away, so zero inference overhead. The dominant method.
  • Variants: DoRA (decomposes weight into magnitude and direction), rsLoRA (rank-stabilized scaling), LoRA+ (different learning rates for A and B), PiSSA/other SVD-based initializations, and IA3 (learned scaling vectors).

For a frozen weight W0∈Rdout×dinW_0\in\mathbb R^{d_{out}\times d_{in}}, LoRA learns an additive update of rank r≪min⁡(din,dout)r\ll\min(d_{in},d_{out}):

W=W0+αr BA,B∈Rdout×r, A∈Rr×din.W = W_0 + \frac{\alpha}{r}\,BA,\qquad B\in\mathbb R^{d_{out}\times r},\ A\in\mathbb R^{r\times d_{in}}.

AA is initialized randomly (Gaussian) and BB to zero, so training starts exactly at the base model. Only A,BA,B get gradients. Trainable parameters per matrix: r(din+dout)r(d_{in}+d_{out}) instead of dindoutd_{in}d_{out}. For d=4096d=4096, r=16r=16: 131,072131{,}072 versus 16.816.8M, about 0.8%.

Why it works. The hypothesis (Aghajanyan et al., 2020, on intrinsic dimensionality) is that weight updates during adaptation have low effective rank. Empirically LoRA matches full fine-tuning on many instruction-tuning and classification tasks, and is weaker where much new capability must be learned (large-scale math or code skill, new languages), a gap quantified in the paper “LoRA Learns Less and Forgets Less” (Biderman et al., 2024).

Hyperparameters.

  • Rank rr: 8 to 64 is typical; larger for harder tasks. Returns diminish quickly.
  • α\alpha: scaling; commonly α=r\alpha=r or 2r2r. Effective step size scales with α/r\alpha/r.
  • Target modules: originally only attention WQ,WVW_Q,W_V. Applying LoRA to all linear layers (attention projections and MLP) generally works better at the same total rank budget.
  • Learning rate: much higher than full fine-tuning, about 10−410^{-4} to 3×10−43\times10^{-4}.
  • Dropout 0 to 0.1.

A hand-written LoRA layer makes the mechanism concrete:

import math, torch, torch.nn as nn
class LoRALinear(nn.Module):
def __init__(self, base: nn.Linear, r=16, alpha=32, dropout=0.0):
super().__init__()
self.base = base
for p in self.base.parameters(): p.requires_grad = False
self.A = nn.Parameter(torch.empty(r, base.in_features))
self.B = nn.Parameter(torch.zeros(base.out_features, r))
nn.init.kaiming_uniform_(self.A, a=math.sqrt(5))
self.scale = alpha / r
self.drop = nn.Dropout(dropout)
def forward(self, x):
return self.base(x) + self.scale * (self.drop(x) @ self.A.T) @ self.B.T
@torch.no_grad()
def merge(self): # fold the update into the base weight
self.base.weight += self.scale * (self.B @ self.A)
return self.base

After merge(), inference cost equals the base model’s.

QLoRA (Dettmers et al., 2023) fine-tunes LoRA adapters on top of a 4-bit-quantized frozen base model, bringing a 65B model within a single 48 GB GPU in the original paper. Ingredients:

  • NF4 (4-bit NormalFloat): a data type whose levels are placed at quantiles of a normal distribution, near-optimal for roughly Gaussian weights.
  • Double quantization: quantize the per-block quantization constants too, saving about 0.4 bits/param.
  • Paged optimizers: offload optimizer state spikes to CPU memory.
  • Weights are dequantized to BF16 on the fly for each matmul; gradients flow through to the (BF16) LoRA matrices.

Memory for a 7B model: about 3.5 to 4 GB for the 4-bit base plus adapters, activations and optimizer states for the adapters, so it fits a consumer 12-24 GB GPU. Quality is typically close to 16-bit LoRA, with a slowdown from dequantization. Memory also depends heavily on sequence length and batch size; use gradient checkpointing to trade compute for activation memory.

Use the target model’s chat template exactly (Tokenization). A typical record:

{"messages": [
{"role": "system", "content": "You are a support agent for Acme."},
{"role": "user", "content": "How do I reset my password?"},
{"role": "assistant", "content": "Go to Settings > Security > Reset password..."}
]}

Rules of thumb:

  • Train on assistant tokens only (completion-only loss). Training on prompts wastes capacity and can teach the model to generate user turns.
  • Include the end-of-turn/EOS token in the labels so the model learns to stop.
  • For tool calling, serialize calls and results in the same format used at inference time (Agents).
  • Keep the distribution of prompts matched to production: real queries, real noise, real length.
  • Deduplicate, remove contradictory labels, strip PII, and hold out an evaluation set (split by source/customer to avoid leakage).

Hundreds to a few thousand high-quality examples often suffice to fix format and style. More helps for harder tasks, but noisy data hurts more than small data. A useful workflow: fine-tune on 200 examples, inspect failures, fix data, repeat. Synthetic data from a stronger model works well for distillation but inherits its errors and license terms; verify with filters or unit tests where possible.

import torch
from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import LoraConfig
from trl import SFTConfig, SFTTrainer
model_id = "your-org/your-base-or-instruct-model" # any causal LM with a chat template
tok = AutoTokenizer.from_pretrained(model_id)
bnb = BitsAndBytesConfig( # QLoRA: 4-bit NF4 base
load_in_4bit=True, bnb_4bit_quant_type="nf4",
bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=torch.bfloat16,
)
model = AutoModelForCausalLM.from_pretrained(
model_id, quantization_config=bnb, torch_dtype=torch.bfloat16, device_map="auto"
)
peft_cfg = LoraConfig(
r=16, lora_alpha=32, lora_dropout=0.05, task_type="CAUSAL_LM",
target_modules="all-linear", # attention + MLP projections
)
ds = load_dataset("json", data_files={"train": "train.jsonl", "eval": "eval.jsonl"})
cfg = SFTConfig(
output_dir="out",
num_train_epochs=2,
per_device_train_batch_size=4,
gradient_accumulation_steps=8, # effective batch 32
learning_rate=2e-4,
lr_scheduler_type="cosine",
warmup_ratio=0.03,
bf16=True,
gradient_checkpointing=True,
logging_steps=10,
eval_strategy="steps", eval_steps=100,
save_strategy="steps", save_steps=100,
max_length=2048,
assistant_only_loss=True, # loss on assistant tokens (needs a template that marks them)
)
trainer = SFTTrainer(
model=model, args=cfg, train_dataset=ds["train"], eval_dataset=ds["eval"],
peft_config=peft_cfg, processing_class=tok,
)
trainer.train()
trainer.save_model("out/adapter")

TRL and PEFT APIs evolve (argument names such as max_length versus max_seq_length, and how assistant-only loss is specified, have changed between versions). Check the installed version’s documentation; the structure above is stable.

Merge and serve:

from peft import AutoPeftModelForCausalLM
m = AutoPeftModelForCausalLM.from_pretrained("out/adapter", torch_dtype="bfloat16")
m = m.merge_and_unload() # plain model, no adapter overhead
m.save_pretrained("out/merged"); tok.save_pretrained("out/merged")

Merge into a 16-bit base (not the 4-bit one) for best fidelity. Alternatively, serve many adapters on one base model with multi-LoRA support in inference servers such as vLLM (Inference & Serving), switching per request.

Fine-tuning on a narrow dataset can erase general abilities: the model improves on your task while losing instruction following, multilingual ability, safety behavior or reasoning. Causes: large updates to weights important for old tasks, and distribution shift in data.

Mitigations, roughly in order of effectiveness:

  1. Use PEFT with small rank and learning rate (LoRA constrains the update).
  2. Mix in general data (replay): 10-50% instruction/chat data resembling the base model’s post-training distribution; self-generated responses from the base model on general prompts are an excellent replay source.
  3. Fewer epochs, lower learning rate, early stopping on a mixed evaluation that includes general benchmarks.
  4. Regularization toward base weights (L2-SP, EWC; rarely needed in practice) or model merging (interpolate fine-tuned and base weights, e.g. linearly with weight 0.3-0.7) which often recovers general ability at small cost to task performance.
  5. Evaluate for regressions explicitly, including safety: fine-tuning on benign data has been shown to weaken refusal behavior.
  • Continued pretraining (raw domain text, next-token loss) differs from SFT: use it to teach a domain’s language/knowledge or a new language, followed by SFT; needs far more data and often higher-rank adapters or full training.
  • Packing and padding: pack short examples for throughput; sort by length or use bucketed batching.
  • Evaluate generations, not just loss. Eval loss can keep decreasing while outputs overfit style.
  • Overfitting signs: verbatim memorization of training examples, loss collapse toward zero, degraded generality.
  • Cost reasoning: LoRA on a 7B model with a few thousand examples typically takes minutes to a few hours on one GPU.
  • Hosted fine-tuning APIs exist from major providers; they expose a subset of knobs, but hide infrastructure. Check data-retention terms.
  • Distillation: fine-tune a small model on outputs of a large one for a narrow task; often the highest-ROI use of fine-tuning.
  1. Loss doesn’t go down: LR too low/high, labels all masked, template mismatch, bad padding side.
  2. Model never stops talking: EOS missing from labels, or wrong EOS ID.
  3. Outputs ignore training format at inference: inference template differs from the training one.
  4. Great training loss, poor generalization: leakage, too many epochs, duplicates between train and eval.
  5. General regressions: raise replay fraction, lower rank/LR, merge-with-base.
  • Fine-tune for behavior and format; use RAG for knowledge; always have an eval set first.
  • LoRA freezes W0W_0 and learns αrBA\tfrac{\alpha}{r}BA; it is cheap, mergeable (no inference overhead), and works best applied to all linear layers with a higher LR than full FT.
  • QLoRA adds a 4-bit NF4 frozen base to fit large models on one GPU with modest quality loss.
  • Data formatting (exact chat template, assistant-only loss, EOS) causes more failures than hyperparameters.
  • Fight catastrophic forgetting with small updates, replay data, early stopping, and weight-space merging; evaluate regressions including safety.
  1. Conceptual. Compute trainable parameters for LoRA with r=16r=16 on all four attention projections and three MLP matrices of a layer with d=4096d=4096, f=14336f=14336 (square attention projections, SwiGLU MLP). What fraction of that layer’s weights?
  2. Hands-on. Wrap the LoRALinear class around the linear layers of the minimal GPT from the architecture lecture, train only adapters on a small new text style, then merge and verify outputs match pre-merge.
  3. Hands-on. Run the TRL script on a few hundred examples for a format-conversion task (for example, free text to JSON). Compare base few-shot prompting versus the fine-tuned model on 100 held-out items.
  4. Conceptual. Why initialize BB at zero rather than both matrices randomly?
  5. Hands-on. Measure forgetting: evaluate a general benchmark subset before and after fine-tuning, with and without 20% replay data.
Solutions (sketch)
  1. Each adapted matrix has r(din+dout)r(d_{in}+d_{out}) parameters. Attention (4 matrices at 4096×40964096\times4096): 4⋅16⋅8192=524,2884\cdot16\cdot8192=524{,}288. MLP (three matrices of size 4096×143364096\times14336): 3⋅16⋅(4096+14336)=884,7363\cdot16\cdot(4096+14336)=884{,}736. Total about 1.411.41M versus about 4⋅16.8M+3⋅58.7M≈2434\cdot16.8\text{M}+3\cdot58.7\text{M}\approx243M base weights: roughly 0.58%.
  2. With B=0B=0 the initial update BABA is exactly zero, so the model begins identical to the base (no random perturbation), while gradients with respect to BB are nonzero immediately.
  • Hu et al., “LoRA: Low-Rank Adaptation of Large Language Models” (2021).
  • Dettmers et al., “QLoRA: Efficient Finetuning of Quantized LLMs” (2023).
  • Houlsby et al., “Parameter-Efficient Transfer Learning for NLP” (2019); Liu et al., “DoRA” (2024); Liu et al., “IA3” (2022).
  • Biderman et al., “LoRA Learns Less and Forgets Less” (2024).
  • Qi et al., “Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To” (2023).
  • Wortsman et al., “Model soups” (2022), and work on weight-space model merging.
  • Hugging Face documentation for peft, trl, transformers and bitsandbytes.