Post-Training: SFT, Preference Tuning & RL
What you’ll learn. The stages that follow pretraining: supervised fine-tuning on demonstrations, learning from preferences with reward models and PPO (RLHF), the direct methods (DPO, IPO, KTO, ORPO, SimPO), AI feedback (RLAIF, constitutional methods), reinforcement learning with verifiable rewards and GRPO that drives modern reasoning models, and safety tuning. We derive the key objectives, show code, and discuss failure modes such as reward hacking, sycophancy and over-refusal.
Why post-training exists
Section titled “Why post-training exists”A base model (Pretraining) is a document continuer: given “What is the capital of France?” it may continue with more quiz questions. Post-training reshapes the output distribution so the model follows instructions, adopts a persona, refuses harmful requests, uses tools and, more recently, thinks for longer before answering. Compute for post-training is a small fraction of pretraining, but it dominates perceived quality. A useful mental model (the “superficial alignment” hypothesis from LIMA) is that most knowledge comes from pretraining and post-training mainly selects and shapes behavior, though RL on verifiable tasks shows it can also teach genuinely new skills such as long multi-step reasoning.
A typical pipeline:
base model -> SFT (demonstrations) -> preference optimization (RLHF / DPO) -> RL with verifiable rewards (math, code, tools) -> safety tuning, distillation(stages are interleaved and iterated in practice)Supervised fine-tuning (SFT)
Section titled “Supervised fine-tuning (SFT)”Data. Pairs of (prompt, ideal response), formatted with the chat template. Sources: human-written demonstrations (high quality, expensive), curated open datasets, and synthetic data generated by stronger models (Self-Instruct, Alpaca-style distillation, evol-instruct). Quality, diversity and correctness matter more than quantity: thousands of careful examples can already change style; frontier SFT mixes use up to millions covering code, math, reasoning, safety, multilingual and tool use.
Loss. Standard next-token cross entropy, but computed only on the assistant tokens (prompt tokens are masked with label ), so the model learns to answer, not to predict user messages:
Details: pack multiple short examples per sequence (with attention masking between them) for efficiency; use small learning rates ( to for full fine-tuning) and 1-3 epochs; over-training causes overfitting and style collapse. SFT teaches imitation, so it inherits annotator errors and cannot exceed demonstrators, and it can teach hallucination: training on answers containing facts the model does not know pushes it to fabricate plausible text.
RLHF: reward models and PPO
Section titled “RLHF: reward models and PPO”Preference data is easier to collect than demonstrations: show annotators two responses and ask which is better. Ouyang et al. (InstructGPT, 2022) popularized three stages: SFT, reward model, RL.
Reward model
Section titled “Reward model”Fit a scalar reward with the Bradley-Terry model: the probability that response is preferred over is , so
The reward model is usually the LM with a scalar head, initialized from the SFT model.
KL-regularized RL
Section titled “KL-regularized RL”Maximize reward while staying close to the reference (SFT) policy :
The KL term prevents the policy from drifting into gibberish that exploits reward-model errors. It is optimized with PPO: sample responses, score them, estimate advantages with a learned value function (critic), and take clipped policy-gradient steps. PPO needs four models in memory (policy, reference, reward, critic), is sensitive to hyperparameters, and is fiddly, which motivated simpler alternatives.
Reward hacking
Section titled “Reward hacking”Optimizing a proxy reward eventually hurts the true objective (Goodhart’s law; Gao et al. 2022 measured the over-optimization curve against KL). Typical hacks: longer answers, flattery (sycophancy: agreeing with the user’s stated beliefs), formatting tricks, confident tone regardless of correctness. Mitigations: KL penalty, early stopping on a held-out gold reward, ensembling reward models, length control, and diverse prompts.
DPO and the direct family
Section titled “DPO and the direct family”Direct Preference Optimization (Rafailov et al., 2023) observes that the KL-regularized objective has the closed-form optimum . Inverting gives the reward as a log-ratio, . Substituting into Bradley-Terry cancels the intractable and yields a pure classification loss on preference pairs:
No reward model, no sampling loop, no critic: just two models (policy and frozen reference) and supervised-style training. The (typically 0.1 to 0.5) controls how far the policy may move from the reference.
import torch, torch.nn.functional as F
def sequence_logp(model, input_ids, labels): """Sum of log-probs of response tokens (labels == -100 are masked).""" logits = model(input_ids).logits[:, :-1] tgt = labels[:, 1:] mask = tgt != -100 lp = torch.gather(logits.log_softmax(-1), 2, tgt.clamp(min=0).unsqueeze(-1)).squeeze(-1) return (lp * mask).sum(-1)
def dpo_loss(policy, ref, chosen, rejected, beta=0.1): pc = sequence_logp(policy, *chosen); pr = sequence_logp(policy, *rejected) with torch.no_grad(): rc = sequence_logp(ref, *chosen); rr = sequence_logp(ref, *rejected) margin = beta * ((pc - rc) - (pr - rr)) return -F.logsigmoid(margin).mean()Known issues and variants. DPO is offline: it trains on a fixed dataset, so the policy can drift off the data distribution, and it can reduce the likelihood of both chosen and rejected responses. Variants address specific problems:
- IPO: replaces the log-sigmoid with a squared loss to resist overfitting to deterministic preferences.
- KTO: uses only binary “good/bad” labels (prospect-theory-inspired), no pairs.
- ORPO: merges SFT and preference loss, no reference model.
- SimPO: reference-free, uses length-normalized average log-prob as the implicit reward plus a margin.
- Online/iterative DPO: sample fresh responses from the current policy, label them (with a reward model or judge), and repeat, recovering on-policy benefits.
Which is best is dataset- and model-dependent; differences are often smaller than data-quality effects. Preference data quality (low noise, clear margins, on-policy samples) matters most.
AI feedback: RLAIF and constitutional methods
Section titled “AI feedback: RLAIF and constitutional methods”Human labels are slow and costly. RLAIF replaces some or all human preference labels with judgments from an LLM. Constitutional AI (Bai et al., 2022) has the model critique and revise its own outputs against a written list of principles (supervised stage), then uses AI-labelled preferences for the RL stage. Advantages: scalable, consistent, principles are explicit and auditable. Risks: the judge’s biases (position bias, verbosity bias, self-preference) are baked in; errors correlate with the generator’s. Usually combined with human data for calibration. See Evaluating LLMs for LLM-judge pitfalls.
RL with verifiable rewards and GRPO
Section titled “RL with verifiable rewards and GRPO”For tasks with automatically checkable answers (math with a final answer, code with unit tests, formal proofs, structured-output format checks, some tool-use outcomes), you don’t need a learned reward model: RLVR (reinforcement learning with verifiable rewards) uses a programmatic check as the reward. This removes much of the reward-hacking surface of learned preference models (though not all: the verifier can be gamed, e.g. unit tests special-cased).
GRPO (Group Relative Policy Optimization, introduced in DeepSeekMath, 2024) is a critic-free PPO variant. For each prompt , sample a group of responses, score each with reward , and use the group-normalized reward as the advantage:
The update is a clipped policy-gradient objective on token log-probability ratios, with KL regularization to a reference:
The group mean acts as the baseline, so no value network is needed, which saves memory. Follow-up work (DAPO, Dr. GRPO and others) fixes details such as length normalization bias, asymmetric clipping, dynamic filtering of prompts whose group rewards are all equal (zero gradient), and removing the KL term for long-horizon runs; the literature here is moving quickly, so treat any specific recipe as provisional. A closely related, simpler family is REINFORCE-style leave-one-out estimators (RLOO).
import torch
def grpo_advantages(rewards): # rewards: (num_prompts, G) mean = rewards.mean(1, keepdim=True) std = rewards.std(1, keepdim=True) + 1e-6 return (rewards - mean) / std
def grpo_policy_loss(logp_new, logp_old, adv, mask, eps=0.2): # logp_*: (B*G, T) token log-probs; adv: (B*G, 1); mask: (B*G, T) ratio = (logp_new - logp_old).exp() unclipped = ratio * adv clipped = ratio.clamp(1 - eps, 1 + eps) * adv per_tok = -torch.minimum(unclipped, clipped) return ((per_tok * mask).sum(1) / mask.sum(1)).mean()Reasoning models
Section titled “Reasoning models”Applying RLVR with long generation budgets produced the reasoning models of 2024-2026. With reward only on final-answer correctness, models spontaneously learn longer chains of thought: self-verification, backtracking, trying alternative approaches (DeepSeek-R1 and open replications documented this; closed labs reported similar results). Typical recipe: a short “cold-start” SFT on reasoning traces for readable format, large-scale RL on math/code/logic, rejection-sampling SFT on the best traces, a final preference/safety stage, and distillation of long-CoT traces into smaller models. We discuss test-time scaling and what is and isn’t understood in Reasoning, Multimodality & Frontier Trends.
Practical hazards: format/length hacking (padding reasoning), verifier exploits, entropy collapse (policy becomes deterministic and stops exploring), and capability gains concentrated on the training task types. Whether RL expands capability beyond what the base model can already sample (pass@k analyses suggest it mostly sharpens in some regimes) is debated.
Safety tuning
Section titled “Safety tuning”Safety is a post-training objective alongside helpfulness:
- Refusal training: SFT and preference data containing harmful requests with safe refusals or safe completions. Balance is crucial: over-refusal (rejecting benign borderline requests) is as real a failure as harmful compliance; include “borderline-but-fine” prompts.
- Red-team data loops: adversarial prompts (jailbreaks, role-play, encoding tricks) found by humans or automated attackers are folded back into training (Evaluating LLMs).
- Rule/policy-based rewards and deliberative approaches: models trained to reason about a written policy before answering.
- Honesty and calibration: training to express uncertainty, say “I don’t know,” and avoid sycophancy.
- Classifiers and system-level defenses complement model training; see Building LLM Applications.
Safety tuning is shallow if done only on the first few tokens of a response (models can be fine-tuned or prompted around it); research on deeper robustness (training on recovering from harmful prefixes, circuit-breaker style interventions) continues. Fine-tuning can also erode safety, even on benign data, which is relevant to anyone customizing a model (Fine-Tuning in Practice).
Practical guidance
Section titled “Practical guidance”- Start with SFT; add DPO-style preference tuning when you have pairs and a quality gap SFT can’t close; use RLVR when you have a cheap, reliable verifier.
- Always keep a held-out set judged by something independent of the training reward; monitor KL, response length and reward versus held-out score.
- Mix in general data when specializing to avoid regressions.
- Tooling: Hugging Face TRL (SFTTrainer, DPOTrainer, GRPOTrainer), OpenRLHF and verl for scaled RL. Code examples in Fine-Tuning in Practice.
Key takeaways
Section titled “Key takeaways”- SFT imitates demonstrations with loss on response tokens only; quality beats quantity, and it can teach hallucination.
- RLHF = reward model (Bradley-Terry) + KL-regularized policy optimization; vulnerable to reward hacking and sycophancy.
- DPO turns the same objective into a closed-form classification loss over preference pairs, with simpler variants (IPO, KTO, ORPO, SimPO) and on-policy/iterative forms.
- RLVR with GRPO uses programmatic rewards and group-normalized advantages, no critic; it underlies reasoning models.
- Safety tuning must balance refusal and helpfulness, and is not a complete defense on its own.
Exercises
Section titled “Exercises”- Conceptual. Derive the optimal policy of the KL-regularized objective and show how it leads to the DPO loss.
- Conceptual. Why is the group mean a valid baseline in GRPO, and what happens if all samples get the same reward?
- Hands-on. Fine-tune a small model with TRL’s
DPOTraineron a public preference dataset; plot chosen and rejected log-ratios over training. Do both decrease? - Hands-on. Write a verifiable reward function for a “multiply two 3-digit numbers” task with a strict answer format, including partial format reward. Consider how a policy might exploit it.
- Conceptual. Give three ways an LLM judge used for RLAIF labels can be biased and how to test for each.
Solutions (sketch)
- Lagrangian/Gibbs variational result: . Solve for , plug into Bradley-Terry; cancels in the difference between winner and loser.
- A baseline that doesn’t depend on the sampled action keeps the gradient unbiased (the leave-in mean is slightly biased but harmless; RLOO removes it). If all rewards are equal, advantages are zero and the group contributes no gradient, which is why dynamic sampling filters such prompts.
- Position bias (swap answer order), verbosity bias (control for length), self-preference (compare judges from different families); test via swap-consistency and agreement with human labels on a calibration set.
Further reading
Section titled “Further reading”- Ouyang et al., “Training language models to follow instructions with human feedback” (InstructGPT, 2022).
- Christiano et al., “Deep reinforcement learning from human preferences” (2017); Stiennon et al., “Learning to summarize from human feedback” (2020).
- Rafailov et al., “Direct Preference Optimization” (2023); Azar et al. (IPO, 2023); Ethayarajh et al. (KTO, 2024); Meng et al. (SimPO, 2024).
- Bai et al., “Constitutional AI: Harmlessness from AI Feedback” (2022).
- Zhou et al., “LIMA: Less Is More for Alignment” (2023).
- Shao et al., “DeepSeekMath” (2024, GRPO); DeepSeek-AI, “DeepSeek-R1” (2025); Schulman et al., “Proximal Policy Optimization Algorithms” (2017).
- Gao, Schulman, Hilton, “Scaling Laws for Reward Model Overoptimization” (2022).
- Lambert, RLHF Book (online); Hugging Face TRL documentation.