Diffusion & Image Generation
What you’ll learn. The forward/reverse diffusion process and the DDPM training objective with its derivation sketch; the score-matching view and the SDE/ODE formulation; samplers (DDIM, DPM-Solver, few-step distillation); latent diffusion (Stable Diffusion); classifier-free guidance; diffusion transformers (DiT) and modern text-to-image stacks; flow matching and rectified flow (the basis of recent frontier open models); and adapters (LoRA, ControlNet, IP-Adapter). Code is minimal PyTorch. Prerequisites: Segmentation (U-Net), Vision Transformers, and basic probability.
The generative modeling picture
Section titled “The generative modeling picture”We want to sample from an unknown data distribution (images) given examples. Earlier approaches: GANs (adversarial, unstable, mode dropping), VAEs (blurry), autoregressive models over pixels or tokens (slow, but see VQ-token models). Diffusion models turn generation into many easy denoising steps: stable training with a simple regression loss, strong mode coverage, at the cost of iterative sampling.
Forward process
Section titled “Forward process”Define a Markov chain that gradually adds Gaussian noise over steps (typically ) with a variance schedule :
With and , the marginal has a closed form:
As , and : pure noise. Schedules: linear (DDPM), cosine (improved DDPM). The schedule matters: a schedule whose final step still leaks signal leads to wrong brightness (a known issue fixed by “zero terminal SNR” schedules).
Reverse process and the DDPM loss
Section titled “Reverse process and the DDPM loss”The true reverse is intractable, but conditional on it is Gaussian: with
We parameterize and maximize the variational lower bound (ELBO). The bound decomposes into KL terms between Gaussians, each reducing to a squared error between means. Reparametrizing the mean through a noise-prediction network and dropping weights as Ho et al. did gives the simple loss:
Intuition: train a network to guess the noise that was added; at each noise level this is equivalent to estimating the clean image.
import torch, torch.nn.functional as F
T = 1000betas = torch.linspace(1e-4, 0.02, T)alphas = 1 - betasabar = torch.cumprod(alphas, 0)
def train_step(model, x0): # x0: (B,C,H,W) scaled to [-1,1] B = x0.size(0) t = torch.randint(0, T, (B,), device=x0.device) eps = torch.randn_like(x0) ab = abar.to(x0.device)[t].view(B, 1, 1, 1) xt = ab.sqrt() * x0 + (1 - ab).sqrt() * eps return F.mse_loss(model(xt, t), eps)
@torch.no_grad()def ddpm_sample(model, shape, device): x = torch.randn(shape, device=device) for t in reversed(range(T)): tt = torch.full((shape[0],), t, device=device, dtype=torch.long) eps = model(x, tt) a, ab, b = alphas[t], abar[t], betas[t] mean = (x - b / (1 - ab).sqrt() * eps) / a.sqrt() x = mean + (b.sqrt() * torch.randn_like(x) if t > 0 else 0) return xThe network (originally a U-Net with ResNet blocks, self-attention at low resolution, timestep embeddings injected via FiLM/adaptive normalization) takes and predicts . Alternative parameterizations: predict , or -prediction (), which is better conditioned at high noise levels and used by many modern models.
Score matching and the SDE view
Section titled “Score matching and the SDE view”The score is . Denoising score matching shows that for , the optimal denoiser gives the score of the noised distribution: (for the variance-preserving case, ). So noise prediction is score estimation up to scaling. Song et al. unified DDPM and score-based models via SDEs: the forward noising is
and the reverse-time SDE (Anderson, 1982) is
Replace the score by the learned network. A key corollary is the probability flow ODE, with the same marginals but no noise:
Sampling then is solving an ODE, which allows deterministic sampling, invertibility (latent encoding) and fast numerical solvers.
Samplers
Section titled “Samplers”- DDIM: deterministic non-Markovian sampler consistent with the DDPM-trained model, 20-50 steps instead of 1000.
- DPM-Solver / DPM-Solver++, UniPC, Heun/EDM samplers (Karras et al., “Elucidating the Design Space”, EDM): higher-order ODE solvers reaching good quality in about 10-25 steps. Karras et al. also clarified preconditioning and noise-level parametrization.
- Distillation to few steps: progressive distillation, consistency models (map any point on the ODE trajectory to its origin, 1-4 steps), Latent Consistency Models, adversarial diffusion distillation (SDXL-Turbo), DMD (distribution matching). Quality/diversity trade-offs; useful for real-time applications.
Latent diffusion
Section titled “Latent diffusion”Running a U-Net at pixels is expensive. Latent Diffusion Models (Rombach et al., 2022; Stable Diffusion) first train an autoencoder (perceptual + adversarial losses, small KL regularization) to compress images by a factor per side (e.g., , 48x fewer elements), then train diffusion in latent space. Decoder returns pixels. The autoencoder handles imperceptible detail; diffusion handles semantics and layout. Modern models use 16 channels or more per latent (SD3, FLUX) for better reconstruction.
Text conditioning: encode the prompt with a text encoder (CLIP; later T5/other LLM encoders, often several) and inject via cross-attention from image tokens to text tokens in the denoiser.
prompt --> text encoder(s) --> text embeddings ---------------+ vnoise z_T ~ N(0,I) (64x64x4) --> [denoiser x ~30 steps, cross-attn] --> z_0 --> VAE decoder --> imageClassifier-free guidance (CFG)
Section titled “Classifier-free guidance (CFG)”We want samples that match the condition (a prompt) more strongly than the conditional model alone provides. Ho and Salimans: train a single network with the condition randomly dropped (e.g., 10% of the time replaced by a null token), obtaining both and . At sampling:
Since , this follows the gradient of , i.e., it sharpens the implicit classifier. Guidance scale of about 5-8 for Stable Diffusion-like models (guidance-distilled models bake it in). Higher : better prompt adherence and fidelity but less diversity and over-saturation. Cost: two network evaluations per step (batched), unless distilled. Negative prompts replace with an unwanted-concept embedding.
Diffusion Transformers (DiT) and modern architectures
Section titled “Diffusion Transformers (DiT) and modern architectures”DiT (Peebles and Xie, 2022) replaces the U-Net with a standard ViT operating on latent patches (Vision Transformers). Timestep and class/text conditioning are injected with adaLN-Zero: LayerNorm scale/shift (and a residual gate initialized to zero) regressed from the conditioning vector. DiTs scale predictably: lower FID with more Gflops, in the usual transformer scaling-law fashion, and benefit from decades of transformer optimization.
Successors: MMDiT (Stable Diffusion 3), in which text tokens and image tokens have separate weights but attend jointly in a single sequence; FLUX-family models (hybrid double- and single-stream blocks); PixArt, HunyuanDiT, Qwen-Image and others. Video generation (Sora-style and open video models) uses space-time latent patches and DiT backbones (Video, 3D & Spatial Vision). I describe lineage rather than rankings, because leaderboards and releases turn over every few months; verify current options.
Flow matching and rectified flow
Section titled “Flow matching and rectified flow”Flow matching (Lipman et al., 2022) learns a velocity field whose ODE transports noise to data. In the simplest (rectified flow; Liu et al.; Albergo and Vanden-Eijnden) formulation, interpolate linearly between a data sample and noise :
and train
Sampling integrates the ODE from (noise) to (data), with Euler or higher-order steps. Why it caught on:
- A simple regression target with straight-line paths, which are friendlier to coarse ODE solvers (fewer steps) than curved diffusion paths. “Reflow” iteratively straightens paths.
- Conceptually simpler than SDE machinery; diffusion is a special case with a different path/weighting.
- SD3 combined rectified flow with logit-normal timestep sampling (focusing training on mid noise levels) and MMDiT, and FLUX followed in the same family, so most frontier open image models since 2024 use flow-matching objectives.
def fm_loss(model, x0): # x0 are latents B = x0.size(0) t = torch.rand(B, device=x0.device).view(B, 1, 1, 1) x1 = torch.randn_like(x0) xt = (1 - t) * x0 + t * x1 return F.mse_loss(model(xt, t.flatten()), x1 - x0)
@torch.no_grad()def euler_sample(model, shape, device, steps=28): x = torch.randn(shape, device=device) ts = torch.linspace(1, 0, steps + 1, device=device) for i in range(steps): dt = ts[i + 1] - ts[i] # negative x = x + model(x, ts[i].expand(shape[0])) * dt return xEquivalence note: with Gaussian paths, flow matching and diffusion parameterizations are related by reparametrization of the noise schedule and the target (velocity vs noise vs score); the practical differences are the path, weighting, and sampler conventions.
Adapters and customization
Section titled “Adapters and customization”Fine-tuning a multi-billion-parameter generator is expensive; parameter-efficient methods dominate:
- LoRA (Hu et al., 2021, applied to diffusion): freeze weight and learn a low-rank update with ( of 4-64). Small files (10-200 MB), mixable. Used for styles, characters, concepts. DreamBooth fine-tunes on 3-20 images of a subject with a rare-token identifier plus a prior-preservation loss; Textual Inversion learns only a new token embedding.
- ControlNet (Zhang et al., 2023): clone the encoder half of the denoiser, connect it through zero-initialized convolutions, and feed spatial conditions (Canny edges, depth, pose, segmentation, scribbles). The original weights stay frozen so quality is preserved; zero-init means training starts as an identity modification. T2I-Adapter is a lighter alternative.
- IP-Adapter: decoupled cross-attention for image prompts (style/identity from a reference image).
- Instruction-based editing models (InstructPix2Pix and in-context editing models built on large DiTs) take an input image plus a text instruction.
# diffusers: text-to-image with a LoRA (model names change; check the Hub)import torchfrom diffusers import DiffusionPipelinepipe = DiffusionPipeline.from_pretrained("stabilityai/stable-diffusion-xl-base-1.0", torch_dtype=torch.float16).to("cuda")# pipe.load_lora_weights("path/to/lora")img = pipe("a watercolor fox in a forest", num_inference_steps=30, guidance_scale=6.0).images[0]img.save("fox.png")Evaluation and pitfalls
Section titled “Evaluation and pitfalls”- FID (Fréchet Inception Distance) compares Gaussian fits of Inception features of real versus generated sets; sensitive to sample size and feature extractor, and does not capture prompt alignment. CLIP score and newer text-image alignment metrics (GenEval, T2I-CompBench, human preference models like ImageReward/PickScore) measure adherence; human evaluation is still the arbiter. Precision/recall for generative models separates fidelity from diversity.
- Failure modes: text rendering (improving with T5/LLM encoders and large DiTs), counting and attribute binding, hands/anatomy, over-saturation at high CFG, memorization of training images.
- Safety and provenance: watermarking, content filters, copyright and consent debates; deepfake risk. Apply organizational policy; this is not only a technical issue.
- Memory: SDXL-class needs about 8-12 GB in fp16; FLUX-class models are larger (use quantization/offload); see Efficient & Edge Vision.
Key takeaways
Section titled “Key takeaways”- DDPM trains a network to predict added noise at random noise levels; sampling reverses the chain. Predicting noise equals estimating the score up to scale.
- The SDE/ODE view gives deterministic samplers and fast solvers; distillation (consistency, adversarial) reaches 1-4 steps.
- Latent diffusion runs the denoiser in an autoencoder’s compressed space; text enters through cross-attention or joint attention.
- Classifier-free guidance extrapolates between unconditional and conditional predictions, trading diversity for fidelity.
- DiT/MMDiT transformers with flow-matching/rectified-flow objectives are the current frontier recipe; LoRA and ControlNet-style adapters customize them cheaply.
Exercises
Section titled “Exercises”- Conceptual. Derive from the one-step transitions for .
- Conceptual. Why does high CFG reduce diversity? Explain in terms of the guided score.
- Hands-on. Train the DDPM loop above on MNIST with a small U-Net (use
TinyUNetfrom the segmentation lecture plus a timestep embedding); sample 16 digits with 1000 steps and with 50 DDIM steps. - Hands-on. Swap the training loss for the flow-matching loss and sample with 10, 20, 50 Euler steps. Compare quality against DDPM at equal step counts.
- Conceptual. What does zero-initialization buy ControlNet and adaLN-Zero?
Solutions (sketch)
- , with . Sum of independent Gaussians: variance , mean coefficient .
- The conditional term is amplified by ; samples are pushed toward high-density modes of the implicit classifier, narrowing the distribution.
- DDPM at 50 steps degrades; DDIM at 50 is close to 1000-step DDPM quality.
- Straight paths usually give clearly better samples at 10-20 steps than ancestral DDPM sampling.
- Training starts from the exact pretrained behavior (no initial perturbation), so the added module cannot damage quality early and gradients still flow to it.
Further reading
Section titled “Further reading”- Sohl-Dickstein et al., “Deep Unsupervised Learning using Nonequilibrium Thermodynamics” (2015).
- Ho, Jain, Abbeel, “Denoising Diffusion Probabilistic Models” (2020); Nichol and Dhariwal, “Improved DDPM” (2021).
- Song et al., “Score-Based Generative Modeling through Stochastic Differential Equations” (2021); Song, Meng, Ermon, “DDIM” (2020).
- Karras et al., “Elucidating the Design Space of Diffusion-Based Generative Models” (2022).
- Rombach et al., “High-Resolution Image Synthesis with Latent Diffusion Models” (2022).
- Ho and Salimans, “Classifier-Free Diffusion Guidance” (2022).
- Peebles and Xie, “Scalable Diffusion Models with Transformers” (2022).
- Lipman et al., “Flow Matching for Generative Modeling” (2022); Liu et al., “Flow Straight and Fast” (2022); Esser et al., “Scaling Rectified Flow Transformers for High-Resolution Image Synthesis” (SD3, 2024).
- Zhang et al., “Adding Conditional Control to Text-to-Image Diffusion Models” (ControlNet, 2023); Hu et al., “LoRA” (2021); Ruiz et al., “DreamBooth” (2022).
- Song et al., “Consistency Models” (2023). Lilian Weng’s blog post “What are diffusion models?” and the Hugging Face diffusers course.