Skip to content

Diffusion & Image Generation

What you’ll learn. The forward/reverse diffusion process and the DDPM training objective with its derivation sketch; the score-matching view and the SDE/ODE formulation; samplers (DDIM, DPM-Solver, few-step distillation); latent diffusion (Stable Diffusion); classifier-free guidance; diffusion transformers (DiT) and modern text-to-image stacks; flow matching and rectified flow (the basis of recent frontier open models); and adapters (LoRA, ControlNet, IP-Adapter). Code is minimal PyTorch. Prerequisites: Segmentation (U-Net), Vision Transformers, and basic probability.

We want to sample from an unknown data distribution pdata(x)p_\text{data}(x) (images) given examples. Earlier approaches: GANs (adversarial, unstable, mode dropping), VAEs (blurry), autoregressive models over pixels or tokens (slow, but see VQ-token models). Diffusion models turn generation into many easy denoising steps: stable training with a simple regression loss, strong mode coverage, at the cost of iterative sampling.

Define a Markov chain that gradually adds Gaussian noise over TT steps (typically T=1000T=1000) with a variance schedule βt\beta_t:

q(xt∣xt−1)=N(xt; 1−βt xt−1, βtI).q(x_t\mid x_{t-1}) = \mathcal{N}\big(x_t;\ \sqrt{1-\beta_t}\,x_{t-1},\ \beta_t I\big).

With αt=1−βt\alpha_t = 1-\beta_t and αˉt=∏s≤tαs\bar\alpha_t = \prod_{s\le t}\alpha_s, the marginal has a closed form:

q(xt∣x0)=N(αˉt x0, (1−αˉt)I)  ⟺  xt=αˉt x0+1−αˉt ϵ,ϵ∼N(0,I).q(x_t\mid x_0) = \mathcal{N}\big(\sqrt{\bar\alpha_t}\,x_0,\ (1-\bar\alpha_t) I\big) \iff x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1-\bar\alpha_t}\,\epsilon,\quad \epsilon\sim\mathcal{N}(0,I).

As t→Tt\to T, αˉT≈0\bar\alpha_T\approx0 and xT≈N(0,I)x_T\approx\mathcal{N}(0,I): pure noise. Schedules: linear β\beta (DDPM), cosine αˉt\bar\alpha_t (improved DDPM). The schedule matters: a schedule whose final step still leaks signal leads to wrong brightness (a known issue fixed by “zero terminal SNR” schedules).

The true reverse q(xt−1∣xt)q(x_{t-1}\mid x_t) is intractable, but conditional on x0x_0 it is Gaussian: q(xt−1∣xt,x0)=N(μ~t(xt,x0),β~tI)q(x_{t-1}\mid x_t,x_0)=\mathcal{N}(\tilde\mu_t(x_t,x_0),\tilde\beta_t I) with

μ~t=1αt(xt−βt1−αˉt ϵ).\tilde\mu_t = \frac{1}{\sqrt{\alpha_t}}\left(x_t - \frac{\beta_t}{\sqrt{1-\bar\alpha_t}}\,\epsilon\right).

We parameterize pθ(xt−1∣xt)=N(μθ(xt,t),σt2I)p_\theta(x_{t-1}\mid x_t)=\mathcal{N}(\mu_\theta(x_t,t),\sigma_t^2I) and maximize the variational lower bound (ELBO). The bound decomposes into KL terms between Gaussians, each reducing to a squared error between means. Reparametrizing the mean through a noise-prediction network ϵθ\epsilon_\theta and dropping weights as Ho et al. did gives the simple loss:

Lsimple=Et∼U{1..T}, x0, ϵ[∥ϵ−ϵθ(αˉtx0+1−αˉt ϵ, t)∥2].\mathcal{L}_\text{simple}=\mathbb{E}_{t\sim U\{1..T\},\,x_0,\,\epsilon}\Big[\big\|\epsilon-\epsilon_\theta\big(\sqrt{\bar\alpha_t}x_0+\sqrt{1-\bar\alpha_t}\,\epsilon,\ t\big)\big\|^2\Big].

Intuition: train a network to guess the noise that was added; at each noise level this is equivalent to estimating the clean image.

import torch, torch.nn.functional as F
T = 1000
betas = torch.linspace(1e-4, 0.02, T)
alphas = 1 - betas
abar = torch.cumprod(alphas, 0)
def train_step(model, x0): # x0: (B,C,H,W) scaled to [-1,1]
B = x0.size(0)
t = torch.randint(0, T, (B,), device=x0.device)
eps = torch.randn_like(x0)
ab = abar.to(x0.device)[t].view(B, 1, 1, 1)
xt = ab.sqrt() * x0 + (1 - ab).sqrt() * eps
return F.mse_loss(model(xt, t), eps)
@torch.no_grad()
def ddpm_sample(model, shape, device):
x = torch.randn(shape, device=device)
for t in reversed(range(T)):
tt = torch.full((shape[0],), t, device=device, dtype=torch.long)
eps = model(x, tt)
a, ab, b = alphas[t], abar[t], betas[t]
mean = (x - b / (1 - ab).sqrt() * eps) / a.sqrt()
x = mean + (b.sqrt() * torch.randn_like(x) if t > 0 else 0)
return x

The network (originally a U-Net with ResNet blocks, self-attention at low resolution, timestep embeddings injected via FiLM/adaptive normalization) takes (xt,t)(x_t,t) and predicts ϵ\epsilon. Alternative parameterizations: predict x0x_0, or vv-prediction (v=αˉtϵ−1−αˉtx0v=\sqrt{\bar\alpha_t}\epsilon-\sqrt{1-\bar\alpha_t}x_0), which is better conditioned at high noise levels and used by many modern models.

The score is ∇xlog⁡p(x)\nabla_x\log p(x). Denoising score matching shows that for xt=x0+σϵx_t=x_0+\sigma\epsilon, the optimal denoiser gives the score of the noised distribution: ∇xtlog⁡pt(xt)=−ϵ∗σt\nabla_{x_t}\log p_t(x_t) = -\frac{\epsilon^*}{\sigma_t} (for the variance-preserving case, −ϵ∗/1−αˉt-\epsilon^*/\sqrt{1-\bar\alpha_t}). So noise prediction is score estimation up to scaling. Song et al. unified DDPM and score-based models via SDEs: the forward noising is

dx=f(x,t) dt+g(t) dw,dx = f(x,t)\,dt + g(t)\,dw,

and the reverse-time SDE (Anderson, 1982) is

dx=[f(x,t)−g(t)2∇xlog⁡pt(x)]dt+g(t) dwˉ.dx = \big[f(x,t)-g(t)^2\nabla_x\log p_t(x)\big]dt + g(t)\,d\bar w.

Replace the score by the learned network. A key corollary is the probability flow ODE, with the same marginals ptp_t but no noise:

dx=[f(x,t)−12g(t)2∇xlog⁡pt(x)]dt.dx = \Big[f(x,t)-\tfrac12 g(t)^2\nabla_x\log p_t(x)\Big]dt.

Sampling then is solving an ODE, which allows deterministic sampling, invertibility (latent encoding) and fast numerical solvers.

  • DDIM: deterministic non-Markovian sampler consistent with the DDPM-trained model, 20-50 steps instead of 1000.
  • DPM-Solver / DPM-Solver++, UniPC, Heun/EDM samplers (Karras et al., “Elucidating the Design Space”, EDM): higher-order ODE solvers reaching good quality in about 10-25 steps. Karras et al. also clarified preconditioning and noise-level parametrization.
  • Distillation to few steps: progressive distillation, consistency models (map any point on the ODE trajectory to its origin, 1-4 steps), Latent Consistency Models, adversarial diffusion distillation (SDXL-Turbo), DMD (distribution matching). Quality/diversity trade-offs; useful for real-time applications.

Running a U-Net at 5122×3512^2\times3 pixels is expensive. Latent Diffusion Models (Rombach et al., 2022; Stable Diffusion) first train an autoencoder (perceptual + adversarial losses, small KL regularization) to compress images by a factor f=8f=8 per side (e.g., 5122×3→642×4512^2\times3\to64^2\times4, 48x fewer elements), then train diffusion in latent space. Decoder returns pixels. The autoencoder handles imperceptible detail; diffusion handles semantics and layout. Modern models use 16 channels or more per latent (SD3, FLUX) for better reconstruction.

Text conditioning: encode the prompt with a text encoder (CLIP; later T5/other LLM encoders, often several) and inject via cross-attention from image tokens to text tokens in the denoiser.

prompt --> text encoder(s) --> text embeddings ---------------+
v
noise z_T ~ N(0,I) (64x64x4) --> [denoiser x ~30 steps, cross-attn] --> z_0 --> VAE decoder --> image

We want samples that match the condition cc (a prompt) more strongly than the conditional model alone provides. Ho and Salimans: train a single network with the condition randomly dropped (e.g., 10% of the time replaced by a null token), obtaining both ϵθ(xt,c)\epsilon_\theta(x_t,c) and ϵθ(xt,∅)\epsilon_\theta(x_t,\varnothing). At sampling:

ϵ~=ϵθ(xt,∅)+w (ϵθ(xt,c)−ϵθ(xt,∅)).\tilde\epsilon = \epsilon_\theta(x_t,\varnothing) + w\,\big(\epsilon_\theta(x_t,c)-\epsilon_\theta(x_t,\varnothing)\big).

Since ϵ∝−∇log⁡p\epsilon\propto-\nabla\log p, this follows the gradient of log⁡p(x∣c)+(w−1)log⁡p(x∣c)p(x)\log p(x\mid c)+(w-1)\log \frac{p(x\mid c)}{p(x)}, i.e., it sharpens the implicit classifier. Guidance scale ww of about 5-8 for Stable Diffusion-like models (guidance-distilled models bake it in). Higher ww: better prompt adherence and fidelity but less diversity and over-saturation. Cost: two network evaluations per step (batched), unless distilled. Negative prompts replace ∅\varnothing with an unwanted-concept embedding.

Diffusion Transformers (DiT) and modern architectures

Section titled “Diffusion Transformers (DiT) and modern architectures”

DiT (Peebles and Xie, 2022) replaces the U-Net with a standard ViT operating on latent patches (Vision Transformers). Timestep and class/text conditioning are injected with adaLN-Zero: LayerNorm scale/shift (and a residual gate initialized to zero) regressed from the conditioning vector. DiTs scale predictably: lower FID with more Gflops, in the usual transformer scaling-law fashion, and benefit from decades of transformer optimization.

Successors: MMDiT (Stable Diffusion 3), in which text tokens and image tokens have separate weights but attend jointly in a single sequence; FLUX-family models (hybrid double- and single-stream blocks); PixArt, HunyuanDiT, Qwen-Image and others. Video generation (Sora-style and open video models) uses space-time latent patches and DiT backbones (Video, 3D & Spatial Vision). I describe lineage rather than rankings, because leaderboards and releases turn over every few months; verify current options.

Flow matching (Lipman et al., 2022) learns a velocity field vθ(x,t)v_\theta(x,t) whose ODE dxdt=vθ(x,t)\frac{dx}{dt}=v_\theta(x,t) transports noise to data. In the simplest (rectified flow; Liu et al.; Albergo and Vanden-Eijnden) formulation, interpolate linearly between a data sample x0x_0 and noise x1∼N(0,I)x_1\sim\mathcal{N}(0,I):

xt=(1−t) x0+t x1,target velocity ut=x1−x0,x_t=(1-t)\,x_0+t\,x_1,\qquad \text{target velocity } u_t = x_1-x_0,

and train

LFM=Et,x0,x1∥vθ(xt,t)−(x1−x0)∥2.\mathcal{L}_\text{FM}=\mathbb{E}_{t,x_0,x_1}\big\|v_\theta(x_t,t)-(x_1-x_0)\big\|^2.

Sampling integrates the ODE from t=1t=1 (noise) to t=0t=0 (data), with Euler or higher-order steps. Why it caught on:

  • A simple regression target with straight-line paths, which are friendlier to coarse ODE solvers (fewer steps) than curved diffusion paths. “Reflow” iteratively straightens paths.
  • Conceptually simpler than SDE machinery; diffusion is a special case with a different path/weighting.
  • SD3 combined rectified flow with logit-normal timestep sampling (focusing training on mid noise levels) and MMDiT, and FLUX followed in the same family, so most frontier open image models since 2024 use flow-matching objectives.
def fm_loss(model, x0): # x0 are latents
B = x0.size(0)
t = torch.rand(B, device=x0.device).view(B, 1, 1, 1)
x1 = torch.randn_like(x0)
xt = (1 - t) * x0 + t * x1
return F.mse_loss(model(xt, t.flatten()), x1 - x0)
@torch.no_grad()
def euler_sample(model, shape, device, steps=28):
x = torch.randn(shape, device=device)
ts = torch.linspace(1, 0, steps + 1, device=device)
for i in range(steps):
dt = ts[i + 1] - ts[i] # negative
x = x + model(x, ts[i].expand(shape[0])) * dt
return x

Equivalence note: with Gaussian paths, flow matching and diffusion parameterizations are related by reparametrization of the noise schedule and the target (velocity vs noise vs score); the practical differences are the path, weighting, and sampler conventions.

Fine-tuning a multi-billion-parameter generator is expensive; parameter-efficient methods dominate:

  • LoRA (Hu et al., 2021, applied to diffusion): freeze weight WW and learn a low-rank update W+αrBAW+\tfrac{\alpha}{r}BA with B∈Rd×r,A∈Rr×kB\in\mathbb{R}^{d\times r},A\in\mathbb{R}^{r\times k} (rr of 4-64). Small files (10-200 MB), mixable. Used for styles, characters, concepts. DreamBooth fine-tunes on 3-20 images of a subject with a rare-token identifier plus a prior-preservation loss; Textual Inversion learns only a new token embedding.
  • ControlNet (Zhang et al., 2023): clone the encoder half of the denoiser, connect it through zero-initialized convolutions, and feed spatial conditions (Canny edges, depth, pose, segmentation, scribbles). The original weights stay frozen so quality is preserved; zero-init means training starts as an identity modification. T2I-Adapter is a lighter alternative.
  • IP-Adapter: decoupled cross-attention for image prompts (style/identity from a reference image).
  • Instruction-based editing models (InstructPix2Pix and in-context editing models built on large DiTs) take an input image plus a text instruction.
# diffusers: text-to-image with a LoRA (model names change; check the Hub)
import torch
from diffusers import DiffusionPipeline
pipe = DiffusionPipeline.from_pretrained("stabilityai/stable-diffusion-xl-base-1.0",
torch_dtype=torch.float16).to("cuda")
# pipe.load_lora_weights("path/to/lora")
img = pipe("a watercolor fox in a forest", num_inference_steps=30, guidance_scale=6.0).images[0]
img.save("fox.png")
  • FID (Fréchet Inception Distance) compares Gaussian fits of Inception features of real versus generated sets; sensitive to sample size and feature extractor, and does not capture prompt alignment. CLIP score and newer text-image alignment metrics (GenEval, T2I-CompBench, human preference models like ImageReward/PickScore) measure adherence; human evaluation is still the arbiter. Precision/recall for generative models separates fidelity from diversity.
  • Failure modes: text rendering (improving with T5/LLM encoders and large DiTs), counting and attribute binding, hands/anatomy, over-saturation at high CFG, memorization of training images.
  • Safety and provenance: watermarking, content filters, copyright and consent debates; deepfake risk. Apply organizational policy; this is not only a technical issue.
  • Memory: SDXL-class needs about 8-12 GB in fp16; FLUX-class models are larger (use quantization/offload); see Efficient & Edge Vision.
  • DDPM trains a network to predict added noise at random noise levels; sampling reverses the chain. Predicting noise equals estimating the score ∇log⁡pt\nabla\log p_t up to scale.
  • The SDE/ODE view gives deterministic samplers and fast solvers; distillation (consistency, adversarial) reaches 1-4 steps.
  • Latent diffusion runs the denoiser in an autoencoder’s compressed space; text enters through cross-attention or joint attention.
  • Classifier-free guidance extrapolates between unconditional and conditional predictions, trading diversity for fidelity.
  • DiT/MMDiT transformers with flow-matching/rectified-flow objectives are the current frontier recipe; LoRA and ControlNet-style adapters customize them cheaply.
  1. Conceptual. Derive q(xt∣x0)q(x_t\mid x_0) from the one-step transitions for t=2t=2.
  2. Conceptual. Why does high CFG reduce diversity? Explain in terms of the guided score.
  3. Hands-on. Train the DDPM loop above on MNIST with a small U-Net (use TinyUNet from the segmentation lecture plus a timestep embedding); sample 16 digits with 1000 steps and with 50 DDIM steps.
  4. Hands-on. Swap the training loss for the flow-matching loss and sample with 10, 20, 50 Euler steps. Compare quality against DDPM at equal step counts.
  5. Conceptual. What does zero-initialization buy ControlNet and adaLN-Zero?
Solutions (sketch)
  1. x2=α2x1+1−α2ϵ2x_2=\sqrt{\alpha_2}x_1+\sqrt{1-\alpha_2}\epsilon_2, with x1=α1x0+1−α1ϵ1x_1=\sqrt{\alpha_1}x_0+\sqrt{1-\alpha_1}\epsilon_1. Sum of independent Gaussians: variance α2(1−α1)+1−α2=1−α1α2\alpha_2(1-\alpha_1)+1-\alpha_2=1-\alpha_1\alpha_2, mean coefficient α1α2\sqrt{\alpha_1\alpha_2}.
  2. The conditional term is amplified by ww; samples are pushed toward high-density modes of the implicit classifier, narrowing the distribution.
  3. DDPM at 50 steps degrades; DDIM at 50 is close to 1000-step DDPM quality.
  4. Straight paths usually give clearly better samples at 10-20 steps than ancestral DDPM sampling.
  5. Training starts from the exact pretrained behavior (no initial perturbation), so the added module cannot damage quality early and gradients still flow to it.
  • Sohl-Dickstein et al., “Deep Unsupervised Learning using Nonequilibrium Thermodynamics” (2015).
  • Ho, Jain, Abbeel, “Denoising Diffusion Probabilistic Models” (2020); Nichol and Dhariwal, “Improved DDPM” (2021).
  • Song et al., “Score-Based Generative Modeling through Stochastic Differential Equations” (2021); Song, Meng, Ermon, “DDIM” (2020).
  • Karras et al., “Elucidating the Design Space of Diffusion-Based Generative Models” (2022).
  • Rombach et al., “High-Resolution Image Synthesis with Latent Diffusion Models” (2022).
  • Ho and Salimans, “Classifier-Free Diffusion Guidance” (2022).
  • Peebles and Xie, “Scalable Diffusion Models with Transformers” (2022).
  • Lipman et al., “Flow Matching for Generative Modeling” (2022); Liu et al., “Flow Straight and Fast” (2022); Esser et al., “Scaling Rectified Flow Transformers for High-Resolution Image Synthesis” (SD3, 2024).
  • Zhang et al., “Adding Conditional Control to Text-to-Image Diffusion Models” (ControlNet, 2023); Hu et al., “LoRA” (2021); Ruiz et al., “DreamBooth” (2022).
  • Song et al., “Consistency Models” (2023). Lilian Weng’s blog post “What are diffusion models?” and the Hugging Face diffusers course.