Skip to content

Generative Models

What you’ll learn. What it means to model a data distribution p(x)p(x); the main families (autoregressive, latent-variable/VAE, adversarial, diffusion/score-based, flows and flow matching, energy-based) and the trade-offs among them; the derivation of the VAE evidence lower bound; the GAN minimax game and its failure modes; the intuition and training objective of diffusion models; and the unifying likelihood and energy perspectives. Code sketches are in PyTorch.

A generative model learns a distribution pθ(x)p_\theta(x) close to the data distribution pdata(x)p_{data}(x) so that we can sample new data, evaluate likelihoods (sometimes), and learn representations. By Information Theory, maximum likelihood minimizes DKL(pdata∥pθ)D_{KL}(p_{data}\parallel p_\theta). The difficulty: high-dimensional p(x)p(x) (images with 10510^5 to 10610^6 dimensions) must be expressible, normalizable, and sampleable. Families differ in how they handle that.

Family Likelihood Sampling Notes
Autoregressive exact sequential, slow Chain rule; GPT-style LLMs, PixelCNN
VAE lower bound (ELBO) fast (one pass) Latent space; blurry-prone with simple decoders
GAN none fast Sharp samples; unstable training; mode collapse
Diffusion / score bound (or via ODE) iterative (10-1000 steps, fewer with distillation) State of the art for images, video, audio
Normalizing flows exact fast Invertible architectures limit expressivity
Flow matching / rectified flow via ODE few-step ODE Simple regression objective; used by recent large image/video models
Energy-based unnormalized MCMC, slow Flexible; hard training

Autoregressive models factor p(x)=∏ip(xi∣x<i)p(x)=\prod_ip(x_i\mid x_{<i}) and are trained with plain cross-entropy; transformers on text tokens are the dominant example. They also model images/audio by tokenizing them. The remaining families below are the non-autoregressive alternatives.

An autoencoder compresses xx into a low-dimensional code z=fϕ(x)z=f_\phi(x) and reconstructs x^=gθ(z)\hat x=g_\theta(z), minimizing ∥x−x^∥2\|x-\hat x\|^2. With linear layers and squared loss it recovers PCA; with nonlinear layers it learns curved manifolds. The bottleneck prevents identity copying. Variants: denoising AE (reconstruct clean xx from corrupted input; the seed of diffusion), sparse AE (also used in interpretability work on LLM activations), masked autoencoders (MAE reconstruct masked image patches, a self-supervised pretraining method).

A plain AE is not generative: the latent space has holes; sampling a random zz gives garbage. The VAE fixes this by imposing structure on zz.

Assume a latent-variable model: z∼p(z)=N(0,I)z\sim p(z)=\mathcal N(0,I), x∼pθ(x∣z)x\sim p_\theta(x\mid z). The marginal likelihood pθ(x)=∫pθ(x∣z)p(z)dzp_\theta(x)=\int p_\theta(x\mid z)p(z)dz is intractable. Introduce an encoder qϕ(z∣x)q_\phi(z\mid x) as an approximate posterior. Derivation of the evidence lower bound (ELBO):

log⁡pθ(x)=log⁡∫qϕ(z∣x)pθ(x,z)qϕ(z∣x)dz≥Eqϕ[log⁡pθ(x,z)qϕ(z∣x)](Jensen)=Eqϕ(z∣x)[log⁡pθ(x∣z)]⏟reconstruction−DKL(qϕ(z∣x)∥p(z))⏟regularizer.\begin{aligned} \log p_\theta(x)&=\log\int q_\phi(z\mid x)\frac{p_\theta(x,z)}{q_\phi(z\mid x)}dz\\ &\ge\mathbb{E}_{q_\phi}\Big[\log\frac{p_\theta(x,z)}{q_\phi(z\mid x)}\Big]\quad\text{(Jensen)}\\ &=\underbrace{\mathbb{E}_{q_\phi(z|x)}[\log p_\theta(x\mid z)]}_{\text{reconstruction}}-\underbrace{D_{KL}\big(q_\phi(z\mid x)\parallel p(z)\big)}_{\text{regularizer}}. \end{aligned}

The gap equals DKL(qϕ(z∣x)∥pθ(z∣x))≥0D_{KL}(q_\phi(z\mid x)\parallel p_\theta(z\mid x))\ge0, so maximizing the ELBO both improves the likelihood and tightens the posterior approximation. Interpretation: reconstruct well while keeping the code distribution close to the prior, which fills the latent space so sampling z∼N(0,I)z\sim\mathcal N(0,I) and decoding gives plausible data.

Reparameterization trick. Sampling z∼qϕz\sim q_\phi blocks gradients. With qϕ=N(μϕ(x),diag⁡σϕ2(x))q_\phi=\mathcal N(\mu_\phi(x),\operatorname{diag}\sigma_\phi^2(x)) write z=μ+σ⊙εz=\mu+\sigma\odot\varepsilon, ε∼N(0,I)\varepsilon\sim\mathcal N(0,I); the randomness is now an input and backprop works. The Gaussian KL has the closed form 12∑j(μj2+σj2−1−ln⁡σj2)\tfrac12\sum_j(\mu_j^2+\sigma_j^2-1-\ln\sigma_j^2).

import torch, torch.nn as nn, torch.nn.functional as F
class VAE(nn.Module):
def __init__(self, d_in=784, d_h=400, d_z=20):
super().__init__()
self.enc = nn.Sequential(nn.Linear(d_in, d_h), nn.ReLU())
self.mu, self.logvar = nn.Linear(d_h, d_z), nn.Linear(d_h, d_z)
self.dec = nn.Sequential(nn.Linear(d_z, d_h), nn.ReLU(), nn.Linear(d_h, d_in))
def forward(self, x):
h = self.enc(x); mu, logvar = self.mu(h), self.logvar(h)
z = mu + torch.randn_like(mu) * torch.exp(0.5 * logvar) # reparameterize
return self.dec(z), mu, logvar
def vae_loss(x, logits, mu, logvar, beta=1.0):
rec = F.binary_cross_entropy_with_logits(logits, x, reduction="sum") / x.size(0)
kl = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp()) / x.size(0)
return rec + beta * kl

Practical notes: a β>1\beta>1 weight (β\beta-VAE) encourages disentangled factors at the cost of reconstruction; posterior collapse (the decoder ignores zz, KL goes to 0) happens with powerful decoders, mitigated by KL annealing; Gaussian decoders with fixed variance yield blurry samples. VAEs remain central as tokenizers/compressors: latent diffusion models run diffusion in the latent space of a (VAE-style or VQ) autoencoder, and VQ-VAE discretizes latents into codes for autoregressive modeling.

Goodfellow et al. (2014): a generator G(z)G(z) maps noise to samples; a discriminator D(x)D(x) estimates the probability that xx is real. They play a minimax game:

min⁡Gmax⁡D Ex∼pdata[log⁡D(x)]+Ez∼p(z)[log⁡(1−D(G(z)))].\min_G\max_D\ \mathbb{E}_{x\sim p_{data}}[\log D(x)]+\mathbb{E}_{z\sim p(z)}[\log(1-D(G(z)))].

For a fixed GG the optimal discriminator is D⋆(x)=pdata(x)pdata(x)+pg(x)D^\star(x)=\frac{p_{data}(x)}{p_{data}(x)+p_g(x)}, and plugging it in shows GG minimizes 2 JS(pdata∥pg)−log⁡42\,\text{JS}(p_{data}\parallel p_g)-\log4. In practice use the non-saturating generator loss (maximize log⁡D(G(z))\log D(G(z))) to keep gradients when DD is confident.

# one GAN step (sketch)
def gan_step(G, D, optG, optD, real, z_dim):
B = real.size(0); z = torch.randn(B, z_dim, device=real.device)
fake = G(z)
# discriminator: real -> 1, fake -> 0
d_loss = F.binary_cross_entropy_with_logits(D(real), torch.ones(B, 1, device=real.device)) + \
F.binary_cross_entropy_with_logits(D(fake.detach()), torch.zeros(B, 1, device=real.device))
optD.zero_grad(); d_loss.backward(); optD.step()
# generator: fool D
g_loss = F.binary_cross_entropy_with_logits(D(fake), torch.ones(B, 1, device=real.device))
optG.zero_grad(); g_loss.backward(); optG.step()

Problems: non-convergent dynamics (simultaneous gradient play can cycle), mode collapse (the generator covers few modes), vanishing gradients when supports do not overlap, and no likelihood for evaluation (FID and related sample-based metrics are used instead). Fixes: WGAN and gradient penalty (Earth-mover distance), spectral normalization, two-timescale updates, architectural advances (DCGAN, progressive growing, StyleGAN, BigGAN). GANs gave the sharpest images of the late 2010s, but diffusion models have largely replaced them for high-fidelity, diverse generation thanks to stable training and better coverage; GANs persist where one-step speed matters (real-time super-resolution, some distillation targets, adversarial losses as add-ons).

Idea. Define a fixed forward process that gradually destroys data with noise, and learn to reverse it. With a variance schedule βt\beta_t and αˉt=∏s≤t(1−βs)\bar\alpha_t=\prod_{s\le t}(1-\beta_s), the noisy sample at step tt has a closed form:

xt=αˉt x0+1−αˉt ε,ε∼N(0,I).x_t=\sqrt{\bar\alpha_t}\,x_0+\sqrt{1-\bar\alpha_t}\,\varepsilon,\qquad\varepsilon\sim\mathcal N(0,I).

At t=Tt=T the signal is essentially pure Gaussian noise. The reverse process pθ(xt−1∣xt)p_\theta(x_{t-1}\mid x_t) is also approximately Gaussian for small steps; the network just needs to predict its mean, which is equivalent to predicting the added noise. The DDPM (Ho et al., 2020) training objective simplifies the ELBO to

L=Ex0, t, ε[∥ε−εθ(xt,t)∥2].L=\mathbb{E}_{x_0,\,t,\,\varepsilon}\Big[\big\|\varepsilon-\varepsilon_\theta(x_t,t)\big\|^2\Big].

It is a plain regression: pick a random tt, noise the data, predict the noise. Training is stable; no adversary.

def diffusion_loss(model, x0, alphas_cumprod):
B = x0.size(0)
t = torch.randint(0, len(alphas_cumprod), (B,), device=x0.device)
a = alphas_cumprod[t].view(B, *([1] * (x0.dim() - 1)))
eps = torch.randn_like(x0)
xt = a.sqrt() * x0 + (1 - a).sqrt() * eps
return F.mse_loss(model(xt, t), eps)

Score view (Song and Ermon; Song et al., 2021). The score ∇xlog⁡pt(x)\nabla_x\log p_t(x) points toward higher density. Noise prediction is equivalent to score estimation: εθ≈−1−αˉt ∇xlog⁡pt(xt)\varepsilon_\theta\approx-\sqrt{1-\bar\alpha_t}\,\nabla_x\log p_t(x_t) (denoising score matching; Tweedie’s formula links the optimal denoiser to the score). Sampling then follows the reverse-time SDE, or the deterministic probability-flow ODE with the same marginals, solved by fast ODE solvers (DDIM, DPM-Solver) in tens of steps; distillation and consistency models reduce this to a few or one step.

Guidance. Classifier-free guidance trains the model with and without a condition cc (randomly dropping it) and samples with ε~=εθ(xt,∅)+w [εθ(xt,c)−εθ(xt,∅)]\tilde\varepsilon=\varepsilon_\theta(x_t,\varnothing)+w\,[\varepsilon_\theta(x_t,c)-\varepsilon_\theta(x_t,\varnothing)], w>1w>1, trading diversity for prompt fidelity. Latent diffusion (Stable Diffusion) runs diffusion in an autoencoder’s latent space, reducing compute by orders of magnitude. Backbones moved from U-Nets to diffusion transformers (DiT). Text conditioning uses cross-attention to text-encoder embeddings (Attention & Transformers).

Normalizing flows build x=fθ(z)x=f_\theta(z) from invertible layers with tractable Jacobians, so log⁡p(x)=log⁡p(z)+log⁡∣det⁡∂z/∂x∣\log p(x)=\log p(z)+\log|\det\partial z/\partial x| exactly; architectural constraints (coupling layers, autoregressive flows) limit flexibility.

Continuous normalizing flows / flow matching (Lipman et al., 2022; Liu et al., 2022 “rectified flow”; Albergo and Vanden-Eijnden) define a velocity field vθ(x,t)v_\theta(x,t) transporting noise to data along an ODE x˙=vθ(x,t)\dot x=v_\theta(x,t). The training objective is again regression: with a simple path xt=(1−t)x0+t x1x_t=(1-t)x_0+t\,x_1 (x0x_0 noise, x1x_1 data), the conditional target velocity is x1−x0x_1-x_0:

L=Et,x0,x1∥vθ(xt,t)−(x1−x0)∥2.L=\mathbb{E}_{t,x_0,x_1}\big\|v_\theta(x_t,t)-(x_1-x_0)\big\|^2.

This is simulation-free, simple, and closely related to diffusion (diffusion’s probability-flow ODE is a special case of a flow with a particular path and weighting). Straighter paths allow fewer sampling steps. Recent large image and video generators have adopted flow-matching or rectified-flow objectives with transformer backbones. The distinction between “diffusion” and “flow matching” has blurred; both learn a time-dependent vector field.

An energy-based model defines pθ(x)=e−Eθ(x)/Zθp_\theta(x)=e^{-E_\theta(x)}/Z_\theta with an unnormalized energy and intractable partition function Zθ=∫e−EθZ_\theta=\int e^{-E_\theta}. The maximum-likelihood gradient is

∇θEpdata[log⁡pθ]=−Epdata[∇θEθ]+Epθ[∇θEθ],\nabla_\theta\mathbb{E}_{p_{data}}[\log p_\theta]=-\mathbb{E}_{p_{data}}[\nabla_\theta E_\theta]+\mathbb{E}_{p_\theta}[\nabla_\theta E_\theta],

“lower energy on data, raise energy on the model’s own samples” (the negative phase needs MCMC, e.g. Langevin dynamics x←x−η2∇xE+η ξx\leftarrow x-\tfrac\eta2\nabla_xE+\sqrt\eta\,\xi). Training is delicate. The score ∇xlog⁡p=−∇xE\nabla_x\log p=-\nabla_xE does not involve ZZ, the key insight behind score matching and thus diffusion: learn the score, not the density, and sample with Langevin-style dynamics.

Unifying summary

  • Autoregressive, flows: exact likelihood by construction.
  • VAEs, diffusion: optimize an ELBO (diffusion is a hierarchical VAE with a fixed encoder).
  • GANs: implicit models; training minimizes a divergence estimated by a learned critic.
  • Score/diffusion/flow matching: learn vector fields (gradients or velocities) and integrate.
  • Energy-based: model log-density up to a constant.

Evaluation: log-likelihood (bits per dimension) where available; FID/KID, precision-recall of samples, CLIP score for text alignment, and human preference; none is perfect, and likelihood does not always track sample quality.

  • Text, code, structured tokens: autoregressive transformers.
  • High-quality images, video, audio: latent diffusion/flow transformers; use pretrained models and fine-tune (LoRA) or condition (ControlNet/adapters).
  • Need a compact learned representation or tokenizer: VAE/VQ-VAE.
  • Real-time single-step synthesis: distilled diffusion, consistency models, or GANs.
  • Always check for memorization and safety/consent issues with training data; generative models can reproduce training examples.
  • A generative model estimates p(x)p(x); likelihood training equals minimizing forward KL.
  • The VAE maximizes the ELBO (reconstruction minus KL to the prior) using the reparameterization trick.
  • GANs play a minimax game; powerful but unstable, with mode collapse and no likelihood.
  • Diffusion trains by predicting noise at random noise levels; equivalently learns the score; sampling reverses the noising iteratively (or via an ODE).
  • Flow matching regresses a velocity field along simple paths; it is simple, fast, and now widely used.
  • The score of an energy-based model avoids the partition function, which is why score-based methods scale.
  1. Derivation. Show that the ELBO gap is DKL(qϕ(z∣x)∥pθ(z∣x))D_{KL}(q_\phi(z|x)\parallel p_\theta(z|x)).
  2. Derivation. Show the optimal GAN discriminator is pdata/(pdata+pg)p_{data}/(p_{data}+p_g).
  3. Hands-on. Train the VAE above on MNIST. Plot reconstructions, samples from N(0,I)\mathcal N(0,I), and interpolation between two encoded digits. Vary β\beta.
  4. Hands-on. Train a tiny 2D diffusion model on a “two moons” dataset with an MLP conditioned on tt; visualize samples at various step counts. Then train a flow-matching model on the same data and compare.
  5. Conceptual. Why does mode-covering (forward KL) training tend to blur samples while GANs produce sharp but less diverse ones?
Solutions (1, 2)
  1. log⁡p(x)−ELBO=Eq[log⁡p(x)−log⁡p(x,z)q(z∣x)]=Eq[log⁡q(z∣x)p(z∣x)]=DKL(q∥p(z∣x))\log p(x)-\text{ELBO}=\mathbb{E}_q[\log p(x)-\log\tfrac{p(x,z)}{q(z|x)}]=\mathbb{E}_q[\log\tfrac{q(z|x)}{p(z|x)}]=D_{KL}(q\parallel p(z|x)).
  2. Maximizing ∫[pdatalog⁡D+pglog⁡(1−D)]dx\int[p_{data}\log D+p_g\log(1-D)]dx pointwise: derivative pdata/D−pg/(1−D)=0p_{data}/D-p_g/(1-D)=0 gives D=pdata/(pdata+pg)D=p_{data}/(p_{data}+p_g).
  • Kingma and Welling, “Auto-Encoding Variational Bayes” (2013); Kingma and Welling, “An Introduction to Variational Autoencoders” (2019).
  • Goodfellow et al., “Generative Adversarial Nets” (2014); Arjovsky et al., “Wasserstein GAN” (2017); Karras et al., “A Style-Based Generator Architecture for GANs” (2018).
  • Ho, Jain, Abbeel, “Denoising Diffusion Probabilistic Models” (2020).
  • Song et al., “Score-Based Generative Modeling through Stochastic Differential Equations” (2021).
  • Rombach et al., “High-Resolution Image Synthesis with Latent Diffusion Models” (2022); Peebles and Xie, “Scalable Diffusion Models with Transformers” (2022).
  • Lipman et al., “Flow Matching for Generative Modeling” (2022); Liu et al., “Flow Straight and Fast: Rectified Flow” (2022).
  • Ho and Salimans, “Classifier-Free Diffusion Guidance” (2022).
  • Lilian Weng, “What are Diffusion Models?” (blog, 2021).
  • LeCun et al., “A Tutorial on Energy-Based Learning” (2006).