Generative Models
What you’ll learn. What it means to model a data distribution ; the main families (autoregressive, latent-variable/VAE, adversarial, diffusion/score-based, flows and flow matching, energy-based) and the trade-offs among them; the derivation of the VAE evidence lower bound; the GAN minimax game and its failure modes; the intuition and training objective of diffusion models; and the unifying likelihood and energy perspectives. Code sketches are in PyTorch.
The goal
Section titled “The goal”A generative model learns a distribution close to the data distribution so that we can sample new data, evaluate likelihoods (sometimes), and learn representations. By Information Theory, maximum likelihood minimizes . The difficulty: high-dimensional (images with to dimensions) must be expressible, normalizable, and sampleable. Families differ in how they handle that.
| Family | Likelihood | Sampling | Notes |
|---|---|---|---|
| Autoregressive | exact | sequential, slow | Chain rule; GPT-style LLMs, PixelCNN |
| VAE | lower bound (ELBO) | fast (one pass) | Latent space; blurry-prone with simple decoders |
| GAN | none | fast | Sharp samples; unstable training; mode collapse |
| Diffusion / score | bound (or via ODE) | iterative (10-1000 steps, fewer with distillation) | State of the art for images, video, audio |
| Normalizing flows | exact | fast | Invertible architectures limit expressivity |
| Flow matching / rectified flow | via ODE | few-step ODE | Simple regression objective; used by recent large image/video models |
| Energy-based | unnormalized | MCMC, slow | Flexible; hard training |
Autoregressive models factor and are trained with plain cross-entropy; transformers on text tokens are the dominant example. They also model images/audio by tokenizing them. The remaining families below are the non-autoregressive alternatives.
Autoencoders
Section titled “Autoencoders”An autoencoder compresses into a low-dimensional code and reconstructs , minimizing . With linear layers and squared loss it recovers PCA; with nonlinear layers it learns curved manifolds. The bottleneck prevents identity copying. Variants: denoising AE (reconstruct clean from corrupted input; the seed of diffusion), sparse AE (also used in interpretability work on LLM activations), masked autoencoders (MAE reconstruct masked image patches, a self-supervised pretraining method).
A plain AE is not generative: the latent space has holes; sampling a random gives garbage. The VAE fixes this by imposing structure on .
Variational autoencoders
Section titled “Variational autoencoders”Assume a latent-variable model: , . The marginal likelihood is intractable. Introduce an encoder as an approximate posterior. Derivation of the evidence lower bound (ELBO):
The gap equals , so maximizing the ELBO both improves the likelihood and tightens the posterior approximation. Interpretation: reconstruct well while keeping the code distribution close to the prior, which fills the latent space so sampling and decoding gives plausible data.
Reparameterization trick. Sampling blocks gradients. With write , ; the randomness is now an input and backprop works. The Gaussian KL has the closed form .
import torch, torch.nn as nn, torch.nn.functional as F
class VAE(nn.Module): def __init__(self, d_in=784, d_h=400, d_z=20): super().__init__() self.enc = nn.Sequential(nn.Linear(d_in, d_h), nn.ReLU()) self.mu, self.logvar = nn.Linear(d_h, d_z), nn.Linear(d_h, d_z) self.dec = nn.Sequential(nn.Linear(d_z, d_h), nn.ReLU(), nn.Linear(d_h, d_in)) def forward(self, x): h = self.enc(x); mu, logvar = self.mu(h), self.logvar(h) z = mu + torch.randn_like(mu) * torch.exp(0.5 * logvar) # reparameterize return self.dec(z), mu, logvar
def vae_loss(x, logits, mu, logvar, beta=1.0): rec = F.binary_cross_entropy_with_logits(logits, x, reduction="sum") / x.size(0) kl = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp()) / x.size(0) return rec + beta * klPractical notes: a weight (-VAE) encourages disentangled factors at the cost of reconstruction; posterior collapse (the decoder ignores , KL goes to 0) happens with powerful decoders, mitigated by KL annealing; Gaussian decoders with fixed variance yield blurry samples. VAEs remain central as tokenizers/compressors: latent diffusion models run diffusion in the latent space of a (VAE-style or VQ) autoencoder, and VQ-VAE discretizes latents into codes for autoregressive modeling.
Generative adversarial networks
Section titled “Generative adversarial networks”Goodfellow et al. (2014): a generator maps noise to samples; a discriminator estimates the probability that is real. They play a minimax game:
For a fixed the optimal discriminator is , and plugging it in shows minimizes . In practice use the non-saturating generator loss (maximize ) to keep gradients when is confident.
# one GAN step (sketch)def gan_step(G, D, optG, optD, real, z_dim): B = real.size(0); z = torch.randn(B, z_dim, device=real.device) fake = G(z) # discriminator: real -> 1, fake -> 0 d_loss = F.binary_cross_entropy_with_logits(D(real), torch.ones(B, 1, device=real.device)) + \ F.binary_cross_entropy_with_logits(D(fake.detach()), torch.zeros(B, 1, device=real.device)) optD.zero_grad(); d_loss.backward(); optD.step() # generator: fool D g_loss = F.binary_cross_entropy_with_logits(D(fake), torch.ones(B, 1, device=real.device)) optG.zero_grad(); g_loss.backward(); optG.step()Problems: non-convergent dynamics (simultaneous gradient play can cycle), mode collapse (the generator covers few modes), vanishing gradients when supports do not overlap, and no likelihood for evaluation (FID and related sample-based metrics are used instead). Fixes: WGAN and gradient penalty (Earth-mover distance), spectral normalization, two-timescale updates, architectural advances (DCGAN, progressive growing, StyleGAN, BigGAN). GANs gave the sharpest images of the late 2010s, but diffusion models have largely replaced them for high-fidelity, diverse generation thanks to stable training and better coverage; GANs persist where one-step speed matters (real-time super-resolution, some distillation targets, adversarial losses as add-ons).
Diffusion models
Section titled “Diffusion models”Idea. Define a fixed forward process that gradually destroys data with noise, and learn to reverse it. With a variance schedule and , the noisy sample at step has a closed form:
At the signal is essentially pure Gaussian noise. The reverse process is also approximately Gaussian for small steps; the network just needs to predict its mean, which is equivalent to predicting the added noise. The DDPM (Ho et al., 2020) training objective simplifies the ELBO to
It is a plain regression: pick a random , noise the data, predict the noise. Training is stable; no adversary.
def diffusion_loss(model, x0, alphas_cumprod): B = x0.size(0) t = torch.randint(0, len(alphas_cumprod), (B,), device=x0.device) a = alphas_cumprod[t].view(B, *([1] * (x0.dim() - 1))) eps = torch.randn_like(x0) xt = a.sqrt() * x0 + (1 - a).sqrt() * eps return F.mse_loss(model(xt, t), eps)Score view (Song and Ermon; Song et al., 2021). The score points toward higher density. Noise prediction is equivalent to score estimation: (denoising score matching; Tweedie’s formula links the optimal denoiser to the score). Sampling then follows the reverse-time SDE, or the deterministic probability-flow ODE with the same marginals, solved by fast ODE solvers (DDIM, DPM-Solver) in tens of steps; distillation and consistency models reduce this to a few or one step.
Guidance. Classifier-free guidance trains the model with and without a condition (randomly dropping it) and samples with , , trading diversity for prompt fidelity. Latent diffusion (Stable Diffusion) runs diffusion in an autoencoder’s latent space, reducing compute by orders of magnitude. Backbones moved from U-Nets to diffusion transformers (DiT). Text conditioning uses cross-attention to text-encoder embeddings (Attention & Transformers).
Flow matching and flows
Section titled “Flow matching and flows”Normalizing flows build from invertible layers with tractable Jacobians, so exactly; architectural constraints (coupling layers, autoregressive flows) limit flexibility.
Continuous normalizing flows / flow matching (Lipman et al., 2022; Liu et al., 2022 “rectified flow”; Albergo and Vanden-Eijnden) define a velocity field transporting noise to data along an ODE . The training objective is again regression: with a simple path ( noise, data), the conditional target velocity is :
This is simulation-free, simple, and closely related to diffusion (diffusion’s probability-flow ODE is a special case of a flow with a particular path and weighting). Straighter paths allow fewer sampling steps. Recent large image and video generators have adopted flow-matching or rectified-flow objectives with transformer backbones. The distinction between “diffusion” and “flow matching” has blurred; both learn a time-dependent vector field.
Energy-based and likelihood views
Section titled “Energy-based and likelihood views”An energy-based model defines with an unnormalized energy and intractable partition function . The maximum-likelihood gradient is
“lower energy on data, raise energy on the model’s own samples” (the negative phase needs MCMC, e.g. Langevin dynamics ). Training is delicate. The score does not involve , the key insight behind score matching and thus diffusion: learn the score, not the density, and sample with Langevin-style dynamics.
Unifying summary
- Autoregressive, flows: exact likelihood by construction.
- VAEs, diffusion: optimize an ELBO (diffusion is a hierarchical VAE with a fixed encoder).
- GANs: implicit models; training minimizes a divergence estimated by a learned critic.
- Score/diffusion/flow matching: learn vector fields (gradients or velocities) and integrate.
- Energy-based: model log-density up to a constant.
Evaluation: log-likelihood (bits per dimension) where available; FID/KID, precision-recall of samples, CLIP score for text alignment, and human preference; none is perfect, and likelihood does not always track sample quality.
Practical choices
Section titled “Practical choices”- Text, code, structured tokens: autoregressive transformers.
- High-quality images, video, audio: latent diffusion/flow transformers; use pretrained models and fine-tune (LoRA) or condition (ControlNet/adapters).
- Need a compact learned representation or tokenizer: VAE/VQ-VAE.
- Real-time single-step synthesis: distilled diffusion, consistency models, or GANs.
- Always check for memorization and safety/consent issues with training data; generative models can reproduce training examples.
Key takeaways
Section titled “Key takeaways”- A generative model estimates ; likelihood training equals minimizing forward KL.
- The VAE maximizes the ELBO (reconstruction minus KL to the prior) using the reparameterization trick.
- GANs play a minimax game; powerful but unstable, with mode collapse and no likelihood.
- Diffusion trains by predicting noise at random noise levels; equivalently learns the score; sampling reverses the noising iteratively (or via an ODE).
- Flow matching regresses a velocity field along simple paths; it is simple, fast, and now widely used.
- The score of an energy-based model avoids the partition function, which is why score-based methods scale.
Exercises
Section titled “Exercises”- Derivation. Show that the ELBO gap is .
- Derivation. Show the optimal GAN discriminator is .
- Hands-on. Train the VAE above on MNIST. Plot reconstructions, samples from , and interpolation between two encoded digits. Vary .
- Hands-on. Train a tiny 2D diffusion model on a “two moons” dataset with an MLP conditioned on ; visualize samples at various step counts. Then train a flow-matching model on the same data and compare.
- Conceptual. Why does mode-covering (forward KL) training tend to blur samples while GANs produce sharp but less diverse ones?
Solutions (1, 2)
- .
- Maximizing pointwise: derivative gives .
Further reading
Section titled “Further reading”- Kingma and Welling, “Auto-Encoding Variational Bayes” (2013); Kingma and Welling, “An Introduction to Variational Autoencoders” (2019).
- Goodfellow et al., “Generative Adversarial Nets” (2014); Arjovsky et al., “Wasserstein GAN” (2017); Karras et al., “A Style-Based Generator Architecture for GANs” (2018).
- Ho, Jain, Abbeel, “Denoising Diffusion Probabilistic Models” (2020).
- Song et al., “Score-Based Generative Modeling through Stochastic Differential Equations” (2021).
- Rombach et al., “High-Resolution Image Synthesis with Latent Diffusion Models” (2022); Peebles and Xie, “Scalable Diffusion Models with Transformers” (2022).
- Lipman et al., “Flow Matching for Generative Modeling” (2022); Liu et al., “Flow Straight and Fast: Rectified Flow” (2022).
- Ho and Salimans, “Classifier-Free Diffusion Guidance” (2022).
- Lilian Weng, “What are Diffusion Models?” (blog, 2021).
- LeCun et al., “A Tutorial on Energy-Based Learning” (2006).