Skip to content

Information Theory

What you’ll learn. What entropy measures and why it has the formula it does, how cross-entropy and KL divergence quantify mismatch between distributions, why minimizing cross-entropy equals maximum likelihood, what mutual information captures, and how these ideas appear in language modeling (perplexity), VAEs, distillation, and RL. Short NumPy examples included.

Claude Shannon asked: how much information does observing an event carry? Information should be larger for rarer events and additive for independent events. The only function (up to scaling) satisfying that is the surprisal

I(x)=−log⁡p(x).I(x)=-\log p(x).

Independent events multiply probabilities, so surprisals add. A fair coin flip carries log⁡22=1\log_2 2=1 bit; an event of probability 1/10241/1024 carries 10 bits.

Entropy is expected surprisal:

H(p)=Ex∼p[−log⁡p(x)]=−∑xp(x)log⁡p(x).H(p)=\mathbb{E}_{x\sim p}[-\log p(x)]=-\sum_x p(x)\log p(x).

Properties:

  • H≥0H\ge0, with 0 iff the outcome is deterministic.
  • Maximum log⁡K\log K for the uniform distribution over KK outcomes.
  • Operational meaning: H(p)H(p) in bits is the minimum average number of bits per symbol of any lossless code for samples from pp (source coding theorem). Huffman and arithmetic coding approach it.
  • Units: log⁡2\log_2 gives bits, natural ln⁡\ln gives nats. ML code uses nats by default. 1 nat ≈1.443\approx1.443 bits.

Example: a biased coin with p=0.9p=0.9 has H=−0.9log⁡20.9−0.1log⁡20.1≈0.469H=-0.9\log_2 0.9-0.1\log_2 0.1\approx0.469 bits, less than 1 because outcomes are predictable.

import numpy as np
def entropy(p, base=2):
p = np.asarray(p, dtype=float)
p = p[p > 0] # 0*log0 = 0 by convention
return -(p * np.log(p)).sum() / np.log(base)
print(entropy([0.5, 0.5]), entropy([0.9, 0.1]), entropy(np.ones(8) / 8)) # 1, 0.469, 3

Differential entropy for continuous variables, h(p)=−∫plog⁡ph(p)=-\int p\log p, can be negative and is not invariant to rescaling; relative quantities (KL, mutual information) are better behaved. A Gaussian has h=12log⁡(2πeσ2)h=\tfrac12\log(2\pi e\sigma^2) and maximizes entropy for a fixed variance.

Maximum entropy principle. Among all distributions consistent with known constraints, pick the one with the highest entropy: it assumes nothing extra. Constraining the mean of a nonnegative variable gives the exponential; constraining expected features gives exponential-family distributions, including softmax/logistic models.

Suppose the true distribution is pp but you encode with a code optimized for a model qq. The average code length is the cross-entropy

H(p,q)=−∑xp(x)log⁡q(x).H(p,q)=-\sum_x p(x)\log q(x).

It is always ≥H(p)\ge H(p) with equality iff q=pq=p. The excess is the KL divergence (next section).

Why this is the classification loss. For a one-hot label p=eyp=e_y, H(p,q)=−log⁡qyH(p,q)=-\log q_y: the negative log-probability the model assigns to the right class. Averaged over data, that is exactly the negative log-likelihood, so minimizing cross-entropy == maximum likelihood (Statistics & Estimation). For softmax outputs q=softmax⁡(z)q=\operatorname{softmax}(z):

ℓ=−zy+log⁡∑jezj,∂ℓ∂zk=qk−1[k=y].\ell=-z_y+\log\sum_j e^{z_j},\qquad\frac{\partial\ell}{\partial z_k}=q_k-\mathbb 1[k=y].

The gradient is simply “prediction minus target,” a neat fact that makes training stable and is why softmax and cross-entropy are always fused (F.cross_entropy takes logits and applies log-softmax internally; never softmax first).

def cross_entropy_from_logits(z, y):
z = z - z.max(axis=1, keepdims=True) # stability
logp = z - np.log(np.exp(z).sum(axis=1, keepdims=True))
return -logp[np.arange(len(y)), y].mean()

Perplexity. For a language model, per-token cross-entropy HH (nats) gives perplexity exp⁡(H)\exp(H): the effective number of equally likely choices per token. A model with loss 2.0 nats has perplexity about 7.4. Bits per byte/character is the tokenizer-independent variant, used to compare models with different vocabularies. See also Attention & Transformers.

The Kullback–Leibler divergence from qq to pp:

DKL(p∥q)=∑xp(x)log⁡p(x)q(x)=H(p,q)−H(p).D_{KL}(p\parallel q)=\sum_x p(x)\log\frac{p(x)}{q(x)}=H(p,q)-H(p).

Properties:

  • DKL≥0D_{KL}\ge0 (Gibbs’ inequality), zero iff p=qp=q. Proof via Jensen: −DKL=Ep[log⁡qp]≤log⁡Ep[qp]=log⁡1=0-D_{KL}=\mathbb{E}_p[\log\tfrac qp]\le\log\mathbb{E}_p[\tfrac qp]=\log1=0.
  • Not symmetric and not a metric; it does not satisfy the triangle inequality.
  • It is the expected extra bits (nats) paid for using qq in place of pp.
  • Since H(p)H(p) is constant w.r.t. the model, minimizing cross-entropy over qq is the same as minimizing DKL(pdata∥qθ)D_{KL}(p_{\text{data}}\parallel q_\theta): maximum likelihood is KL projection of the empirical distribution onto the model family.

Closed form for Gaussians (used in VAEs): for p=N(μ,σ2)p=\mathcal N(\mu,\sigma^2) and q=N(0,1)q=\mathcal N(0,1),

DKL=12(μ2+σ2−1−ln⁡σ2).D_{KL}=\tfrac12\big(\mu^2+\sigma^2-1-\ln\sigma^2\big).

The direction matters qualitatively. Fit a unimodal qq to a bimodal pp:

  • Forward DKL(p∥q)D_{KL}(p\parallel q) (MLE, “mean-seeking”/mass-covering): qq must put mass wherever pp does, or the penalty blows up, so it spreads across both modes.
  • Reverse DKL(q∥p)D_{KL}(q\parallel p) (variational inference, “mode-seeking”): qq is penalized for putting mass where pp is small, so it locks onto one mode.

This distinction explains blurry samples from MLE-trained models, mode collapse in some objectives, and why RLHF’s KL penalty against a reference policy keeps outputs close to the base model (Reinforcement Learning Basics).

Where KL shows up

  • VAEs: the ELBO has a KL between approximate posterior and prior (Generative Models).
  • Knowledge distillation: minimize DKL(pteacherT∥qstudentT)D_{KL}(p_{\text{teacher}}^{T}\parallel q_{\text{student}}^{T}) with softened temperatures.
  • PPO/TRPO and RLHF: constrain policy updates with a KL term.
  • Monitoring distribution shift between training and serving data.

Jensen–Shannon divergence JS(p,q)=12DKL(p∥m)+12DKL(q∥m)\mathrm{JS}(p,q)=\tfrac12D_{KL}(p\parallel m)+\tfrac12D_{KL}(q\parallel m) with m=p+q2m=\tfrac{p+q}2 is symmetric and bounded; the original GAN objective relates to it. Support mismatches make KL infinite, which is one reason alternatives such as Wasserstein distance exist.

Joint, conditional entropy, mutual information

Section titled “Joint, conditional entropy, mutual information”
H(X,Y)=−∑p(x,y)log⁡p(x,y),H(Y∣X)=∑xp(x)H(Y∣X=x),H(X,Y)=-\sum p(x,y)\log p(x,y),\qquad H(Y\mid X)=\sum_x p(x)H(Y\mid X=x),

with the chain rule H(X,Y)=H(X)+H(Y∣X)H(X,Y)=H(X)+H(Y\mid X). Conditioning cannot increase entropy on average: H(Y∣X)≤H(Y)H(Y\mid X)\le H(Y).

Mutual information is the reduction in uncertainty about one variable from knowing the other:

I(X;Y)=H(Y)−H(Y∣X)=DKL(p(x,y)∥p(x)p(y)).I(X;Y)=H(Y)-H(Y\mid X)=D_{KL}\big(p(x,y)\parallel p(x)p(y)\big).

It is symmetric, ≥0\ge0, and equals 0 iff XX and YY are independent. Unlike correlation it captures arbitrary nonlinear dependence: Y=X2Y=X^2 with symmetric XX has zero correlation but positive mutual information.

def mutual_information(joint):
px = joint.sum(1, keepdims=True); py = joint.sum(0, keepdims=True)
mask = joint > 0
return (joint[mask] * np.log(joint[mask] / (px @ py)[mask])).sum()
joint = np.array([[0.4, 0.1], [0.1, 0.4]])
print(mutual_information(joint)) # nats; >0 since dependent

Uses in ML

  • Feature selection: rank features by I(feature;label)I(\text{feature};\text{label}) (e.g. scikit-learn mutual_info_classif), capturing non-linear relevance.
  • Decision trees: the split criterion information gain is the mutual information between the split and the label (Classical Models).
  • Contrastive learning: InfoNCE is a lower bound on mutual information between views; CLIP-style objectives optimize it.
  • Information bottleneck: learn a representation ZZ maximizing I(Z;Y)I(Z;Y) while minimizing I(Z;X)I(Z;X). Its interpretation as an explanation of deep learning remains debated.

Estimating MI from samples in high dimensions is hard; estimators are biased and variance grows quickly with MI value, so treat reported numbers cautiously.

If X→Y→ZX\to Y\to Z forms a Markov chain, then I(X;Z)≤I(X;Y)I(X;Z)\le I(X;Y): processing cannot create information about XX. Layers of a deterministic network cannot add information about the input; they can only discard it or re-express it in a more usable form. That “usable” aspect, not raw information, is what learning optimizes.

H(p,q)⏟cross-entropy=H(p)⏟irreducible uncertainty+DKL(p∥q)⏟model error.\underbrace{H(p,q)}_{\text{cross-entropy}}=\underbrace{H(p)}_{\text{irreducible uncertainty}}+\underbrace{D_{KL}(p\parallel q)}_{\text{model error}}.

When a language model reaches loss 1.8 nats per token, part of that is irreducible entropy of text itself and only the KL part can be reduced by better modeling. This decomposition also frames scaling-law fits that separate an irreducible term from reducible loss.

  • Applying softmax and then a cross-entropy that already applies softmax (double softmax): a very common PyTorch bug.
  • Computing log⁡(q)\log(q) with q=0q=0 yields infinity: use log-softmax or clip.
  • Treating KL as a distance: it is directional and can be infinite.
  • Mixing bits and nats when comparing numbers.
  • Comparing perplexity across different tokenizers or datasets.
  • Reading small differences in MI estimates as meaningful without confidence intervals.
  • Entropy is average surprisal and the lower bound on lossless code length.
  • Cross-entropy == entropy ++ KL; minimizing it is maximum likelihood.
  • KL is nonnegative, asymmetric, and its direction decides mode-covering vs mode-seeking behavior.
  • Mutual information is the KL between the joint and the product of marginals, a general measure of dependence.
  • Softmax plus cross-entropy has the clean gradient q−onehotq-\text{onehot}.
  • Perplexity is elosse^{\text{loss}} in nats.
  1. Computation. Compute HH (bits) for the distribution (1/2,1/4,1/8,1/8)(1/2,1/4,1/8,1/8). Design a prefix code achieving it.
  2. Derivation. Show ∂ℓ/∂zk=qk−1[k=y]\partial\ell/\partial z_k=q_k-\mathbb 1[k=y] for softmax cross-entropy.
  3. Derivation. Derive DKL(N(μ,σ2)∥N(0,1))D_{KL}(\mathcal N(\mu,\sigma^2)\parallel\mathcal N(0,1)).
  4. Hands-on. Fit a single Gaussian to a two-mode mixture by minimizing forward KL and reverse KL on a grid. Plot both fits.
  5. Conceptual. Why can a classifier have low cross-entropy but be badly calibrated or vice versa? What does temperature scaling fix?
Solutions (1, 2)
  1. H=12⋅1+14⋅2+18⋅3+18⋅3=1.75H=\tfrac12\cdot1+\tfrac14\cdot2+\tfrac18\cdot3+\tfrac18\cdot3=1.75 bits. Code: 0, 10, 110, 111 has average length exactly 1.75.
  2. ℓ=−zy+log⁡∑jezj\ell=-z_y+\log\sum_je^{z_j}. Differentiate: −1[k=y]+ezk/∑jezj=qk−1[k=y]-\mathbb 1[k=y]+e^{z_k}/\sum_je^{z_j}=q_k-\mathbb 1[k=y].
  • Cover and Thomas, Elements of Information Theory (2nd ed., 2006).
  • MacKay, Information Theory, Inference, and Learning Algorithms (2003), free online.
  • Shannon, “A Mathematical Theory of Communication” (1948).
  • Murphy, Probabilistic Machine Learning: An Introduction (2022), chapter 6.
  • van den Oord, Li, Vinyals, “Representation Learning with Contrastive Predictive Coding” (2018).
  • Tishby and Zaslavsky, “Deep Learning and the Information Bottleneck Principle” (2015).