Information Theory
What you’ll learn. What entropy measures and why it has the formula it does, how cross-entropy and KL divergence quantify mismatch between distributions, why minimizing cross-entropy equals maximum likelihood, what mutual information captures, and how these ideas appear in language modeling (perplexity), VAEs, distillation, and RL. Short NumPy examples included.
Surprise and entropy
Section titled “Surprise and entropy”Claude Shannon asked: how much information does observing an event carry? Information should be larger for rarer events and additive for independent events. The only function (up to scaling) satisfying that is the surprisal
Independent events multiply probabilities, so surprisals add. A fair coin flip carries bit; an event of probability carries 10 bits.
Entropy is expected surprisal:
Properties:
- , with 0 iff the outcome is deterministic.
- Maximum for the uniform distribution over outcomes.
- Operational meaning: in bits is the minimum average number of bits per symbol of any lossless code for samples from (source coding theorem). Huffman and arithmetic coding approach it.
- Units: gives bits, natural gives nats. ML code uses nats by default. 1 nat bits.
Example: a biased coin with has bits, less than 1 because outcomes are predictable.
import numpy as np
def entropy(p, base=2): p = np.asarray(p, dtype=float) p = p[p > 0] # 0*log0 = 0 by convention return -(p * np.log(p)).sum() / np.log(base)
print(entropy([0.5, 0.5]), entropy([0.9, 0.1]), entropy(np.ones(8) / 8)) # 1, 0.469, 3Differential entropy for continuous variables, , can be negative and is not invariant to rescaling; relative quantities (KL, mutual information) are better behaved. A Gaussian has and maximizes entropy for a fixed variance.
Maximum entropy principle. Among all distributions consistent with known constraints, pick the one with the highest entropy: it assumes nothing extra. Constraining the mean of a nonnegative variable gives the exponential; constraining expected features gives exponential-family distributions, including softmax/logistic models.
Cross-entropy
Section titled “Cross-entropy”Suppose the true distribution is but you encode with a code optimized for a model . The average code length is the cross-entropy
It is always with equality iff . The excess is the KL divergence (next section).
Why this is the classification loss. For a one-hot label , : the negative log-probability the model assigns to the right class. Averaged over data, that is exactly the negative log-likelihood, so minimizing cross-entropy maximum likelihood (Statistics & Estimation). For softmax outputs :
The gradient is simply “prediction minus target,” a neat fact that makes training stable and is why softmax and cross-entropy are always fused (F.cross_entropy takes logits and applies log-softmax internally; never softmax first).
def cross_entropy_from_logits(z, y): z = z - z.max(axis=1, keepdims=True) # stability logp = z - np.log(np.exp(z).sum(axis=1, keepdims=True)) return -logp[np.arange(len(y)), y].mean()Perplexity. For a language model, per-token cross-entropy (nats) gives perplexity : the effective number of equally likely choices per token. A model with loss 2.0 nats has perplexity about 7.4. Bits per byte/character is the tokenizer-independent variant, used to compare models with different vocabularies. See also Attention & Transformers.
KL divergence
Section titled “KL divergence”The Kullback–Leibler divergence from to :
Properties:
- (Gibbs’ inequality), zero iff . Proof via Jensen: .
- Not symmetric and not a metric; it does not satisfy the triangle inequality.
- It is the expected extra bits (nats) paid for using in place of .
- Since is constant w.r.t. the model, minimizing cross-entropy over is the same as minimizing : maximum likelihood is KL projection of the empirical distribution onto the model family.
Closed form for Gaussians (used in VAEs): for and ,
Forward vs reverse KL
Section titled “Forward vs reverse KL”The direction matters qualitatively. Fit a unimodal to a bimodal :
- Forward (MLE, “mean-seeking”/mass-covering): must put mass wherever does, or the penalty blows up, so it spreads across both modes.
- Reverse (variational inference, “mode-seeking”): is penalized for putting mass where is small, so it locks onto one mode.
This distinction explains blurry samples from MLE-trained models, mode collapse in some objectives, and why RLHF’s KL penalty against a reference policy keeps outputs close to the base model (Reinforcement Learning Basics).
Where KL shows up
- VAEs: the ELBO has a KL between approximate posterior and prior (Generative Models).
- Knowledge distillation: minimize with softened temperatures.
- PPO/TRPO and RLHF: constrain policy updates with a KL term.
- Monitoring distribution shift between training and serving data.
Jensen–Shannon divergence with is symmetric and bounded; the original GAN objective relates to it. Support mismatches make KL infinite, which is one reason alternatives such as Wasserstein distance exist.
Joint, conditional entropy, mutual information
Section titled “Joint, conditional entropy, mutual information”with the chain rule . Conditioning cannot increase entropy on average: .
Mutual information is the reduction in uncertainty about one variable from knowing the other:
It is symmetric, , and equals 0 iff and are independent. Unlike correlation it captures arbitrary nonlinear dependence: with symmetric has zero correlation but positive mutual information.
def mutual_information(joint): px = joint.sum(1, keepdims=True); py = joint.sum(0, keepdims=True) mask = joint > 0 return (joint[mask] * np.log(joint[mask] / (px @ py)[mask])).sum()
joint = np.array([[0.4, 0.1], [0.1, 0.4]])print(mutual_information(joint)) # nats; >0 since dependentUses in ML
- Feature selection: rank features by (e.g. scikit-learn
mutual_info_classif), capturing non-linear relevance. - Decision trees: the split criterion information gain is the mutual information between the split and the label (Classical Models).
- Contrastive learning: InfoNCE is a lower bound on mutual information between views; CLIP-style objectives optimize it.
- Information bottleneck: learn a representation maximizing while minimizing . Its interpretation as an explanation of deep learning remains debated.
Estimating MI from samples in high dimensions is hard; estimators are biased and variance grows quickly with MI value, so treat reported numbers cautiously.
Data processing inequality
Section titled “Data processing inequality”If forms a Markov chain, then : processing cannot create information about . Layers of a deterministic network cannot add information about the input; they can only discard it or re-express it in a more usable form. That “usable” aspect, not raw information, is what learning optimizes.
Connecting the pieces
Section titled “Connecting the pieces”When a language model reaches loss 1.8 nats per token, part of that is irreducible entropy of text itself and only the KL part can be reduced by better modeling. This decomposition also frames scaling-law fits that separate an irreducible term from reducible loss.
Pitfalls
Section titled “Pitfalls”- Applying softmax and then a cross-entropy that already applies softmax (double softmax): a very common PyTorch bug.
- Computing with yields infinity: use log-softmax or clip.
- Treating KL as a distance: it is directional and can be infinite.
- Mixing bits and nats when comparing numbers.
- Comparing perplexity across different tokenizers or datasets.
- Reading small differences in MI estimates as meaningful without confidence intervals.
Key takeaways
Section titled “Key takeaways”- Entropy is average surprisal and the lower bound on lossless code length.
- Cross-entropy entropy KL; minimizing it is maximum likelihood.
- KL is nonnegative, asymmetric, and its direction decides mode-covering vs mode-seeking behavior.
- Mutual information is the KL between the joint and the product of marginals, a general measure of dependence.
- Softmax plus cross-entropy has the clean gradient .
- Perplexity is in nats.
Exercises
Section titled “Exercises”- Computation. Compute (bits) for the distribution . Design a prefix code achieving it.
- Derivation. Show for softmax cross-entropy.
- Derivation. Derive .
- Hands-on. Fit a single Gaussian to a two-mode mixture by minimizing forward KL and reverse KL on a grid. Plot both fits.
- Conceptual. Why can a classifier have low cross-entropy but be badly calibrated or vice versa? What does temperature scaling fix?
Solutions (1, 2)
- bits. Code: 0, 10, 110, 111 has average length exactly 1.75.
- . Differentiate: .
Further reading
Section titled “Further reading”- Cover and Thomas, Elements of Information Theory (2nd ed., 2006).
- MacKay, Information Theory, Inference, and Learning Algorithms (2003), free online.
- Shannon, “A Mathematical Theory of Communication” (1948).
- Murphy, Probabilistic Machine Learning: An Introduction (2022), chapter 6.
- van den Oord, Li, Vinyals, “Representation Learning with Contrastive Predictive Coding” (2018).
- Tishby and Zaslavsky, “Deep Learning and the Information Bottleneck Principle” (2015).