Skip to content

Attention & Transformers

What you’ll learn. Why attention was invented and how scaled dot-product self-attention is derived from a soft dictionary lookup; what multi-head attention adds; why positional information must be injected and the main schemes (sinusoidal, learned, RoPE, ALiBi); the structure of a modern pre-norm transformer block; how encoder, decoder, and encoder-decoder variants differ; compute and memory costs; and a complete minimal GPT in PyTorch.

RNN encoder-decoders compress a whole input sequence into one fixed-size vector, a bottleneck, and process tokens sequentially, preventing parallelism and making long-range dependencies hard. Attention (Bahdanau et al., 2014) lets each output step look directly at all input positions and take a weighted average. Vaswani et al. (2017, “Attention Is All You Need”) showed that attention alone, with no recurrence or convolution, suffices, and trains in parallel across positions.

A dictionary maps a key to a value; a query retrieves the value whose key matches. Soft lookup compares the query with all keys, converts similarities into weights, and returns a weighted mix of values:

Attn(q,K,V)=∑jαjvj,α=softmax⁡(s(q,k1),…,s(q,kn)).\text{Attn}(q,K,V)=\sum_{j}\alpha_j v_j,\qquad\alpha=\operatorname{softmax}\big(s(q,k_1),\dots,s(q,k_n)\big).

With dot-product similarity s(q,k)=q⊤k/dks(q,k)=q^\top k/\sqrt{d_k} and matrices Q∈Rnq×dkQ\in\mathbb{R}^{n_q\times d_k}, K∈Rnk×dkK\in\mathbb{R}^{n_k\times d_k}, V∈Rnk×dvV\in\mathbb{R}^{n_k\times d_v}:

Attention(Q,K,V)=softmax⁡ ⁣(QK⊤dk)V\boxed{\text{Attention}(Q,K,V)=\operatorname{softmax}\!\Big(\frac{QK^\top}{\sqrt{d_k}}\Big)V}

Softmax is applied row-wise, so each query’s weights sum to 1. Dot products are similarity measures; softmax makes the weights a probability distribution (Probability).

If components of qq and kk are independent with zero mean and unit variance, q⊤k=∑i=1dkqikiq^\top k=\sum_{i=1}^{d_k}q_ik_i has variance dkd_k. Large-magnitude logits push softmax into saturated regions where its gradient is nearly zero (one-hot outputs). Dividing by dk\sqrt{d_k} restores unit variance and keeps training healthy.

In self-attention queries, keys, and values all come from the same sequence X∈RT×dmodelX\in\mathbb{R}^{T\times d_{model}} through learned linear projections:

Q=XWQ,K=XWK,V=XWV,WQ,WK∈Rdmodel×dk, WV∈Rdmodel×dv.Q=XW_Q,\quad K=XW_K,\quad V=XW_V,\qquad W_Q,W_K\in\mathbb{R}^{d_{model}\times d_k},\ W_V\in\mathbb{R}^{d_{model}\times d_v}.

Each token produces a query (“what am I looking for?”), a key (“what do I offer?”), and a value (“what do I contribute?”). The score matrix S=QK⊤/dk∈RT×TS=QK^\top/\sqrt{d_k}\in\mathbb{R}^{T\times T} holds all pairwise interactions; softmax(S)V\text{softmax}(S)V mixes values. Every position connects to every other in one step (path length O(1)O(1), versus O(T)O(T) for an RNN). Compare with a convolution that mixes a fixed local window with fixed weights: attention weights are input-dependent.

Causal masking for autoregressive models: add −∞-\infty to scores where key position >> query position, so token tt attends only to positions ≤t\le t. This lets one forward pass train all TT next-token predictions in parallel (teacher forcing).

A tiny NumPy version:

import numpy as np
def softmax(x, axis=-1):
x = x - x.max(axis=axis, keepdims=True)
e = np.exp(x); return e / e.sum(axis=axis, keepdims=True)
def attention(Q, K, V, causal=False):
S = Q @ K.T / np.sqrt(K.shape[-1])
if causal:
S = np.where(np.triu(np.ones_like(S), k=1) == 1, -1e9, S)
A = softmax(S)
return A @ V, A

Permutation property: self-attention without positional information is permutation-equivariant: shuffling input tokens just shuffles outputs. It sees a set, not a sequence, hence position must be added.

One attention operation averages over positions with a single weighting pattern. Multi-head attention runs hh attention operations in parallel in lower-dimensional subspaces, letting different heads specialize (syntactic relations, previous token, copying, etc.):

headi=Attention(XWQi,XWKi,XWVi),MHA(X)=Concat⁡(head1,…,headh) WO.\text{head}_i=\text{Attention}(XW_Q^i,XW_K^i,XW_V^i),\qquad\text{MHA}(X)=\operatorname{Concat}(\text{head}_1,\dots,\text{head}_h)\,W_O.

With dk=dv=dmodel/hd_k=d_v=d_{model}/h the total cost matches single-head full-dimension attention. Implementation trick: compute Q,K,VQ,K,V with one big matrix each, then reshape to (batch, heads, T, d_head).

Efficiency variants (important for inference memory): multi-query attention (MQA) shares one K/V head across all query heads; grouped-query attention (GQA) shares K/V among groups of query heads. They shrink the KV cache, the stored keys/values for past tokens during generation, which dominates memory at long context. Multi-head latent attention (as introduced in the DeepSeek models) compresses K/V into a low-rank latent for the same purpose.

Since attention is order-agnostic, inject position.

  • Sinusoidal (original): add fixed vectors PE(p,2i)=sin⁡(p/100002i/d)PE_{(p,2i)}=\sin(p/10000^{2i/d}), PE(p,2i+1)=cos⁡(p/100002i/d)PE_{(p,2i+1)}=\cos(p/10000^{2i/d}) to token embeddings. Each frequency encodes position at a different scale; relative offsets correspond to linear maps on these vectors.
  • Learned absolute embeddings (GPT-2, BERT): a trainable table up to a max length; cannot extrapolate beyond it.
  • Relative position biases (T5): add a learned scalar bias to attention logits by distance bucket.
  • RoPE (rotary, Su et al. 2021): rotate each pair of query/key dimensions by an angle proportional to position, qm→Rmqmq_m\to R_mq_m, so that ⟨Rmq,Rnk⟩\langle R_mq,R_nk\rangle depends only on m−nm-n. It encodes relative position inside the dot product, with no extra parameters. The default in most open LLMs (Llama family and many others). Context extension tricks (position interpolation, NTK-aware scaling, YaRN) rescale the rotation frequencies to run at longer contexts than trained.
  • ALiBi: add a head-specific linear penalty −m⋅(i−j)-m\cdot(i-j) to logits; extrapolates reasonably to longer sequences.

Modern decoder blocks are pre-norm with residual connections:

x←x+MHA(Norm(x)),x←x+FFN(Norm(x)).x\leftarrow x+\text{MHA}(\text{Norm}(x)),\qquad x\leftarrow x+\text{FFN}(\text{Norm}(x)).
x ──────────────┬──────────────┐
│ │
[RMSNorm] │
│ │
[Multi-head attention] │
│ │
└────── + ─────┘
x ──────────────┬──────────────┐
[RMSNorm] │
[Feed-forward (MLP)] │
└────── + ─────┘
  • Feed-forward network (FFN): applied independently per token, FFN(x)=W2 ϕ(W1x)\text{FFN}(x)=W_2\,\phi(W_1x) with hidden size ≈4dmodel\approx4d_{model}; gated variants (SwiGLU) W2(SiLU(W1x)⊙W3x)W_2(\text{SiLU}(W_1x)\odot W_3x) with hidden ≈83dmodel\approx\tfrac83d_{model} are common. The FFN holds roughly two-thirds of block parameters and is often interpreted as key-value memory storing factual associations; attention routes information between tokens, the FFN transforms it within a token.
  • Residual stream: a shared dmodeld_{model}-dimensional vector per token that every layer reads from and writes to additively: the central object in mechanistic interpretability and the reason deep stacks train (Training Deep Networks).
  • Normalization: LayerNorm or RMSNorm, pre-norm placement for stability.
  • Final norm and an output projection to vocabulary logits (often tied with the input embedding matrix).

Parameter count of a decoder with LL layers: about 12Ldmodel212Ld_{model}^2 (4 for attention projections plus 8 for a 4x FFN) plus embeddings VdmodelVd_{model}. Example: L=32,d=4096L=32,d=4096 gives ≈6.4B\approx6.4\text{B} non-embedding parameters (GPT-style 7B scale).

  • Encoder-only (BERT): bidirectional attention, masked-token pretraining; strong for classification and embeddings.
  • Decoder-only (GPT, Llama, most modern LLMs): causal attention, next-token prediction; unified for generation and, with scale, nearly everything else.
  • Encoder-decoder (original Transformer, T5, Whisper): encoder reads the input bidirectionally; decoder generates with cross-attention where queries come from the decoder and keys/values from encoder outputs.
  • Mixture-of-experts (MoE): replace the FFN with many expert FFNs and a router that activates top-kk per token, giving large total parameter counts with smaller active compute; widely used in recent frontier models.

Training objective for decoder-only: minimize cross-entropy of the next token (Information Theory), L=−∑tlog⁡pθ(xt∣x<t)L=-\sum_t\log p_\theta(x_t\mid x_{<t}). Text is first split into tokens by a subword tokenizer (BPE, SentencePiece/unigram), typically 32k to 200k vocabulary entries.

Let TT be sequence length, dd the model width.

  • Attention: O(T2d)O(T^2d) time and O(T2)O(T^2) memory for the score matrix (per head). Projections and FFN: O(Td2)O(Td^2). For T<dT<d the FFN/projections dominate; for long contexts attention does.
  • FlashAttention (Dao et al., 2022) computes exact attention in tiles that stay in on-chip SRAM, never materializing the T×TT\times T matrix: same math, large speedups and memory savings by reducing memory traffic. Use F.scaled_dot_product_attention, which dispatches to fused kernels.
  • Inference generates one token at a time, reusing a KV cache: per token per layer, 2⋅nkv_heads⋅dhead2\cdot n_{kv\_heads}\cdot d_{head} values. For example, 32 layers, 8 KV heads, head dim 128, fp16: 32⋅2⋅8⋅128⋅232\cdot2\cdot8\cdot128\cdot2 bytes =131=131 KB per token, so a 100k-token context is about 13 GB per sequence.
  • Sparse/sliding-window attention, linear attention, and state-space hybrids trade exactness for O(T)O(T) scaling.

A complete small GPT-style model (pre-norm, causal, multi-head):

import math, torch, torch.nn as nn, torch.nn.functional as F
class CausalSelfAttention(nn.Module):
def __init__(self, d_model, n_heads, dropout=0.0):
super().__init__()
assert d_model % n_heads == 0
self.h, self.dk = n_heads, d_model // n_heads
self.qkv = nn.Linear(d_model, 3 * d_model, bias=False)
self.proj = nn.Linear(d_model, d_model, bias=False)
self.dropout = dropout
def forward(self, x): # x: (B, T, C)
B, T, C = x.shape
q, k, v = self.qkv(x).split(C, dim=2)
q, k, v = (t.view(B, T, self.h, self.dk).transpose(1, 2) for t in (q, k, v)) # (B,h,T,dk)
y = F.scaled_dot_product_attention(q, k, v, is_causal=True,
dropout_p=self.dropout if self.training else 0.0)
y = y.transpose(1, 2).contiguous().view(B, T, C)
return self.proj(y)
class Block(nn.Module):
def __init__(self, d_model, n_heads):
super().__init__()
self.n1, self.n2 = nn.LayerNorm(d_model), nn.LayerNorm(d_model)
self.attn = CausalSelfAttention(d_model, n_heads)
self.mlp = nn.Sequential(nn.Linear(d_model, 4 * d_model), nn.GELU(),
nn.Linear(4 * d_model, d_model))
def forward(self, x):
x = x + self.attn(self.n1(x))
return x + self.mlp(self.n2(x))
class GPT(nn.Module):
def __init__(self, vocab, d_model=256, n_heads=4, n_layers=4, max_len=256):
super().__init__()
self.tok = nn.Embedding(vocab, d_model)
self.pos = nn.Embedding(max_len, d_model) # learned absolute positions (simple)
self.blocks = nn.ModuleList(Block(d_model, n_heads) for _ in range(n_layers))
self.norm = nn.LayerNorm(d_model)
self.head = nn.Linear(d_model, vocab, bias=False)
self.head.weight = self.tok.weight # weight tying
self.apply(self._init)
def _init(self, m):
if isinstance(m, (nn.Linear, nn.Embedding)):
nn.init.normal_(m.weight, std=0.02)
def forward(self, idx, targets=None):
B, T = idx.shape
x = self.tok(idx) + self.pos(torch.arange(T, device=idx.device))
for blk in self.blocks: x = blk(x)
logits = self.head(self.norm(x))
loss = None if targets is None else F.cross_entropy(
logits.view(-1, logits.size(-1)), targets.view(-1))
return logits, loss
@torch.no_grad()
def generate(self, idx, n_new, temperature=1.0, top_k=None):
for _ in range(n_new):
logits, _ = self(idx[:, -self.pos.num_embeddings:])
logits = logits[:, -1] / temperature
if top_k:
v, _ = torch.topk(logits, top_k)
logits[logits < v[:, [-1]]] = -float("inf")
idx = torch.cat([idx, torch.multinomial(F.softmax(logits, -1), 1)], dim=1)
return idx

Training loop essentials: sample random windows of length T+1T+1 from the token stream, input x = w[:-1], target y = w[1:], AdamW with warmup plus cosine decay, gradient clipping at 1.0, bf16 autocast (Training Deep Networks, PyTorch in Practice). On a char-level Shakespeare corpus this 4-layer model reaches plausible text within minutes on one GPU. Sanity checks: initial loss ≈ln⁡V\approx\ln V; causal mask correct (changing a future token must not change earlier logits); overfit a single batch.

Decoding: greedy (argmax) is repetitive; sampling with temperature, top-kk, or top-pp (nucleus) trades diversity and quality; beam search suits translation. Generation cost per token is dominated by memory bandwidth (reading weights and KV cache), motivating quantization, speculative decoding, and batching.

Parallel training over positions maps onto GPUs; the attention path length of O(1)O(1) eases long-range dependencies; they scale smoothly with data and compute (scaling laws) and transfer across modalities (ViT for images, Whisper for audio, multimodal LLMs). The costs: quadratic attention and large KV caches, motivating the efficiency work above. The tension between attention’s recall ability and the efficiency of recurrent/state-space layers is an active area, with hybrid models appearing in recent releases; this is fast-moving, so check current literature.

  • Forgetting the causal mask, or applying the mask after softmax: the model “cheats” and reaches suspiciously low training loss, then fails at generation.
  • Missing the 1/dk1/\sqrt{d_k} scaling in hand-written attention.
  • Padding tokens attended to: pass an attention mask.
  • Position embeddings shorter than the sequence; mismatched RoPE settings between training and inference.
  • Mixing up (B,T,h,d) and (B,h,T,d) layouts in reshapes (use .transpose then .contiguous() before .view).
  • Softmax in low precision overflowing; keep it in fp32 (fused kernels do).
  • Attention is a differentiable soft dictionary lookup: softmax(QK⊤/dk)V\text{softmax}(QK^\top/\sqrt{d_k})V; the scaling stabilizes softmax gradients.
  • Self-attention gives all-pairs, input-dependent mixing with O(1)O(1) path length and O(T2)O(T^2) cost; masking enforces causality.
  • Multi-head attention learns several relations in parallel; GQA/MQA shrink the KV cache.
  • Position must be injected: sinusoidal, learned, relative biases, RoPE (most common now), or ALiBi.
  • A block is attention plus FFN, each wrapped in pre-norm and a residual connection; the FFN holds most parameters.
  • Training is next-token cross-entropy; inference uses a KV cache; FlashAttention-style kernels make long sequences practical.
  1. Derivation. Show that if q,k∈Rdq,k\in\mathbb{R}^{d} have i.i.d. unit-variance zero-mean entries then Var⁡(q⊤k)=d\operatorname{Var}(q^\top k)=d.
  2. Conceptual. Prove self-attention without positions is permutation-equivariant. What changes if you add positional encodings?
  3. Hands-on. Implement the NumPy attention above with a causal mask and verify that perturbing token jj does not alter outputs at positions <j<j.
  4. Hands-on. Train the GPT above on a small text file; plot loss and generate samples at several temperatures. Replace learned positions with RoPE.
  5. Computation. Compute parameters for L=24L=24, d=2048d=2048, V=50,000V=50{,}000 with a 4x FFN and tied embeddings. Estimate KV-cache size for 8k tokens with 16 heads of dim 128 in fp16 (full multi-head).
Solutions (1, 5)
  1. Var⁡(∑qiki)=∑Var⁡(qiki)=d⋅E[qi2]E[ki2]=d\operatorname{Var}(\sum q_ik_i)=\sum\operatorname{Var}(q_ik_i)=d\cdot\mathbb{E}[q_i^2]\mathbb{E}[k_i^2]=d (independence, zero means).
  2. Non-embedding ≈12⋅24⋅20482≈1.21B\approx12\cdot24\cdot2048^2\approx1.21\text{B}; embeddings 50,000⋅2048≈0.10B50{,}000\cdot2048\approx0.10\text{B} (tied); total ≈1.3B\approx1.3\text{B}. KV cache per token =24⋅2⋅16⋅128⋅2=24\cdot2\cdot16\cdot128\cdot2 B ≈196.6\approx196.6 KB, times 8192 ≈1.6\approx1.6 GB.
  • Vaswani et al., “Attention Is All You Need” (2017).
  • Bahdanau, Cho, Bengio, “Neural Machine Translation by Jointly Learning to Align and Translate” (2014).
  • Radford et al., “Language Models are Unsupervised Multitask Learners” (GPT-2, 2019); Devlin et al., “BERT” (2018).
  • Su et al., “RoFormer: Enhanced Transformer with Rotary Position Embedding” (2021); Press et al., “Train Short, Test Long” (ALiBi, 2021).
  • Dao et al., “FlashAttention” (2022); Shazeer, “Fast Transformer Decoding: One Write-Head is All You Need” (2019); Ainslie et al., “GQA” (2023).
  • Shazeer, “GLU Variants Improve Transformer” (2020).
  • Elhage et al., “A Mathematical Framework for Transformer Circuits” (2021).
  • Karpathy, “Let’s build GPT: from scratch, in code, spelled out” (video, 2023) and nanoGPT.
  • Phuong and Hutter, “Formal Algorithms for Transformers” (2022).