LLM Architecture in Depth
What you’ll learn. The architecture of a contemporary decoder-only LLM and why each component differs from the 2017 original transformer: pre-norm with RMSNorm, rotary position embeddings, SwiGLU feed-forward blocks, grouped-query and latent attention, mixture-of-experts, and the KV cache. You’ll see how context is extended beyond training length, do parameter accounting, and write a compact working GPT. This assumes you know basic self-attention from Attention and Transformers.
The decoder-only block
Section titled “The decoder-only block”A decoder-only LLM maps token IDs to next-token logits:
ids -> embedding -> [ Block x L ] -> final norm -> unembedding (lm head) -> logitsBlock: x = x + Attention(Norm(x)) # causal self-attention x = x + FFN(Norm(x)) # per-token MLPKey differences from the original encoder-decoder transformer:
- Causal masking only: token attends to positions , so one forward pass trains on all next-token predictions in parallel (teacher forcing).
- Pre-norm: normalization applied before each sublayer, with a clean residual stream. This trains stably at depth without elaborate warmup, unlike the original post-norm.
- No cross-attention and no encoder. Instruction and prompt are simply earlier tokens in the same sequence.
Why decoder-only won: one objective, one stack, efficient pretraining, a natural fit for generation, and in-context prompting subsumes the encoder’s role.
RMSNorm
Section titled “RMSNorm”LayerNorm subtracts the mean and divides by standard deviation. RMSNorm (Zhang and Sennrich, 2019) drops the mean-centering and the bias:
It is cheaper and works as well in practice. Some recent models add QK-norm (normalizing queries and keys before the dot product) to prevent attention-logit blowup at scale.
Rotary position embeddings (RoPE)
Section titled “Rotary position embeddings (RoPE)”Attention is permutation-invariant, so position must be injected. Early models added learned or sinusoidal vectors to the input. RoPE (Su et al., 2021) instead rotates queries and keys by an angle proportional to position, so that the dot product depends only on the relative offset.
Split the head dimension into pairs. For position and pair , rotate by angle with (base typically 10,000 or larger). For a 2D pair:
Because rotations compose, : the score depends on only. Low-index pairs rotate fast (encode local order), high-index pairs rotate slowly (coarse, long-range position). RoPE is applied to Q and K, not V, and no extra parameters are needed.
import torch
def rope_cache(T, d, base=10000.0, device=None): inv = 1.0 / (base ** (torch.arange(0, d, 2, device=device).float() / d)) ang = torch.outer(torch.arange(T, device=device).float(), inv) # (T, d/2) return ang.cos(), ang.sin()
def apply_rope(x, cos, sin): # x: (B, H, T, d) x1, x2 = x[..., 0::2], x[..., 1::2] cos, sin = cos[None, None], sin[None, None] out = torch.stack((x1 * cos - x2 * sin, x1 * sin + x2 * cos), dim=-1) return out.flatten(-2)SwiGLU feed-forward
Section titled “SwiGLU feed-forward”The classic FFN is with hidden width . Gated variants (Shazeer, 2020) multiply two projections:
It has three matrices instead of two, so the hidden width is reduced (about , rounded to a hardware-friendly multiple) to keep parameter count comparable. It consistently gives better loss for the same compute and is now standard.
Attention variants: MHA, MQA, GQA, MLA
Section titled “Attention variants: MHA, MQA, GQA, MLA”Standard multi-head attention (MHA) has query heads and key/value heads. During generation, K and V of all previous tokens are cached (see below), and that cache dominates memory and bandwidth.
- Multi-query attention (MQA): all query heads share one K/V head. Cache shrinks by ; some quality loss.
- Grouped-query attention (GQA): query heads are split into groups, each sharing one K/V head. is MHA, is MQA. A typical setting (e.g. 32 query heads, 8 KV heads) cuts the cache 4x with negligible quality loss; GQA is the de facto standard in open models.
- Multi-head latent attention (MLA), introduced in the DeepSeek-V2 line: project keys and values jointly into a small latent vector per token, cache only that latent, and reconstruct K/V (or fold the up-projection into the query/output matrices) at use time. It achieves a smaller cache than GQA while keeping the expressiveness of many heads. RoPE needs special handling here (a decoupled rotary component) since position rotation does not commute with the low-rank projection.
- Sliding-window / local attention: some layers attend only to the last tokens (cache bounded), often interleaved with full-attention layers.
Mixture of experts (MoE)
Section titled “Mixture of experts (MoE)”Dense FFNs use all parameters for every token. An MoE layer replaces the FFN with expert FFNs and a small router; each token is sent to its top- experts (commonly to 8) and the outputs are combined with the router’s weights:
Total parameters grow with , but active parameters per token only with . A model quoted as, say, 600B total / 40B active has the memory footprint of the former and roughly the per-token FLOPs of the latter. This is a favourable trade in training compute terms, but serving needs all experts resident in memory and requires inter-device communication (expert parallelism).
Practical issues:
- Load balancing: without help the router collapses onto a few experts. Classic remedies are an auxiliary balancing loss; some recent designs use bias-based, loss-free balancing.
- Shared experts: a few experts always active capture common knowledge, leaving routed experts to specialize; fine-grained (many small) experts improve specialization.
- Dropped tokens and capacity: each expert has a capacity limit in batched training.
- Instability and fine-tuning difficulty compared with dense models.
The KV cache
Section titled “The KV cache”During generation at step , token ‘s query attends to keys/values of all tokens . K and V of earlier tokens never change (causal mask), so recomputing them is wasteful. The KV cache stores them per layer and per head:
prefill: run the whole prompt in parallel; fill cache (compute-bound)decode: per new token: compute q,k,v for that token only, append k,v, attend over the cache (memory-bandwidth-bound)It turns per-token decode cost from projections to projections plus an attention read. The price is memory growing linearly with context and batch. Many serving optimizations exist solely to manage it (Inference & Serving).
Long-context techniques
Section titled “Long-context techniques”Attention costs compute and the cache memory. To work at 100K to 1M+ tokens:
- Position extrapolation / interpolation. A model trained at 4K or 8K degrades past its trained length because rotation angles go out of distribution. Position interpolation rescales positions so longer inputs map into the trained range. NTK-aware scaling and YaRN scale RoPE frequencies non-uniformly (preserve high-frequency local detail, stretch low frequencies). Increasing the RoPE base for continued pretraining at longer lengths is a common recipe.
- Staged length training. Pretrain mostly at short context, then run a shorter phase at progressively longer lengths on long-document data, because long-sequence training is expensive.
- Efficient attention kernels. FlashAttention computes exact attention without materializing the matrix, making memory and speed IO-optimal. Ring/sequence parallelism splits the sequence across devices.
- Hybrid local/global layers and sparse attention, e.g. mostly sliding-window layers with periodic global layers.
- Cache compression: quantizing the cache (8- or 4-bit), eviction heuristics, or latent compression (MLA).
- Alternatives to attention such as state-space and hybrid models; see Reasoning, Multimodality & Frontier Trends.
Parameter accounting
Section titled “Parameter accounting”For a dense model with layers, width , query heads, KV heads, head dim , FFN hidden width , vocabulary :
- Attention: : ; : each; : .
- SwiGLU FFN: .
- Norms: negligible. Embeddings: (plus another if untied).
Per layer roughly . With and GQA this is about , hence the rule of thumb for non-embedding parameters. Training compute is about FLOPs for tokens (Pretraining).
A minimal GPT in PyTorch
Section titled “A minimal GPT in PyTorch”A compact modern-style decoder (RMSNorm, RoPE, GQA, SwiGLU, pre-norm, tied embeddings) that you can train on a character-level corpus:
import math, torch, torch.nn as nn, torch.nn.functional as F
class RMSNorm(nn.Module): def __init__(self, d, eps=1e-6): super().__init__(); self.eps = eps; self.g = nn.Parameter(torch.ones(d)) def forward(self, x): return x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + self.eps) * self.g
def rope_cache(T, dh, base=10000.0): inv = 1.0 / (base ** (torch.arange(0, dh, 2).float() / dh)) ang = torch.outer(torch.arange(T).float(), inv) return ang.cos(), ang.sin()
def apply_rope(x, cos, sin): # x: (B,H,T,dh) x1, x2 = x[..., 0::2], x[..., 1::2] c, s = cos[None, None, :x.size(2)], sin[None, None, :x.size(2)] return torch.stack((x1 * c - x2 * s, x1 * s + x2 * c), -1).flatten(-2)
class Attention(nn.Module): def __init__(self, d, n_h, n_kv): super().__init__() self.n_h, self.n_kv, self.dh = n_h, n_kv, d // n_h self.q = nn.Linear(d, n_h * self.dh, bias=False) self.k = nn.Linear(d, n_kv * self.dh, bias=False) self.v = nn.Linear(d, n_kv * self.dh, bias=False) self.o = nn.Linear(n_h * self.dh, d, bias=False) def forward(self, x, cos, sin): B, T, _ = x.shape q = self.q(x).view(B, T, self.n_h, self.dh).transpose(1, 2) k = self.k(x).view(B, T, self.n_kv, self.dh).transpose(1, 2) v = self.v(x).view(B, T, self.n_kv, self.dh).transpose(1, 2) q, k = apply_rope(q, cos, sin), apply_rope(k, cos, sin) rep = self.n_h // self.n_kv # GQA: share each KV head across `rep` query heads k, v = k.repeat_interleave(rep, 1), v.repeat_interleave(rep, 1) y = F.scaled_dot_product_attention(q, k, v, is_causal=True) return self.o(y.transpose(1, 2).reshape(B, T, -1))
class SwiGLU(nn.Module): def __init__(self, d, f): super().__init__() self.w1 = nn.Linear(d, f, bias=False); self.w3 = nn.Linear(d, f, bias=False) self.w2 = nn.Linear(f, d, bias=False) def forward(self, x): return self.w2(F.silu(self.w1(x)) * self.w3(x))
class Block(nn.Module): def __init__(self, d, n_h, n_kv, f): super().__init__() self.n1, self.n2 = RMSNorm(d), RMSNorm(d) self.attn, self.ffn = Attention(d, n_h, n_kv), SwiGLU(d, f) def forward(self, x, cos, sin): x = x + self.attn(self.n1(x), cos, sin) return x + self.ffn(self.n2(x))
class GPT(nn.Module): def __init__(self, V, d=256, L=6, n_h=8, n_kv=2, f=704, max_T=1024): super().__init__() self.emb = nn.Embedding(V, d) self.blocks = nn.ModuleList(Block(d, n_h, n_kv, f) for _ in range(L)) self.norm = RMSNorm(d) self.head = nn.Linear(d, V, bias=False) self.head.weight = self.emb.weight # tie embeddings cos, sin = rope_cache(max_T, d // n_h) self.register_buffer("cos", cos, persistent=False) self.register_buffer("sin", sin, persistent=False) self.apply(self._init) def _init(self, m): if isinstance(m, (nn.Linear, nn.Embedding)): nn.init.normal_(m.weight, std=0.02) def forward(self, ids, targets=None): x = self.emb(ids) for b in self.blocks: x = b(x, self.cos, self.sin) logits = self.head(self.norm(x)) loss = None if targets is None else F.cross_entropy( logits.view(-1, logits.size(-1)), targets.view(-1)) return logits, loss @torch.no_grad() def generate(self, ids, n, temperature=1.0): for _ in range(n): logits, _ = self(ids[:, -1024:]) # no KV cache here: clarity over speed probs = F.softmax(logits[:, -1] / temperature, -1) ids = torch.cat([ids, torch.multinomial(probs, 1)], 1) return ids
# training loop sketch on a character corpus `data` (1D LongTensor of ids)def get_batch(data, B=32, T=128): ix = torch.randint(len(data) - T - 1, (B,)) x = torch.stack([data[i:i + T] for i in ix]) y = torch.stack([data[i + 1:i + T + 1] for i in ix]) return x, y
# model = GPT(V=vocab_size); opt = torch.optim.AdamW(model.parameters(), lr=3e-4, weight_decay=0.1)# for step in range(5000):# x, y = get_batch(data); _, loss = model(x, y)# opt.zero_grad(set_to_none=True); loss.backward()# torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0); opt.step()Notes: generate recomputes the full prefix each step (quadratic). A production version caches K/V per layer, as sketched above. Weight init for deep models often scales output projections by .
Other design choices worth knowing
Section titled “Other design choices worth knowing”- No biases in linear layers (simplifies, no loss in quality).
- Tied vs untied embeddings: tying saves parameters in small models; large models usually untie.
- Parallel attention + FFN (compute both from the same normed input) saves latency in some designs.
- Multi-token prediction heads (predicting several future tokens at training time) improve data efficiency and enable speculative decoding at inference.
- Logit soft-capping and z-loss stabilize training at scale by bounding logits.
Key takeaways
Section titled “Key takeaways”- The standard modern recipe: pre-norm RMSNorm, RoPE on Q/K, SwiGLU FFN, no biases, and GQA (or MLA) for a small KV cache.
- MoE decouples total parameters from per-token compute, at the cost of memory, routing complexity and communication.
- The KV cache makes decoding cheap in FLOPs but expensive in memory; and latent compression are the main levers.
- Long context combines RoPE rescaling, staged training, exact memory-efficient kernels, and hybrid/local attention; advertised windows overstate usable context.
- Non-embedding parameters scale as about ; training FLOPs as about .
Exercises
Section titled “Exercises”- Conceptual. Show that RoPE attention scores depend only on relative position. (Hint: rotation matrices.)
- Hands-on. Count parameters of the
GPTabove analytically and compare withsum(p.numel() for p in model.parameters()). - Conceptual. For , , , 16-bit cache, compute KV-cache bytes per token and for a 32K context. How does it change with MHA ()?
- Hands-on. Implement a KV-cached
generatefor the minimal GPT and verify that its output equals the uncached version under greedy decoding. - Conceptual. A MoE has 64 experts, top-2 routing, expert FFN size equal to the dense model’s FFN. Roughly how do total and active FFN parameters compare to the dense model?
Solutions (sketch)
- because 2D rotations compose additively in angle; so .
- Bytes/token B = 128 KiB; at 32K tokens = 4 GiB per sequence. MHA multiplies by 4 (16 GiB).
- Total FFN parameters are 64x the dense FFN; active parameters are 2x.
Further reading
Section titled “Further reading”- Vaswani et al., “Attention Is All You Need” (2017).
- Su et al., “RoFormer: Enhanced Transformer with Rotary Position Embedding” (2021).
- Zhang and Sennrich, “Root Mean Square Layer Normalization” (2019); Shazeer, “GLU Variants Improve Transformer” (2020).
- Shazeer, “Fast Transformer Decoding: One Write-Head is All You Need” (2019); Ainslie et al., “GQA” (2023).
- Fedus, Zoph, Shazeer, “Switch Transformers” (2021); DeepSeek-AI, DeepSeek-V2 and V3 technical reports (MLA, fine-grained MoE).
- Dao et al., “FlashAttention” (2022) and FlashAttention-2 (2023).
- Touvron et al., Llama technical reports (2023 onward); Peng et al., “YaRN” (2023).
- Karpathy, nanoGPT and “Let’s build GPT” video.