Skip to content

LLM Architecture in Depth

What you’ll learn. The architecture of a contemporary decoder-only LLM and why each component differs from the 2017 original transformer: pre-norm with RMSNorm, rotary position embeddings, SwiGLU feed-forward blocks, grouped-query and latent attention, mixture-of-experts, and the KV cache. You’ll see how context is extended beyond training length, do parameter accounting, and write a compact working GPT. This assumes you know basic self-attention from Attention and Transformers.

A decoder-only LLM maps token IDs to next-token logits:

ids -> embedding -> [ Block x L ] -> final norm -> unembedding (lm head) -> logits
Block: x = x + Attention(Norm(x)) # causal self-attention
x = x + FFN(Norm(x)) # per-token MLP

Key differences from the original encoder-decoder transformer:

  • Causal masking only: token tt attends to positions ≤t\le t, so one forward pass trains on all TT next-token predictions in parallel (teacher forcing).
  • Pre-norm: normalization applied before each sublayer, with a clean residual stream. This trains stably at depth without elaborate warmup, unlike the original post-norm.
  • No cross-attention and no encoder. Instruction and prompt are simply earlier tokens in the same sequence.

Why decoder-only won: one objective, one stack, efficient pretraining, a natural fit for generation, and in-context prompting subsumes the encoder’s role.

LayerNorm subtracts the mean and divides by standard deviation. RMSNorm (Zhang and Sennrich, 2019) drops the mean-centering and the bias:

RMSNorm(x)=x1d∑ixi2+ϵ⊙g.\mathrm{RMSNorm}(\mathbf x) = \frac{\mathbf x}{\sqrt{\frac1d\sum_i x_i^2+\epsilon}}\odot \mathbf g.

It is cheaper and works as well in practice. Some recent models add QK-norm (normalizing queries and keys before the dot product) to prevent attention-logit blowup at scale.

Attention is permutation-invariant, so position must be injected. Early models added learned or sinusoidal vectors to the input. RoPE (Su et al., 2021) instead rotates queries and keys by an angle proportional to position, so that the dot product depends only on the relative offset.

Split the head dimension dd into d/2d/2 pairs. For position mm and pair ii, rotate by angle mθim\theta_i with θi=b−2i/d\theta_i = b^{-2i/d} (base bb typically 10,000 or larger). For a 2D pair:

(q2i′q2i+1′)=(cos⁡mθi−sin⁡mθisin⁡mθicos⁡mθi)(q2iq2i+1).\begin{pmatrix} q'_{2i}\\ q'_{2i+1}\end{pmatrix}= \begin{pmatrix}\cos m\theta_i & -\sin m\theta_i\\ \sin m\theta_i & \cos m\theta_i\end{pmatrix} \begin{pmatrix} q_{2i}\\ q_{2i+1}\end{pmatrix}.

Because rotations compose, ⟨Rmq,Rnk⟩=⟨q,Rn−mk⟩\langle R_m q, R_n k\rangle = \langle q, R_{n-m}k\rangle: the score depends on n−mn-m only. Low-index pairs rotate fast (encode local order), high-index pairs rotate slowly (coarse, long-range position). RoPE is applied to Q and K, not V, and no extra parameters are needed.

import torch
def rope_cache(T, d, base=10000.0, device=None):
inv = 1.0 / (base ** (torch.arange(0, d, 2, device=device).float() / d))
ang = torch.outer(torch.arange(T, device=device).float(), inv) # (T, d/2)
return ang.cos(), ang.sin()
def apply_rope(x, cos, sin): # x: (B, H, T, d)
x1, x2 = x[..., 0::2], x[..., 1::2]
cos, sin = cos[None, None], sin[None, None]
out = torch.stack((x1 * cos - x2 * sin, x1 * sin + x2 * cos), dim=-1)
return out.flatten(-2)

The classic FFN is W2 ϕ(W1x)W_2\,\phi(W_1 x) with hidden width 4d4d. Gated variants (Shazeer, 2020) multiply two projections:

SwiGLU(x)=W2(SiLU(W1x)⊙W3x),SiLU(z)=z σ(z).\mathrm{SwiGLU}(x) = W_2\big(\mathrm{SiLU}(W_1 x)\odot W_3 x\big),\qquad \mathrm{SiLU}(z)=z\,\sigma(z).

It has three matrices instead of two, so the hidden width is reduced (about 83d\tfrac{8}{3}d, rounded to a hardware-friendly multiple) to keep parameter count comparable. It consistently gives better loss for the same compute and is now standard.

Standard multi-head attention (MHA) has HH query heads and HH key/value heads. During generation, K and V of all previous tokens are cached (see below), and that cache dominates memory and bandwidth.

  • Multi-query attention (MQA): all query heads share one K/V head. Cache shrinks by H×H\times; some quality loss.
  • Grouped-query attention (GQA): query heads are split into GG groups, each sharing one K/V head. G=HG=H is MHA, G=1G=1 is MQA. A typical setting (e.g. 32 query heads, 8 KV heads) cuts the cache 4x with negligible quality loss; GQA is the de facto standard in open models.
  • Multi-head latent attention (MLA), introduced in the DeepSeek-V2 line: project keys and values jointly into a small latent vector per token, cache only that latent, and reconstruct K/V (or fold the up-projection into the query/output matrices) at use time. It achieves a smaller cache than GQA while keeping the expressiveness of many heads. RoPE needs special handling here (a decoupled rotary component) since position rotation does not commute with the low-rank projection.
  • Sliding-window / local attention: some layers attend only to the last ww tokens (cache bounded), often interleaved with full-attention layers.

Dense FFNs use all parameters for every token. An MoE layer replaces the FFN with EE expert FFNs and a small router; each token is sent to its top-kk experts (commonly k=1k=1 to 8) and the outputs are combined with the router’s weights:

y=∑e∈TopK(g(x))ge(x) FFNe(x),g(x)=softmax(Wrx).y = \sum_{e\in \mathrm{TopK}(g(x))} g_e(x)\, \mathrm{FFN}_e(x),\quad g(x)=\mathrm{softmax}(W_r x).

Total parameters grow with EE, but active parameters per token only with kk. A model quoted as, say, 600B total / 40B active has the memory footprint of the former and roughly the per-token FLOPs of the latter. This is a favourable trade in training compute terms, but serving needs all experts resident in memory and requires inter-device communication (expert parallelism).

Practical issues:

  • Load balancing: without help the router collapses onto a few experts. Classic remedies are an auxiliary balancing loss; some recent designs use bias-based, loss-free balancing.
  • Shared experts: a few experts always active capture common knowledge, leaving routed experts to specialize; fine-grained (many small) experts improve specialization.
  • Dropped tokens and capacity: each expert has a capacity limit in batched training.
  • Instability and fine-tuning difficulty compared with dense models.

During generation at step tt, token tt‘s query attends to keys/values of all tokens ≤t\le t. K and V of earlier tokens never change (causal mask), so recomputing them is wasteful. The KV cache stores them per layer and per head:

prefill: run the whole prompt in parallel; fill cache (compute-bound)
decode: per new token: compute q,k,v for that token only, append k,v,
attend over the cache (memory-bandwidth-bound)

It turns per-token decode cost from O(t)O(t) projections to O(1)O(1) projections plus an O(t)O(t) attention read. The price is memory growing linearly with context and batch. Many serving optimizations exist solely to manage it (Inference & Serving).

Attention costs O(T2)O(T^2) compute and the cache O(T)O(T) memory. To work at 100K to 1M+ tokens:

  1. Position extrapolation / interpolation. A model trained at 4K or 8K degrades past its trained length because rotation angles go out of distribution. Position interpolation rescales positions so longer inputs map into the trained range. NTK-aware scaling and YaRN scale RoPE frequencies non-uniformly (preserve high-frequency local detail, stretch low frequencies). Increasing the RoPE base for continued pretraining at longer lengths is a common recipe.
  2. Staged length training. Pretrain mostly at short context, then run a shorter phase at progressively longer lengths on long-document data, because long-sequence training is expensive.
  3. Efficient attention kernels. FlashAttention computes exact attention without materializing the T×TT\times T matrix, making memory O(T)O(T) and speed IO-optimal. Ring/sequence parallelism splits the sequence across devices.
  4. Hybrid local/global layers and sparse attention, e.g. mostly sliding-window layers with periodic global layers.
  5. Cache compression: quantizing the cache (8- or 4-bit), eviction heuristics, or latent compression (MLA).
  6. Alternatives to attention such as state-space and hybrid models; see Reasoning, Multimodality & Frontier Trends.

For a dense model with LL layers, width dd, nhn_h query heads, nkvn_{kv} KV heads, head dim dh=d/nhd_h=d/n_h, FFN hidden width ff, vocabulary VV:

  • Attention: WQW_Q: d⋅nhdhd\cdot n_h d_h; WK,WVW_K,W_V: d⋅nkvdhd\cdot n_{kv}d_h each; WOW_O: nhdh⋅dn_h d_h\cdot d.
  • SwiGLU FFN: 3df3 d f.
  • Norms: negligible. Embeddings: VdVd (plus another VdVd if untied).

Per layer roughly (2+2 nkv/nh)d2+3df\big(2 + 2\,n_{kv}/n_h\big)d^2 + 3df. With f≈3.5df\approx 3.5d and GQA this is about 12.5d212.5 d^2, hence the rule of thumb N≈12Ld2N \approx 12 L d^2 for non-embedding parameters. Training compute is about 6ND6ND FLOPs for DD tokens (Pretraining).

A compact modern-style decoder (RMSNorm, RoPE, GQA, SwiGLU, pre-norm, tied embeddings) that you can train on a character-level corpus:

import math, torch, torch.nn as nn, torch.nn.functional as F
class RMSNorm(nn.Module):
def __init__(self, d, eps=1e-6):
super().__init__(); self.eps = eps; self.g = nn.Parameter(torch.ones(d))
def forward(self, x):
return x * torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + self.eps) * self.g
def rope_cache(T, dh, base=10000.0):
inv = 1.0 / (base ** (torch.arange(0, dh, 2).float() / dh))
ang = torch.outer(torch.arange(T).float(), inv)
return ang.cos(), ang.sin()
def apply_rope(x, cos, sin): # x: (B,H,T,dh)
x1, x2 = x[..., 0::2], x[..., 1::2]
c, s = cos[None, None, :x.size(2)], sin[None, None, :x.size(2)]
return torch.stack((x1 * c - x2 * s, x1 * s + x2 * c), -1).flatten(-2)
class Attention(nn.Module):
def __init__(self, d, n_h, n_kv):
super().__init__()
self.n_h, self.n_kv, self.dh = n_h, n_kv, d // n_h
self.q = nn.Linear(d, n_h * self.dh, bias=False)
self.k = nn.Linear(d, n_kv * self.dh, bias=False)
self.v = nn.Linear(d, n_kv * self.dh, bias=False)
self.o = nn.Linear(n_h * self.dh, d, bias=False)
def forward(self, x, cos, sin):
B, T, _ = x.shape
q = self.q(x).view(B, T, self.n_h, self.dh).transpose(1, 2)
k = self.k(x).view(B, T, self.n_kv, self.dh).transpose(1, 2)
v = self.v(x).view(B, T, self.n_kv, self.dh).transpose(1, 2)
q, k = apply_rope(q, cos, sin), apply_rope(k, cos, sin)
rep = self.n_h // self.n_kv # GQA: share each KV head across `rep` query heads
k, v = k.repeat_interleave(rep, 1), v.repeat_interleave(rep, 1)
y = F.scaled_dot_product_attention(q, k, v, is_causal=True)
return self.o(y.transpose(1, 2).reshape(B, T, -1))
class SwiGLU(nn.Module):
def __init__(self, d, f):
super().__init__()
self.w1 = nn.Linear(d, f, bias=False); self.w3 = nn.Linear(d, f, bias=False)
self.w2 = nn.Linear(f, d, bias=False)
def forward(self, x):
return self.w2(F.silu(self.w1(x)) * self.w3(x))
class Block(nn.Module):
def __init__(self, d, n_h, n_kv, f):
super().__init__()
self.n1, self.n2 = RMSNorm(d), RMSNorm(d)
self.attn, self.ffn = Attention(d, n_h, n_kv), SwiGLU(d, f)
def forward(self, x, cos, sin):
x = x + self.attn(self.n1(x), cos, sin)
return x + self.ffn(self.n2(x))
class GPT(nn.Module):
def __init__(self, V, d=256, L=6, n_h=8, n_kv=2, f=704, max_T=1024):
super().__init__()
self.emb = nn.Embedding(V, d)
self.blocks = nn.ModuleList(Block(d, n_h, n_kv, f) for _ in range(L))
self.norm = RMSNorm(d)
self.head = nn.Linear(d, V, bias=False)
self.head.weight = self.emb.weight # tie embeddings
cos, sin = rope_cache(max_T, d // n_h)
self.register_buffer("cos", cos, persistent=False)
self.register_buffer("sin", sin, persistent=False)
self.apply(self._init)
def _init(self, m):
if isinstance(m, (nn.Linear, nn.Embedding)):
nn.init.normal_(m.weight, std=0.02)
def forward(self, ids, targets=None):
x = self.emb(ids)
for b in self.blocks: x = b(x, self.cos, self.sin)
logits = self.head(self.norm(x))
loss = None if targets is None else F.cross_entropy(
logits.view(-1, logits.size(-1)), targets.view(-1))
return logits, loss
@torch.no_grad()
def generate(self, ids, n, temperature=1.0):
for _ in range(n):
logits, _ = self(ids[:, -1024:]) # no KV cache here: clarity over speed
probs = F.softmax(logits[:, -1] / temperature, -1)
ids = torch.cat([ids, torch.multinomial(probs, 1)], 1)
return ids
# training loop sketch on a character corpus `data` (1D LongTensor of ids)
def get_batch(data, B=32, T=128):
ix = torch.randint(len(data) - T - 1, (B,))
x = torch.stack([data[i:i + T] for i in ix])
y = torch.stack([data[i + 1:i + T + 1] for i in ix])
return x, y
# model = GPT(V=vocab_size); opt = torch.optim.AdamW(model.parameters(), lr=3e-4, weight_decay=0.1)
# for step in range(5000):
# x, y = get_batch(data); _, loss = model(x, y)
# opt.zero_grad(set_to_none=True); loss.backward()
# torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0); opt.step()

Notes: generate recomputes the full prefix each step (quadratic). A production version caches K/V per layer, as sketched above. Weight init for deep models often scales output projections by 1/2L1/\sqrt{2L}.

  • No biases in linear layers (simplifies, no loss in quality).
  • Tied vs untied embeddings: tying saves parameters in small models; large models usually untie.
  • Parallel attention + FFN (compute both from the same normed input) saves latency in some designs.
  • Multi-token prediction heads (predicting several future tokens at training time) improve data efficiency and enable speculative decoding at inference.
  • Logit soft-capping and z-loss stabilize training at scale by bounding logits.
  • The standard modern recipe: pre-norm RMSNorm, RoPE on Q/K, SwiGLU FFN, no biases, and GQA (or MLA) for a small KV cache.
  • MoE decouples total parameters from per-token compute, at the cost of memory, routing complexity and communication.
  • The KV cache makes decoding cheap in FLOPs but expensive in memory; nkvn_{kv} and latent compression are the main levers.
  • Long context combines RoPE rescaling, staged training, exact memory-efficient kernels, and hybrid/local attention; advertised windows overstate usable context.
  • Non-embedding parameters scale as about 12Ld212Ld^2; training FLOPs as about 6ND6ND.
  1. Conceptual. Show that RoPE attention scores depend only on relative position. (Hint: rotation matrices.)
  2. Hands-on. Count parameters of the GPT above analytically and compare with sum(p.numel() for p in model.parameters()).
  3. Conceptual. For L=32L=32, nkv=8n_{kv}=8, dh=128d_h=128, 16-bit cache, compute KV-cache bytes per token and for a 32K context. How does it change with MHA (nkv=32n_{kv}=32)?
  4. Hands-on. Implement a KV-cached generate for the minimal GPT and verify that its output equals the uncached version under greedy decoding.
  5. Conceptual. A MoE has 64 experts, top-2 routing, expert FFN size equal to the dense model’s FFN. Roughly how do total and active FFN parameters compare to the dense model?
Solutions (sketch)
  1. Rm⊤Rn=Rn−mR_m^\top R_n = R_{n-m} because 2D rotations compose additively in angle; so (Rmq)⊤(Rnk)=q⊤Rn−mk(R_mq)^\top(R_nk)=q^\top R_{n-m}k.
  2. Bytes/token =2⋅32⋅8⋅128⋅2=131,072=2\cdot32\cdot8\cdot128\cdot2=131{,}072 B = 128 KiB; at 32K tokens = 4 GiB per sequence. MHA multiplies by 4 (16 GiB).
  3. Total FFN parameters are 64x the dense FFN; active parameters are 2x.
  • Vaswani et al., “Attention Is All You Need” (2017).
  • Su et al., “RoFormer: Enhanced Transformer with Rotary Position Embedding” (2021).
  • Zhang and Sennrich, “Root Mean Square Layer Normalization” (2019); Shazeer, “GLU Variants Improve Transformer” (2020).
  • Shazeer, “Fast Transformer Decoding: One Write-Head is All You Need” (2019); Ainslie et al., “GQA” (2023).
  • Fedus, Zoph, Shazeer, “Switch Transformers” (2021); DeepSeek-AI, DeepSeek-V2 and V3 technical reports (MLA, fine-grained MoE).
  • Dao et al., “FlashAttention” (2022) and FlashAttention-2 (2023).
  • Touvron et al., Llama technical reports (2023 onward); Peng et al., “YaRN” (2023).
  • Karpathy, nanoGPT and “Let’s build GPT” video.