Skip to content

Inference & Serving

What you’ll learn. Why LLM inference is memory-bandwidth-bound and how that shapes everything; the two phases (prefill and decode); sampling strategies (temperature, top-k, top-p, min-p, beam search, constrained decoding); KV-cache memory arithmetic; the serving techniques that dominate throughput (continuous batching, PagedAttention/vLLM, prefix caching, chunked prefill); speculative decoding; quantization (GPTQ, AWQ, GGUF, FP8); FlashAttention; and how to reason about latency, throughput and cost.

Generation is a loop: run the model on the prompt to get a distribution over the next token, choose one, append it, repeat until an end token or limit.

prompt tokens --[PREFILL: parallel, fills KV cache]--> logits for token 1
token 1 --[DECODE step: 1 token, reads whole cache]--> logits for token 2
...

Two phases with opposite character:

  • Prefill: all prompt tokens are processed in one parallel pass. Large matrix-matrix multiplies, high arithmetic intensity, compute-bound. Determines time-to-first-token (TTFT).
  • Decode: one token per sequence per step. For each step the GPU must read all model weights (and the KV cache) from memory to do very little arithmetic per byte: memory-bandwidth-bound. Determines time-per-output-token (TPOT).

Batch-1 decode of an NN-parameter model reading weights in 16-bit moves 2N2N bytes per token. With memory bandwidth BWBW, an upper bound is

tokens/s ≲ BW2N.\text{tokens/s}\ \lesssim\ \frac{BW}{2N}.

For a 70B model (140 GB of weights) and 3 TB/s of bandwidth that is about 21 tokens/s on a single device that could even hold it (it cannot on most), a hard ceiling no software trick beats for batch 1, except by reducing bytes (quantization, MoE active parameters), reading fewer weights per generated token (speculation), or amortizing the read over many sequences (batching). That last point is the core of serving: at batch size BB the weight read is shared, so throughput rises almost linearly with BB until compute or KV-cache reads become the bottleneck.

The model outputs logits zz; a decoding strategy converts them to a token.

  • Greedy: argmax. Deterministic, often repetitive for open-ended text, good for extraction/classification.
  • Temperature TT: sample from softmax(z/T)\mathrm{softmax}(z/T). T<1T<1 sharpens; T→0T\to0 is greedy; T>1T>1 flattens.
  • Top-k: keep the kk highest-probability tokens, renormalize.
  • Top-p (nucleus): keep the smallest set whose cumulative probability ≥p\ge p (e.g. 0.9). Adapts the candidate set size to model confidence.
  • Min-p: keep tokens with probability at least pmin⁡×pmax⁡p_{\min}\times p_{\max}; scales the cutoff with the top token’s confidence and behaves well at high temperature.
  • Repetition / frequency / presence penalties: down-weight tokens already generated.
  • Beam search: keep the bb best partial sequences by total log-probability. Useful in translation/ASR; for open-ended chat it tends to produce bland, repetitive text, and it is rarely used for LLM assistants.
  • Best-of-nn / majority vote / reranking: sample multiple completions and select by a verifier, reward model, or self-consistency (Reasoning Models).
import torch
def sample(logits, temperature=0.8, top_k=50, top_p=0.95):
logits = logits / max(temperature, 1e-5)
if top_k:
kth = torch.topk(logits, top_k).values[..., -1, None]
logits = logits.masked_fill(logits < kth, float("-inf"))
probs = logits.softmax(-1)
if top_p < 1.0:
sp, si = probs.sort(descending=True)
cum = sp.cumsum(-1)
drop = cum - sp > top_p # tokens beyond the nucleus
sp = sp.masked_fill(drop, 0.0)
probs = torch.zeros_like(probs).scatter(-1, si, sp)
probs = probs / probs.sum(-1, keepdim=True)
return torch.multinomial(probs, 1)

Reproducibility caveat. Even at temperature 0, outputs of hosted models may vary because batch composition changes floating-point reduction order. True determinism needs batch-invariant kernels, which cost throughput.

To guarantee output matches a grammar (JSON schema, regex, a tool-call format), mask logits at each step so only tokens that keep the output valid are allowed, using a finite-state machine or pushdown automaton compiled from the schema (libraries: Outlines, XGrammar, llama.cpp grammars; most serving engines integrate one). This ensures syntactic validity, not semantic correctness, and can slightly distort the distribution. See Prompting for structured outputs from the application side.

For each token, each layer stores a key and value vector per KV head (architecture):

KV bytes per token=2⋅L⋅nkv⋅dh⋅b,\text{KV bytes per token}=2\cdot L\cdot n_{kv}\cdot d_h\cdot b,

with bb bytes per element (2 for FP16/BF16, 1 for FP8/INT8).

Worked example (a 70B-class model with GQA: L=80L=80, nkv=8n_{kv}=8, dh=128d_h=128, BF16):

2⋅80⋅8⋅128⋅2=327,680 bytes≈320 KiB per token.2\cdot80\cdot8\cdot128\cdot2=327{,}680\ \text{bytes}\approx 320\ \text{KiB per token}.

A 8K-token sequence needs about 2.5 GiB, a 128K-token sequence about 40 GiB. With full MHA (64 KV heads) it would be 8x larger. If the weights take 140 GB across devices and you have 200 GB left over for cache, you fit about 600K tokens of total live context: only 75 concurrent 8K requests. Serving capacity is therefore usually limited by KV-cache memory, not compute.

Also budget for activations and a safety margin; engines expose a gpu_memory_utilization parameter.

Static batching waits for a full batch and runs until the longest sequence finishes, wasting slots on completed requests. Continuous batching (Orca, 2022) makes scheduling decisions per decoding step: finished sequences leave immediately and new requests join the batch. This gives multiple-fold throughput improvements over static batching and is standard in every modern engine.

Naively allocating a contiguous max-length KV buffer per request wastes memory through internal fragmentation (reserved but unused) and external fragmentation; the vLLM paper (Kwon et al., 2023) reported 60-80% waste in prior systems. PagedAttention applies operating-system virtual memory: the cache is split into fixed-size blocks (e.g. 16 tokens) allocated on demand, with a per-sequence block table mapping logical to physical blocks. Benefits: near-zero fragmentation, so far larger batches, and sharing: sequences with a common prefix (system prompt, few-shot examples, parallel samples from one prompt, beam candidates) point to the same physical blocks with copy-on-write.

Reuse cached KV for any request whose prompt starts with a previously seen prefix (matched by block hash or a radix tree as in SGLang). Large wins for chat (growing conversation), agents with long fixed system prompts, and RAG with shared documents. Hosted APIs expose this as prompt caching with discounted input prices (Building LLM Applications). Design prompts with stable content first and variable content last to maximize hits.

A long prefill hogs the GPU and stalls decoding for other users, hurting TPOT. Chunked prefill splits long prompts into chunks interleaved with decode steps. Prefill/decode disaggregation goes further: run prefill on one pool of devices and decode on another, transferring the KV cache between them, so each can be provisioned and tuned for its bottleneck (compute vs bandwidth).

Tensor parallelism splits each layer across GPUs in a node (reduces per-token latency and memory per GPU); pipeline parallelism splits layers across nodes; for MoE, expert parallelism places experts on different devices with all-to-all routing. For models that fit on one GPU, replicating instances behind a load balancer is simpler and more efficient.

Decode is bandwidth-bound, so verifying several tokens in one forward pass costs almost the same as generating one. Speculative decoding (Leviathan et al.; Chen et al., 2023):

  1. A cheap draft mechanism proposes γ\gamma tokens (a small model, or heads attached to the big model such as Medusa/EAGLE-style or multi-token-prediction heads, or n-gram lookup from the prompt).
  2. The target model scores all γ\gamma positions in a single parallel pass.
  3. Accept the longest prefix consistent with the target’s distribution using a rejection-sampling rule: accept draft token xx with probability min⁡(1,p(x)/q(x))\min(1, p(x)/q(x)); on rejection, resample from the normalized residual max⁡(0,p−q)\max(0,p-q). This provably preserves the target model’s output distribution exactly.

Expected speedup depends on the acceptance rate α\alpha; with γ\gamma proposals the expected tokens per target pass is 1−αγ+11−α\frac{1-\alpha^{\gamma+1}}{1-\alpha}. Typical speedups are 2-3x at low batch sizes, falling as batch size grows (the system becomes compute-bound and verification is no longer free). Most effective on predictable text (code, structured output).

Quantization reduces precision of weights (and optionally activations and KV cache) to cut memory and bandwidth, the dominant decode cost.

Basic idea: map floating weights to integers via q=round(w/s)q=\mathrm{round}(w/s) with scale ss (per-tensor, per-channel or per-group of e.g. 128 weights), and dequantize on the fly. Group-wise scales limit the damage from outliers.

  • Weight-only INT4/INT8 (W4A16, W8A16): activations stay 16-bit; the win is bytes moved. 4-bit cuts weight reads 4x vs FP16, with noticeable speedup in decode.
  • GPTQ (Frantar et al., 2022): post-training, layer-by-layer quantization that uses approximate second-order (Hessian) information from a small calibration set to adjust remaining weights as each is rounded, minimizing layer output error.
  • AWQ (Lin et al., 2023): observes that a small fraction of weight channels (those multiplied by large activations) are salient; scales them up before quantization to protect them. No backprop, fast, robust.
  • GGUF / llama.cpp “k-quants”: a file format and family of block-wise quantization schemes (e.g. Q4_K_M, Q5_K_M, Q8_0) built for CPU and consumer-GPU inference, with mixed precision across tensors. Standard for local use (llama.cpp, Ollama).
  • FP8 and W8A8: quantize both weights and activations to 8 bits (FP8 on recent GPUs, or INT8 with SmoothQuant-style outlier migration) so matmuls run in low precision natively: speeds prefill as well as decode. 4-bit floating-point formats with block scaling (such as MXFP4/NVFP4) are increasingly supported on the newest hardware.
  • KV-cache quantization (FP8/INT8/INT4) multiplies concurrency at a quality cost that must be checked for long contexts.

Rules of thumb: 8-bit is nearly lossless; 4-bit weight-only loses a little, more for small models and for reasoning/code/long-context tasks; below 4 bits degrades quickly without quantization-aware training. Larger models tolerate quantization better. Always evaluate the quantized model on your own tasks; perplexity alone understates damage to reasoning.

Standard attention materializes the T×TT\times T score matrix in GPU high-bandwidth memory (HBM), so it is dominated by memory traffic. FlashAttention (Dao et al., 2022) computes exact attention in tiles in fast on-chip SRAM, using an online softmax (running max and sum) so the full matrix is never written to HBM. Result: memory linear in TT, and 2-4x faster. FlashAttention-2/3 improve parallelism and exploit newer hardware features. For decoding, “flash-decoding” splits the KV sequence across thread blocks to keep the GPU busy at batch size 1. Use it via torch.nn.functional.scaled_dot_product_attention or engine defaults.

  • TTFT (queueing + prefill), TPOT/ITL (inter-token latency), end-to-end latency ≈TTFT+nout⋅TPOT\approx \text{TTFT}+n_{out}\cdot\text{TPOT}.
  • Throughput (tokens/s per GPU) versus latency is a trade-off curve controlled mostly by batch size: larger batches mean higher throughput and cost efficiency but higher per-request latency.
  • Cost per million tokens ≈GPU cost per hour3600⋅tokens/s×106\approx \dfrac{\text{GPU cost per hour}}{3600\cdot\text{tokens/s}}\times10^6. Output tokens are more expensive than input tokens (decode is bandwidth-bound, prefill is efficient), which is why APIs price them that way.
  • Set SLOs (e.g. p95 TTFT < 1 s, TPOT < 50 ms) and load-test with realistic length distributions.
Terminal window
# an OpenAI-compatible server with vLLM (flags evolve; check the docs for your version)
vllm serve your-org/your-model \
--max-model-len 16384 \
--gpu-memory-utilization 0.90 \
--enable-prefix-caching \
--tensor-parallel-size 1
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
r = client.chat.completions.create(
model="your-org/your-model",
messages=[{"role": "user", "content": "Explain PagedAttention in two sentences."}],
max_tokens=128, temperature=0.2,
)
print(r.choices[0].message.content)

Common engines: vLLM, SGLang, TensorRT-LLM, Hugging Face TGI, llama.cpp (local/CPU), and managed endpoints. The landscape shifts fast; the underlying ideas above are stable.

  • Decode is memory-bandwidth-bound: tokens/s is roughly bandwidth divided by bytes read per token; batching amortizes weight reads.
  • KV cache costs 2Lnkvdhb2Ln_{kv}d_hb bytes/token and typically limits concurrency; GQA/MLA, cache quantization and paging are the remedies.
  • Continuous batching, PagedAttention and prefix caching are the pillars of high-throughput serving; chunked prefill and disaggregation fix prefill/decode interference.
  • Speculative decoding speeds up low-batch decoding without changing the output distribution.
  • Quantization (AWQ/GPTQ/GGUF/FP8) trades small quality loss for large memory and speed gains; evaluate on your tasks.
  • FlashAttention makes exact attention IO-efficient with online softmax.
  1. Hands-on. Compute KV bytes per token for a model with L=32L=32, nkv=8n_{kv}=8, dh=128d_h=128 in BF16 and FP8. How many 4K-token sequences fit in 20 GB of free memory?
  2. Conceptual. Explain why throughput grows with batch size but per-request latency also grows, and what limits the benefit.
  3. Conceptual. Prove that for speculative decoding, accepting with probability min⁡(1,p/q)\min(1,p/q) and resampling from the normalized max⁡(0,p−q)\max(0,p-q) yields samples exactly from pp.
  4. Hands-on. Implement a KV-cached decoding loop for the GPT in the architecture lecture and measure speedup versus recomputing the full prefix for 500 tokens.
  5. Hands-on. Quantize a small model with llama.cpp (Q4_K_M versus Q8_0) or AutoAWQ, and compare perplexity and a task metric against BF16.
Solutions (sketch)
  1. BF16: 2⋅32⋅8⋅128⋅2=131,0722\cdot32\cdot8\cdot128\cdot2=131{,}072 B per token; FP8: 65,536 B. A 4K sequence is 0.5 GiB in BF16, so about 40 sequences in 20 GB; FP8 gives about 80 (ignoring overhead).
  2. Weight reads amortize over more sequences, so tokens/s per GPU rises; but each step takes longer (more KV reads, more compute) and requests wait to be admitted. Limits: KV memory, then compute saturation, then attention’s KV bandwidth.
  3. P(emit x)=q(x)min⁡(1,p/q)+P(reject)max⁡(0,p(x)−q(x))∑max⁡(0,p−q)P(\text{emit }x)=q(x)\min(1,p/q)+P(\text{reject})\frac{\max(0,p(x)-q(x))}{\sum\max(0,p-q)}; the first term is min⁡(q(x),p(x))\min(q(x),p(x)) and P(reject)=∑xmax⁡(0,q−p)=∑xmax⁡(0,p−q)P(\text{reject})=\sum_x\max(0,q-p)=\sum_x\max(0,p-q), so the second term is max⁡(0,p(x)−q(x))\max(0,p(x)-q(x)); the sum is p(x)p(x).
  • Kwon et al., “Efficient Memory Management for Large Language Model Serving with PagedAttention” (vLLM, 2023).
  • Yu et al., “Orca: A Distributed Serving System for Transformer-Based Generative Models” (2022).
  • Dao et al., “FlashAttention” (2022), FlashAttention-2 (2023); Milakov and Gimelshein, “Online normalizer calculation for softmax” (2018).
  • Leviathan, Kalman, Matias, “Fast Inference from Transformers via Speculative Decoding” (2023); Chen et al., “Accelerating LLM Decoding with Speculative Sampling” (2023).
  • Frantar et al., “GPTQ” (2022); Lin et al., “AWQ” (2023); Xiao et al., “SmoothQuant” (2022); Dettmers et al., “LLM.int8()” (2022).
  • Holtzman et al., “The Curious Case of Neural Text Degeneration” (nucleus sampling, 2019).
  • Zheng et al., “SGLang” (2023); Zhong et al., “DistServe” (2024); Pope et al., “Efficiently Scaling Transformer Inference” (2022).
  • Chip Huyen, AI Engineering (2025), inference optimization chapters; vLLM and llama.cpp documentation.