Inference & Serving
What you’ll learn. Why LLM inference is memory-bandwidth-bound and how that shapes everything; the two phases (prefill and decode); sampling strategies (temperature, top-k, top-p, min-p, beam search, constrained decoding); KV-cache memory arithmetic; the serving techniques that dominate throughput (continuous batching, PagedAttention/vLLM, prefix caching, chunked prefill); speculative decoding; quantization (GPTQ, AWQ, GGUF, FP8); FlashAttention; and how to reason about latency, throughput and cost.
Autoregressive decoding
Section titled “Autoregressive decoding”Generation is a loop: run the model on the prompt to get a distribution over the next token, choose one, append it, repeat until an end token or limit.
prompt tokens --[PREFILL: parallel, fills KV cache]--> logits for token 1token 1 --[DECODE step: 1 token, reads whole cache]--> logits for token 2...Two phases with opposite character:
- Prefill: all prompt tokens are processed in one parallel pass. Large matrix-matrix multiplies, high arithmetic intensity, compute-bound. Determines time-to-first-token (TTFT).
- Decode: one token per sequence per step. For each step the GPU must read all model weights (and the KV cache) from memory to do very little arithmetic per byte: memory-bandwidth-bound. Determines time-per-output-token (TPOT).
The roofline intuition
Section titled “The roofline intuition”Batch-1 decode of an -parameter model reading weights in 16-bit moves bytes per token. With memory bandwidth , an upper bound is
For a 70B model (140 GB of weights) and 3 TB/s of bandwidth that is about 21 tokens/s on a single device that could even hold it (it cannot on most), a hard ceiling no software trick beats for batch 1, except by reducing bytes (quantization, MoE active parameters), reading fewer weights per generated token (speculation), or amortizing the read over many sequences (batching). That last point is the core of serving: at batch size the weight read is shared, so throughput rises almost linearly with until compute or KV-cache reads become the bottleneck.
Sampling
Section titled “Sampling”The model outputs logits ; a decoding strategy converts them to a token.
- Greedy: argmax. Deterministic, often repetitive for open-ended text, good for extraction/classification.
- Temperature : sample from . sharpens; is greedy; flattens.
- Top-k: keep the highest-probability tokens, renormalize.
- Top-p (nucleus): keep the smallest set whose cumulative probability (e.g. 0.9). Adapts the candidate set size to model confidence.
- Min-p: keep tokens with probability at least ; scales the cutoff with the top token’s confidence and behaves well at high temperature.
- Repetition / frequency / presence penalties: down-weight tokens already generated.
- Beam search: keep the best partial sequences by total log-probability. Useful in translation/ASR; for open-ended chat it tends to produce bland, repetitive text, and it is rarely used for LLM assistants.
- Best-of- / majority vote / reranking: sample multiple completions and select by a verifier, reward model, or self-consistency (Reasoning Models).
import torch
def sample(logits, temperature=0.8, top_k=50, top_p=0.95): logits = logits / max(temperature, 1e-5) if top_k: kth = torch.topk(logits, top_k).values[..., -1, None] logits = logits.masked_fill(logits < kth, float("-inf")) probs = logits.softmax(-1) if top_p < 1.0: sp, si = probs.sort(descending=True) cum = sp.cumsum(-1) drop = cum - sp > top_p # tokens beyond the nucleus sp = sp.masked_fill(drop, 0.0) probs = torch.zeros_like(probs).scatter(-1, si, sp) probs = probs / probs.sum(-1, keepdim=True) return torch.multinomial(probs, 1)Reproducibility caveat. Even at temperature 0, outputs of hosted models may vary because batch composition changes floating-point reduction order. True determinism needs batch-invariant kernels, which cost throughput.
Constrained / structured decoding
Section titled “Constrained / structured decoding”To guarantee output matches a grammar (JSON schema, regex, a tool-call format), mask logits at each step so only tokens that keep the output valid are allowed, using a finite-state machine or pushdown automaton compiled from the schema (libraries: Outlines, XGrammar, llama.cpp grammars; most serving engines integrate one). This ensures syntactic validity, not semantic correctness, and can slightly distort the distribution. See Prompting for structured outputs from the application side.
KV-cache memory math
Section titled “KV-cache memory math”For each token, each layer stores a key and value vector per KV head (architecture):
with bytes per element (2 for FP16/BF16, 1 for FP8/INT8).
Worked example (a 70B-class model with GQA: , , , BF16):
A 8K-token sequence needs about 2.5 GiB, a 128K-token sequence about 40 GiB. With full MHA (64 KV heads) it would be 8x larger. If the weights take 140 GB across devices and you have 200 GB left over for cache, you fit about 600K tokens of total live context: only 75 concurrent 8K requests. Serving capacity is therefore usually limited by KV-cache memory, not compute.
Also budget for activations and a safety margin; engines expose a gpu_memory_utilization parameter.
Serving systems techniques
Section titled “Serving systems techniques”Continuous (in-flight) batching
Section titled “Continuous (in-flight) batching”Static batching waits for a full batch and runs until the longest sequence finishes, wasting slots on completed requests. Continuous batching (Orca, 2022) makes scheduling decisions per decoding step: finished sequences leave immediately and new requests join the batch. This gives multiple-fold throughput improvements over static batching and is standard in every modern engine.
PagedAttention and vLLM
Section titled “PagedAttention and vLLM”Naively allocating a contiguous max-length KV buffer per request wastes memory through internal fragmentation (reserved but unused) and external fragmentation; the vLLM paper (Kwon et al., 2023) reported 60-80% waste in prior systems. PagedAttention applies operating-system virtual memory: the cache is split into fixed-size blocks (e.g. 16 tokens) allocated on demand, with a per-sequence block table mapping logical to physical blocks. Benefits: near-zero fragmentation, so far larger batches, and sharing: sequences with a common prefix (system prompt, few-shot examples, parallel samples from one prompt, beam candidates) point to the same physical blocks with copy-on-write.
Prefix caching
Section titled “Prefix caching”Reuse cached KV for any request whose prompt starts with a previously seen prefix (matched by block hash or a radix tree as in SGLang). Large wins for chat (growing conversation), agents with long fixed system prompts, and RAG with shared documents. Hosted APIs expose this as prompt caching with discounted input prices (Building LLM Applications). Design prompts with stable content first and variable content last to maximize hits.
Chunked prefill and disaggregation
Section titled “Chunked prefill and disaggregation”A long prefill hogs the GPU and stalls decoding for other users, hurting TPOT. Chunked prefill splits long prompts into chunks interleaved with decode steps. Prefill/decode disaggregation goes further: run prefill on one pool of devices and decode on another, transferring the KV cache between them, so each can be provisioned and tuned for its bottleneck (compute vs bandwidth).
Parallelism at inference
Section titled “Parallelism at inference”Tensor parallelism splits each layer across GPUs in a node (reduces per-token latency and memory per GPU); pipeline parallelism splits layers across nodes; for MoE, expert parallelism places experts on different devices with all-to-all routing. For models that fit on one GPU, replicating instances behind a load balancer is simpler and more efficient.
Speculative decoding
Section titled “Speculative decoding”Decode is bandwidth-bound, so verifying several tokens in one forward pass costs almost the same as generating one. Speculative decoding (Leviathan et al.; Chen et al., 2023):
- A cheap draft mechanism proposes tokens (a small model, or heads attached to the big model such as Medusa/EAGLE-style or multi-token-prediction heads, or n-gram lookup from the prompt).
- The target model scores all positions in a single parallel pass.
- Accept the longest prefix consistent with the target’s distribution using a rejection-sampling rule: accept draft token with probability ; on rejection, resample from the normalized residual . This provably preserves the target model’s output distribution exactly.
Expected speedup depends on the acceptance rate ; with proposals the expected tokens per target pass is . Typical speedups are 2-3x at low batch sizes, falling as batch size grows (the system becomes compute-bound and verification is no longer free). Most effective on predictable text (code, structured output).
Quantization
Section titled “Quantization”Quantization reduces precision of weights (and optionally activations and KV cache) to cut memory and bandwidth, the dominant decode cost.
Basic idea: map floating weights to integers via with scale (per-tensor, per-channel or per-group of e.g. 128 weights), and dequantize on the fly. Group-wise scales limit the damage from outliers.
- Weight-only INT4/INT8 (W4A16, W8A16): activations stay 16-bit; the win is bytes moved. 4-bit cuts weight reads 4x vs FP16, with noticeable speedup in decode.
- GPTQ (Frantar et al., 2022): post-training, layer-by-layer quantization that uses approximate second-order (Hessian) information from a small calibration set to adjust remaining weights as each is rounded, minimizing layer output error.
- AWQ (Lin et al., 2023): observes that a small fraction of weight channels (those multiplied by large activations) are salient; scales them up before quantization to protect them. No backprop, fast, robust.
- GGUF / llama.cpp “k-quants”: a file format and family of block-wise quantization schemes (e.g. Q4_K_M, Q5_K_M, Q8_0) built for CPU and consumer-GPU inference, with mixed precision across tensors. Standard for local use (llama.cpp, Ollama).
- FP8 and W8A8: quantize both weights and activations to 8 bits (FP8 on recent GPUs, or INT8 with SmoothQuant-style outlier migration) so matmuls run in low precision natively: speeds prefill as well as decode. 4-bit floating-point formats with block scaling (such as MXFP4/NVFP4) are increasingly supported on the newest hardware.
- KV-cache quantization (FP8/INT8/INT4) multiplies concurrency at a quality cost that must be checked for long contexts.
Rules of thumb: 8-bit is nearly lossless; 4-bit weight-only loses a little, more for small models and for reasoning/code/long-context tasks; below 4 bits degrades quickly without quantization-aware training. Larger models tolerate quantization better. Always evaluate the quantized model on your own tasks; perplexity alone understates damage to reasoning.
FlashAttention
Section titled “FlashAttention”Standard attention materializes the score matrix in GPU high-bandwidth memory (HBM), so it is dominated by memory traffic. FlashAttention (Dao et al., 2022) computes exact attention in tiles in fast on-chip SRAM, using an online softmax (running max and sum) so the full matrix is never written to HBM. Result: memory linear in , and 2-4x faster. FlashAttention-2/3 improve parallelism and exploit newer hardware features. For decoding, “flash-decoding” splits the KV sequence across thread blocks to keep the GPU busy at batch size 1. Use it via torch.nn.functional.scaled_dot_product_attention or engine defaults.
Metrics and capacity planning
Section titled “Metrics and capacity planning”- TTFT (queueing + prefill), TPOT/ITL (inter-token latency), end-to-end latency .
- Throughput (tokens/s per GPU) versus latency is a trade-off curve controlled mostly by batch size: larger batches mean higher throughput and cost efficiency but higher per-request latency.
- Cost per million tokens . Output tokens are more expensive than input tokens (decode is bandwidth-bound, prefill is efficient), which is why APIs price them that way.
- Set SLOs (e.g. p95 TTFT < 1 s, TPOT < 50 ms) and load-test with realistic length distributions.
Running a server
Section titled “Running a server”# an OpenAI-compatible server with vLLM (flags evolve; check the docs for your version)vllm serve your-org/your-model \ --max-model-len 16384 \ --gpu-memory-utilization 0.90 \ --enable-prefix-caching \ --tensor-parallel-size 1from openai import OpenAIclient = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")r = client.chat.completions.create( model="your-org/your-model", messages=[{"role": "user", "content": "Explain PagedAttention in two sentences."}], max_tokens=128, temperature=0.2,)print(r.choices[0].message.content)Common engines: vLLM, SGLang, TensorRT-LLM, Hugging Face TGI, llama.cpp (local/CPU), and managed endpoints. The landscape shifts fast; the underlying ideas above are stable.
Key takeaways
Section titled “Key takeaways”- Decode is memory-bandwidth-bound: tokens/s is roughly bandwidth divided by bytes read per token; batching amortizes weight reads.
- KV cache costs bytes/token and typically limits concurrency; GQA/MLA, cache quantization and paging are the remedies.
- Continuous batching, PagedAttention and prefix caching are the pillars of high-throughput serving; chunked prefill and disaggregation fix prefill/decode interference.
- Speculative decoding speeds up low-batch decoding without changing the output distribution.
- Quantization (AWQ/GPTQ/GGUF/FP8) trades small quality loss for large memory and speed gains; evaluate on your tasks.
- FlashAttention makes exact attention IO-efficient with online softmax.
Exercises
Section titled “Exercises”- Hands-on. Compute KV bytes per token for a model with , , in BF16 and FP8. How many 4K-token sequences fit in 20 GB of free memory?
- Conceptual. Explain why throughput grows with batch size but per-request latency also grows, and what limits the benefit.
- Conceptual. Prove that for speculative decoding, accepting with probability and resampling from the normalized yields samples exactly from .
- Hands-on. Implement a KV-cached decoding loop for the GPT in the architecture lecture and measure speedup versus recomputing the full prefix for 500 tokens.
- Hands-on. Quantize a small model with
llama.cpp(Q4_K_M versus Q8_0) or AutoAWQ, and compare perplexity and a task metric against BF16.
Solutions (sketch)
- BF16: B per token; FP8: 65,536 B. A 4K sequence is 0.5 GiB in BF16, so about 40 sequences in 20 GB; FP8 gives about 80 (ignoring overhead).
- Weight reads amortize over more sequences, so tokens/s per GPU rises; but each step takes longer (more KV reads, more compute) and requests wait to be admitted. Limits: KV memory, then compute saturation, then attention’s KV bandwidth.
- ; the first term is and , so the second term is ; the sum is .
Further reading
Section titled “Further reading”- Kwon et al., “Efficient Memory Management for Large Language Model Serving with PagedAttention” (vLLM, 2023).
- Yu et al., “Orca: A Distributed Serving System for Transformer-Based Generative Models” (2022).
- Dao et al., “FlashAttention” (2022), FlashAttention-2 (2023); Milakov and Gimelshein, “Online normalizer calculation for softmax” (2018).
- Leviathan, Kalman, Matias, “Fast Inference from Transformers via Speculative Decoding” (2023); Chen et al., “Accelerating LLM Decoding with Speculative Sampling” (2023).
- Frantar et al., “GPTQ” (2022); Lin et al., “AWQ” (2023); Xiao et al., “SmoothQuant” (2022); Dettmers et al., “LLM.int8()” (2022).
- Holtzman et al., “The Curious Case of Neural Text Degeneration” (nucleus sampling, 2019).
- Zheng et al., “SGLang” (2023); Zhong et al., “DistServe” (2024); Pope et al., “Efficiently Scaling Transformer Inference” (2022).
- Chip Huyen, AI Engineering (2025), inference optimization chapters; vLLM and llama.cpp documentation.