Pretraining
What you’ll learn. How a raw-text corpus becomes a base model: the data pipeline (filtering, deduplication, mixing), the autoregressive objective, scaling laws and the Chinchilla compute-optimal result (and why practitioners now deliberately “overtrain”), optimizer and schedule choices, the parallelism toolbox (DP, ZeRO/FSDP, TP, PP, sequence/expert parallelism), mixed precision (FP16, BF16, FP8), and how to estimate FLOPs, memory and cost.
The objective
Section titled “The objective”A base model is trained to minimize the negative log-likelihood of each token given its prefix:
Documents are tokenized (Tokenization), concatenated with end-of-document separators, and cut into fixed-length sequences (e.g. 4K to 8K tokens early on). Because of the causal mask (architecture), each sequence supplies a loss at every position. Nothing in the objective mentions “helpfulness”; behavior such as following instructions comes later (Post-Training).
Data curation
Section titled “Data curation”Data quality is arguably the largest lever after scale, and the least published. The generic pipeline:
- Sourcing. Web crawls (Common Crawl and derived corpora), code (public repositories), books, papers, encyclopedias, forums, math, multilingual text; increasingly licensed and synthetic data (model-generated explanations, textbook-style rewrites, reasoning traces).
- Extraction. Strip HTML boilerplate (navigation, ads) to the main text; language identification.
- Heuristic filtering. Remove documents with extreme length, repeated lines, low symbol-to-word ratios, too many bullet-only lines, etc.
- Model-based quality filtering. Train a classifier (or use an LLM) to score “educational/high quality” versus random web text, and keep the upper tail. Several public datasets showed that aggressive quality filtering lets smaller models match larger ones trained on unfiltered data.
- Deduplication. Exact (hashing) and near-duplicate (MinHash + LSH, suffix arrays for substring duplicates) removal. Duplicates waste compute, encourage memorization and verbatim regurgitation, and inflate benchmark contamination.
- Safety and privacy filtering. Remove toxic content (with caveats: over-filtering removes minority dialects and reduces ability to recognize toxicity), PII scrubbing, copyright/licensing and opt-out handling.
- Decontamination. Remove n-gram overlaps with known evaluation sets (Evaluating LLMs).
- Mixing. Choose domain proportions (web vs code vs math vs multilingual). Mixtures are tuned with small-scale proxy experiments (ablations at 1B-ish scale), sometimes with annealing: upweighting high-quality data (math, code, instruction-like) during the final stage of training as the learning rate decays.
Multi-epoch. The web is finite. Repeating data up to about 4 epochs costs surprisingly little in loss versus fresh data (Muennighoff et al., 2023); beyond that returns diminish. As frontier runs approach the end of freely available high-quality text, synthetic data and reasoning-trace data are key levers, and the risk of model collapse when training recursively on one’s own unfiltered output is a documented concern.
Scaling laws
Section titled “Scaling laws”Kaplan et al. (2020) found that test loss falls as a smooth power law in parameters , data and compute :
Here is the irreducible loss (entropy of the text), and the other terms are the penalties for finite model size and finite data. The practical consequence: you can fit these curves on cheap small runs and extrapolate to predict the loss of a large run before paying for it. Scaling-law fitting is the core planning tool of a pretraining team.
Compute accounting
Section titled “Compute accounting”Training FLOPs for a dense transformer are well approximated by
Intuition: the forward pass costs about FLOPs per token (one multiply-add per parameter), and backward costs about twice the forward, giving per token. (Attention’s terms add a bit at long context; for MoE use active parameters.)
Chinchilla
Section titled “Chinchilla”Hoffmann et al. (2022, “Training Compute-Optimal Large Language Models”) re-ran the analysis with properly tuned schedules and concluded that earlier large models were undertrained: for a fixed compute budget, parameters and tokens should be scaled in equal proportion. Minimizing subject to gives
and the famous rule of thumb of roughly 20 training tokens per parameter. Their 70B model trained on 1.4T tokens beat the 280B Gopher trained on 300B tokens at the same compute.
Worked example. A 7B-parameter model at 20 tokens/parameter needs B tokens, so FLOPs.
Beyond Chinchilla: overtraining and inference cost
Section titled “Beyond Chinchilla: overtraining and inference cost”Chinchilla minimizes training compute for a target loss. But a deployed model is queried billions of times; inference cost scales with , not . If you expect heavy usage, it is better to train a smaller model on far more tokens than “optimal.” Recent open models are trained on hundreds to thousands of tokens per parameter (e.g. 7B-8B models on 10T+ tokens), well past 20:1, and keep improving, with diminishing but real returns. Sardana and Frankle (2023) formalize this by adding lifetime inference cost to the objective. The practical takeaway: Chinchilla ratios are a baseline for compute-optimal training, not a rule for what to ship.
Other findings and caveats: scaling laws predict aggregate loss well but downstream benchmark accuracy less smoothly; some capabilities appear to emerge abruptly, though much of that “emergence” disappears under continuous metrics (Schaeffer et al., 2023); data quality shifts the constants, not just the exponents.
Optimization
Section titled “Optimization”Optimizer. AdamW (Adam with decoupled weight decay) dominates, with typical settings , (lower than the default 0.999, for stability with large batches), , weight decay about 0.1, gradient clipping at norm 1.0. Newer optimizers such as Muon (orthogonalized-update momentum) and variants have shown efficiency gains in several 2025 reports; treat that as evolving.
Learning-rate schedule. Linear warmup over the first few thousand steps, then cosine decay to about 10% of peak. A popular alternative is WSD (warmup-stable-decay): constant LR for most of training, then a short sharp decay, which makes it easy to branch checkpoints and extend training.
Batch size. Measured in tokens: from about 1M tokens early to 4M-16M+ later in training. Batch size can be increased as training progresses (batch-size warmup) since the critical batch size (beyond which more data parallelism stops helping per step) grows as loss decreases.
Learning rate and width. The optimal LR shrinks as models widen. P (maximal update parametrization) reparametrizes so that hyperparameters tuned on a small proxy transfer to the large model, avoiding expensive tuning at scale.
Instabilities. Loss spikes happen. Mitigations: lower LR, QK-norm, z-loss on the logits, careful initialization, skipping the offending batches, and restarting from an earlier checkpoint. Large runs keep frequent checkpoints and hardware-failure automation; at thousands of GPUs, failures occur every few hours.
Memory: why one GPU is not enough
Section titled “Memory: why one GPU is not enough”For a model with parameters trained with mixed-precision Adam, per-parameter state is roughly:
| State | Bytes/param |
|---|---|
| BF16 weights (working copy) | 2 |
| BF16 gradients | 2 |
| FP32 master weights | 4 |
| Adam first moment (FP32) | 4 |
| Adam second moment (FP32) | 4 |
| Total | ~16 |
So a 7B model needs about GB before counting activations, which scale with batch size, sequence length and depth. This exceeds a single 80 GB accelerator. Hence parallelism.
Parallelism toolbox
Section titled “Parallelism toolbox”Data parallel (DP): same model on every GPU, different batch shards; all-reduce gradientsZeRO / FSDP: DP, but shard optimizer state / gradients / params across GPUsTensor parallel (TP): split each weight matrix across GPUs; all-reduce inside every layerPipeline parallel (PP): split layers into stages across GPUs; micro-batches flow throughSequence/context par.: split the sequence dimension (long context)Expert parallel (EP): place different MoE experts on different GPUs; all-to-all token routingData parallelism and ZeRO/FSDP
Section titled “Data parallelism and ZeRO/FSDP”Plain DP replicates everything, wasting memory. ZeRO (Rajbhandari et al., 2020) removes redundancy in three stages:
- Stage 1: shard optimizer states (the biggest 12 bytes/param) across devices.
- Stage 2: also shard gradients.
- Stage 3: also shard parameters; each layer’s weights are all-gathered just in time, used, then freed. PyTorch’s FSDP implements this idea.
Per-device memory falls roughly as at stage 3, at the price of extra communication (about 1.5x DP’s volume). Overlapping communication with compute hides most of it on fast interconnect.
Tensor parallelism
Section titled “Tensor parallelism”For an MLP , split by columns and by rows (Megatron-LM style): each device computes its slice without communicating until a single all-reduce after . Attention heads split naturally across devices. TP needs very fast links (NVLink), so it is typically used within a node (8 GPUs). Communication per layer makes it latency-sensitive.
Pipeline parallelism
Section titled “Pipeline parallelism”Layers are divided into stages. To avoid idle devices, each batch is split into micro-batches that flow through the pipeline. The idle “bubble” fraction is about for stages (GPipe schedule); interleaved 1F1B and zero-bubble schedules reduce it. PP communicates only activations between adjacent stages, so it tolerates slower inter-node links.
Combining: 3D/4D parallelism
Section titled “Combining: 3D/4D parallelism”Large runs combine them: TP within a node, PP across a few nodes, DP/FSDP across replicas, plus sequence parallelism for long contexts and expert parallelism for MoE. The configuration is chosen against hardware topology. A simple sizing heuristic: use as little model parallelism as fits memory, because DP scales most efficiently.
Utilization: MFU
Section titled “Utilization: MFU”Model FLOPs utilization . Well-tuned large runs report on the order of 35-55% MFU; the rest is lost to communication, memory-bound ops, pipeline bubbles and stragglers. Techniques: fused kernels, FlashAttention, activation (gradient) checkpointing (recompute activations in backward, trading about 33% extra compute for large memory savings), and overlapping communication.
Mixed precision
Section titled “Mixed precision”Lower-precision arithmetic multiplies throughput and cuts memory traffic.
- FP16: 5 exponent bits, 10 mantissa bits. Narrow range means gradients underflow, so it needs loss scaling.
- BF16: 8 exponent bits (the FP32 range), 7 mantissa bits. No loss scaling needed; the default for LLM training on recent GPUs/TPUs. Keep the master weights, optimizer states, and numerically sensitive ops (softmax, normalization, loss) in FP32.
- FP8: two formats (E4M3 for forward activations/weights with more precision, E5M2 for gradients with more range), supported on newer GPUs, roughly doubling matmul throughput over BF16. Requires per-tensor or fine-grained (block/tile) scaling factors to keep values in range. Large-scale FP8 training was reported publicly in 2024-2025 with fine-grained scaling and high-precision accumulation; lower precision formats (e.g. 4-bit with block scaling) are an active research area.
import torchmodel = model.cuda()opt = torch.optim.AdamW(model.parameters(), lr=3e-4, betas=(0.9, 0.95), weight_decay=0.1)for x, y in loader: with torch.autocast("cuda", dtype=torch.bfloat16): # matmuls in bf16, reductions in fp32 _, loss = model(x.cuda(), y.cuda()) loss.backward() torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0) opt.step(); opt.zero_grad(set_to_none=True)Practical compute and cost estimate
Section titled “Practical compute and cost estimate”Example: train a 8B-parameter dense model on 15T tokens.
- FLOPs: .
- A GPU with about peak dense BF16 FLOPs/s at 40% MFU delivers FLOP/s, so you need GPU-seconds, about 500,000 GPU-hours.
- On 2,048 GPUs that is about 10 days. At a few USD per GPU-hour that is on the order of a few million USD in compute alone, excluding experiments (which typically cost as much or more), data work and staff.
(The numbers show the method; check current hardware specs and prices before using them.)
Monitoring a run
Section titled “Monitoring a run”Track: training loss (smoothed), gradient norm (spikes precede divergence), per-layer activation statistics, learning rate, tokens/s and MFU, data-loader stalls, and periodic downstream evals (few-shot benchmarks, held-out perplexity per domain). Evaluate at checkpoints, since final-loss differences are small and loss alone does not reveal capability differences.
Key takeaways
Section titled “Key takeaways”- Pretraining is next-token prediction at scale; data curation (filtering, dedup, mixing, annealing) often matters as much as architecture.
- Loss follows power laws in , and ; fit small runs to predict large ones. Training cost is about FLOPs.
- Chinchilla: about 20 tokens/parameter is compute-optimal for training, but because inference dominates lifetime cost, modern models are deliberately overtrained.
- A 16-bytes-per-parameter Adam footprint forces sharding: FSDP/ZeRO for states, TP for wide layers, PP across nodes, EP for MoE.
- BF16 is the default; FP8 gives more throughput with careful scaling. MFU of 35-55% is good.
Exercises
Section titled “Exercises”- Conceptual. Derive : why is backward about twice the forward cost?
- Hands-on. Given a budget of FLOPs, compute Chinchilla-optimal and using and . Then compute them for a model constrained to be 2x smaller than optimal; how many tokens would it need to reach the same loss approximately (use the form qualitatively)?
- Conceptual. A pipeline has 8 stages. How many micro-batches keep the bubble under 10%?
- Hands-on. Using
torch.distributed.fsdp.FullyShardedDataParallel(or Hugging Face Accelerate), wrap the minimal GPT from the architecture lecture and compare peak memory per GPU against DDP. - Conceptual. Why does BF16 not need loss scaling while FP16 does?
Solutions (sketch)
- Forward does one matmul per weight: FLOPs per parameter per token. Backward needs gradients with respect to activations (another matmul per weight) and with respect to weights (another), so . Total 6.
- (29B), (580B tokens).
- Bubble .
- BF16 has FP32’s exponent range, so small gradients do not underflow; FP16’s narrow range does.
Further reading
Section titled “Further reading”- Kaplan et al., “Scaling Laws for Neural Language Models” (2020).
- Hoffmann et al., “Training Compute-Optimal Large Language Models” (Chinchilla, 2022).
- Sardana and Frankle, “Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws” (2023).
- Muennighoff et al., “Scaling Data-Constrained Language Models” (2023).
- Penedo et al., “The FineWeb Datasets” (2024); Raffel et al., T5 / C4 paper (2020); Lee et al., “Deduplicating Training Data Makes Language Models Better” (2021).
- Rajbhandari et al., “ZeRO” (2020); Shoeybi et al., “Megatron-LM” (2019); Narayanan et al., “Efficient Large-Scale Language Model Training on GPU Clusters” (2021).
- Micikevicius et al., “Mixed Precision Training” (2018) and “FP8 Formats for Deep Learning” (2022).
- Yang et al., “Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer” (P, 2022).
- The Hugging Face “Ultra-Scale Playbook” and open technical reports of Llama, OLMo and DeepSeek for practical details.