PyTorch in Practice
What you’ll learn. The working knowledge needed to write and debug real PyTorch code: tensor shapes, dtypes, devices, and views; how autograd records and differentiates; building models with nn.Module; a complete, production-shaped training loop (validation, checkpointing, mixed precision, gradient accumulation, clipping, schedulers); efficient data loading; reproducibility; and profiling and speed-ups (torch.compile, memory tools). Examples target PyTorch 2.x.
Tensors
Section titled “Tensors”A torch.Tensor is an n-dimensional array with a dtype, a device, and optionally a gradient history.
import torchx = torch.tensor([[1., 2., 3.], [4., 5., 6.]]) # float32 by default for float literalsprint(x.shape, x.dtype, x.device) # torch.Size([2, 3]) torch.float32 cpuz = torch.zeros(2, 3, dtype=torch.bfloat16, device="cuda" if torch.cuda.is_available() else "cpu")Creation: zeros, ones, full, arange, linspace, eye, rand, randn, randint, empty; torch.from_numpy(a) shares memory with the NumPy array, t.numpy() likewise (CPU only).
Shapes and views. view/reshape change shape; view requires compatible (contiguous) memory and returns a view sharing storage, so in-place edits affect both. transpose/permute return non-contiguous views; call .contiguous() before .view. unsqueeze/squeeze add or remove size-1 dims; expand broadcasts without copying.
Broadcasting aligns shapes from the right; dims must be equal or 1. Mistakes here are the most common silent bug:
a = torch.randn(5, 1); b = torch.randn(5)print((a + b).shape) # (5, 5), not (5,): a (5,1) plus (5,) broadcasts to a matrixIndexing: slicing is a view; advanced (integer/boolean) indexing copies. torch.gather/scatter, index_select, masked_fill, torch.where cover most needs.
Linear algebra and reductions: @/matmul (batched over leading dims), einsum (clear for attention-style ops: torch.einsum("bhtd,bhsd->bhts", q, k)), sum/mean/max/argmax(dim=, keepdim=). Remember (Linear Algebra) shape discipline: write the expected shape in a comment next to each line.
dtypes: float32 default; use bf16/fp16 under autocast (Training Deep Networks); integer index tensors must be torch.long for embeddings and cross-entropy targets. Mixed-dtype arithmetic follows promotion rules; explicit .to(dtype) avoids surprises.
Devices: tensors in an operation must live on the same device. .to("cuda") copies; create tensors directly on the device when possible (device= arg) to avoid host-to-device copies. Apple silicon uses "mps".
Autograd
Section titled “Autograd”PyTorch builds the graph dynamically during the forward pass (Backpropagation). Any tensor with requires_grad=True (parameters, by default) records its operations.
w = torch.tensor([2.0, -1.0], requires_grad=True)x = torch.tensor([1.0, 3.0])loss = ((w @ x - 4.0) ** 2)loss.backward() # populates w.gradprint(w.grad) # 2 (w.x - 4) x = 2*(-5)*[1, 3] -> tensor([-10., -30.])Key facts:
.backward()on a scalar accumulates gradients into.gradof leaf tensors. Gradients accumulate across calls, so zero them each step (optimizer.zero_grad(set_to_none=True)). Accumulation is also the basis of gradient accumulation for large effective batches.- The graph is freed after
backward(); passretain_graph=Trueonly when needed. torch.no_grad()/torch.inference_mode()disable tracking (evaluation, inference) and save memory..detach()returns a tensor cut from the graph (used for targets, EMA, logging; use.item()to log scalars, never accumulate loss tensors across steps or you keep graphs alive).- In-place ops on tensors needed for backward can raise “modified by an inplace operation” errors.
torch.autograd.grad(outputs, inputs, create_graph=True)gives higher-order derivatives (gradient penalties, meta-learning);torch.func(grad,vmap,jacrev,hessian) provides JAX-style functional transforms, including per-sample gradients.- Custom ops: subclass
torch.autograd.Functionwithforwardandbackward, and verify withtorch.autograd.gradcheckin float64.
nn.Module
Section titled “nn.Module”A Module holds parameters (nn.Parameter) and submodules, defines forward, and is called as model(x) (never model.forward(x), which skips hooks).
import torch.nn as nn, torch.nn.functional as F
class MLP(nn.Module): def __init__(self, d_in, d_hidden, n_classes, p=0.1): super().__init__() self.net = nn.Sequential( nn.Linear(d_in, d_hidden), nn.GELU(), nn.Dropout(p), nn.Linear(d_hidden, d_hidden), nn.GELU(), nn.Dropout(p), nn.Linear(d_hidden, n_classes)) def forward(self, x): return self.net(x) # logits; no softmax here
model = MLP(784, 256, 10)print(sum(p.numel() for p in model.parameters() if p.requires_grad))Important details:
- Submodules and parameters are registered only if assigned as attributes; use
nn.ModuleList/nn.ModuleDictfor lists, not plain Python lists (otherwise parameters are invisible to the optimizer and.to(device)). register_bufferstores non-trainable state that moves with the module and is saved instate_dict(e.g. BatchNorm running stats, causal masks).model.train()/model.eval()toggle dropout and BatchNorm behavior: forgettingeval()during validation is a classic bug.state_dict()is the recommended way to save weights;load_state_dict(strict=True)catches key mismatches.- Freeze layers by
p.requires_grad_(False); pass only trainable parameters to the optimizer, or use parameter groups for different LRs/weight decay. - Initialization: override via
self.apply(init_fn)(Training Deep Networks). - Hooks (
register_forward_hook,register_full_backward_hook) help inspect activations and gradients.
Loss functions: F.cross_entropy(logits, targets) expects raw logits and integer class indices (it includes log-softmax; see Information Theory); F.binary_cross_entropy_with_logits for binary/multi-label; F.mse_loss; F.nll_loss after log_softmax.
A robust training loop
Section titled “A robust training loop”import time, torchfrom torch.utils.data import DataLoader
def evaluate(model, loader, device): model.eval(); total, correct, loss_sum = 0, 0, 0.0 with torch.inference_mode(): for x, y in loader: x, y = x.to(device, non_blocking=True), y.to(device, non_blocking=True) logits = model(x) loss_sum += F.cross_entropy(logits, y, reduction="sum").item() correct += (logits.argmax(1) == y).sum().item(); total += y.numel() return loss_sum / total, correct / total
def train(model, train_ds, val_ds, epochs=10, bs=128, lr=3e-4, wd=0.01, accum=1, clip=1.0, device="cuda", ckpt="best.pt"): model.to(device) tl = DataLoader(train_ds, batch_size=bs, shuffle=True, num_workers=4, pin_memory=True, drop_last=True, persistent_workers=True) vl = DataLoader(val_ds, batch_size=bs * 2, num_workers=2, pin_memory=True)
decay = [p for n, p in model.named_parameters() if p.ndim >= 2] no_decay = [p for n, p in model.named_parameters() if p.ndim < 2] # biases, norms opt = torch.optim.AdamW([{"params": decay, "weight_decay": wd}, {"params": no_decay, "weight_decay": 0.0}], lr=lr) total_steps = epochs * (len(tl) // accum) sched = torch.optim.lr_scheduler.OneCycleLR(opt, max_lr=lr, total_steps=total_steps, pct_start=0.05, anneal_strategy="cos") use_amp = device.startswith("cuda") best = float("inf") for epoch in range(epochs): model.train(); t0 = time.time(); run = 0.0 for i, (x, y) in enumerate(tl): x, y = x.to(device, non_blocking=True), y.to(device, non_blocking=True) with torch.autocast(device_type="cuda", dtype=torch.bfloat16, enabled=use_amp): loss = F.cross_entropy(model(x), y) / accum loss.backward() run += loss.item() * accum if (i + 1) % accum == 0: gnorm = torch.nn.utils.clip_grad_norm_(model.parameters(), clip) opt.step(); sched.step(); opt.zero_grad(set_to_none=True) vloss, vacc = evaluate(model, vl, device) print(f"epoch {epoch} train {run/len(tl):.4f} val {vloss:.4f} acc {vacc:.3f} " f"gnorm {gnorm:.2f} lr {sched.get_last_lr()[0]:.2e} {time.time()-t0:.1f}s") if vloss < best: best = vloss torch.save({"model": model.state_dict(), "opt": opt.state_dict(), "epoch": epoch}, ckpt)What this loop gets right: decoupled decay applied only to matrices; bf16 autocast (no loss scaler needed; for fp16 use torch.amp.GradScaler); gradient accumulation (divide loss by accum); clipping after accumulation; zero_grad(set_to_none=True); LR scheduler stepping once per optimizer step; best-checkpoint saving including optimizer state for resuming; eval() plus inference_mode() for validation; and logging of the gradient norm and LR.
Resuming: load model and opt state dicts, restore the scheduler and RNG states and epoch counter. Multi-GPU: DistributedDataParallel (launched with torchrun) for data parallelism; FSDP/FSDP2 shards parameters, gradients, and optimizer states for large models; higher-level wrappers (Hugging Face Accelerate, PyTorch Lightning) hide the boilerplate.
Don’t write your own loop for everything, but do write one at least once: knowing it helps you debug any framework. Keep experiment configuration in one place (dataclass or YAML) and log metrics to a tracker (TensorBoard, Weights & Biases, MLflow).
Data loading
Section titled “Data loading”Dataset defines __len__ and __getitem__; DataLoader batches, shuffles, and parallelizes loading.
from torch.utils.data import Dataset
class CSVDataset(Dataset): def __init__(self, X, y): self.X, self.y = torch.as_tensor(X, dtype=torch.float32), torch.as_tensor(y) def __len__(self): return len(self.y) def __getitem__(self, i): return self.X[i], self.y[i]Performance levers:
num_workers > 0loads in parallel worker processes; start at 4 to 8 per GPU and measure.pin_memory=Truewithnon_blocking=Truespeeds host-to-GPU transfer;prefetch_factorandpersistent_workersreduce stalls.- If GPU utilization is low, you are probably input-bound: decode/augment is slower than the model. Cache preprocessed data, use faster decoders (e.g. GPU JPEG decode, DALI), larger or sharded datasets in formats like WebDataset/Parquet/memory-mapped arrays instead of millions of small files.
- Variable-length sequences need a
collate_fnthat pads and returns masks; bucket by length to cut padding waste. - For streaming or huge datasets use
IterableDataset; handle per-worker sharding to avoid duplicated samples. - Split train/val/test correctly (group-aware, time-aware) before building datasets (ML Fundamentals); apply random augmentation to training only.
- Random state in workers: PyTorch seeds each worker differently; with NumPy/
randominside transforms useworker_init_fnor aGenerator.
Reproducibility
Section titled “Reproducibility”Exact bitwise reproducibility is expensive; aim for statistical reproducibility (same behavior across seeds) and controlled determinism when debugging.
import random, numpy as np, torchdef seed_everything(seed=0, deterministic=False): random.seed(seed); np.random.seed(seed) torch.manual_seed(seed); torch.cuda.manual_seed_all(seed) if deterministic: torch.use_deterministic_algorithms(True) # errors on nondeterministic ops torch.backends.cudnn.benchmark = False # also set CUBLAS_WORKSPACE_CONFIG=:4096:8 in the environment for CUDA matmul determinismg = torch.Generator().manual_seed(0) # pass to DataLoader(generator=g) / random_splitSources of nondeterminism: GPU atomics in some kernels (scatter/index_add backward), cuDNN autotuning (benchmark=True), different hardware or library versions, data-loader worker order, floating-point non-associativity across different parallel reductions. Record: seed, git commit, library versions (torch.__version__, CUDA), hardware, config, and data version. Report results over several seeds (Statistics & Estimation); seed-to-seed variation often exceeds small claimed improvements.
Debugging checklist
Section titled “Debugging checklist”- Print shapes/dtypes/devices at each stage on one batch (
torch.set_printoptions; orassertshapes). - Overfit a single batch to near-zero loss; if impossible, suspect labels, loss,
zero_grad, frozen params, or LR. torch.autograd.set_detect_anomaly(True)locates NaN-producing ops (slow; debug only). Checktorch.isfinite(loss).- After
backward, inspectp.gradforNone(param not used or detached), zeros, or huge norms. - Confirm
model.eval()for validation and that dropout/BN behave as intended. - Compare to a trusted reference implementation or the same computation in NumPy on tiny inputs.
- Use
torch.testing.assert_closefor numerical comparisons.
Profiling and speed
Section titled “Profiling and speed”Measure first. GPU kernels are asynchronous: wrap timing with torch.cuda.synchronize() or use CUDA events; otherwise you time only kernel launches.
from torch.profiler import profile, ProfilerActivity, schedule
with profile(activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA], schedule=schedule(wait=1, warmup=2, active=3), record_shapes=True, profile_memory=True) as prof: for step, (x, y) in enumerate(loader): loss = F.cross_entropy(model(x.cuda()), y.cuda()); loss.backward() opt.step(); opt.zero_grad(); prof.step() if step >= 6: breakprint(prof.key_averages().table(sort_by="cuda_time_total", row_limit=10))Typical wins, roughly in order of effort:
- Fix the data pipeline (workers, pinning, caching) if the GPU idles.
- Mixed precision (bf16 autocast): often 1.5-3x on tensor-core GPUs plus half the activation memory.
- Larger batch until memory or generalization limits it; avoid tiny kernels.
torch.compile(model): captures the model into fused, optimized kernels (TorchInductor); commonly 1.2-2x for transformer-style models after a compilation warmup; watch for graph breaks (TORCH_LOGS=graph_breaks) and dynamic shapes causing recompilation.- Fused attention via
F.scaled_dot_product_attention(FlashAttention-style kernels) (Attention & Transformers); fused optimizers (AdamW(fused=True)). - Avoid host-device syncs in the hot loop:
.item(),print(tensor), and.cpu()force synchronization; log every N steps. - Memory:
torch.cuda.memory_summary()andmax_memory_allocated(); activation checkpointing (torch.utils.checkpoint) trades compute for memory; gradient accumulation reduces peak memory; sharding (FSDP) for large models; 8-bit optimizers. - Channels-last memory format for convolutions (CNNs and RNNs).
Remember arithmetic intensity: large matmuls are compute-bound, while elementwise ops, normalization, and attention’s softmax are memory-bandwidth-bound, which is why fusion matters. Report throughput (samples or tokens/s) and, for big runs, MFU.
Interoperability and deployment
Section titled “Interoperability and deployment”- Save weights with
state_dict(orsafetensors, a safe, fast format preferred over pickle-basedtorch.savefor sharing;torch.loadof untrusted files can execute code, so useweights_only=Trueor safetensors). - Export:
torch.export/ ONNX for portable graphs; TorchScript is legacy for new work. Quantization (int8/int4) and inference engines (vLLM for LLM serving, TensorRT, ONNX Runtime) for production. - The ecosystem:
torchvision,torchaudio, Hugging Facetransformers/datasets/peft/accelerate,timm, Lightning. Learn the core before relying on the wrappers.
Pitfalls
Section titled “Pitfalls”- Applying softmax before
F.cross_entropy; passing one-hot targets where indices are expected (newer versions accept probabilities, but check). - Forgetting
optimizer.zero_grad()or calling it betweenbackward()andstep(). - Putting layers in a plain Python list; creating the optimizer before moving the model to the GPU (fine for modern PyTorch parameters, but historically a bug; be consistent).
- Using
losstensors in logs/sums (retains graph); use.item(). - Evaluating in train mode, or computing metrics on a different preprocessing path than training.
- Data leakage via shuffling before splitting time-ordered or grouped data.
- Benchmarking without warmup/synchronization, or counting compile time as step time.
Key takeaways
Section titled “Key takeaways”- Tensors have shape, dtype, and device; views share memory; broadcasting silently expands shapes, so check shapes constantly.
- Autograd records ops dynamically; gradients accumulate on leaves, so zero them each step; use
no_grad/inference_modefor evaluation. nn.Moduleregisters parameters and buffers;train()/eval()matter;cross_entropytakes logits.- A good loop has validation, bf16 autocast, accumulation, clipping, a scheduler, parameter groups, and checkpointing.
- Throughput is usually limited by data loading or memory traffic before raw FLOPs; profile before optimizing.
- Seed everything, log versions and configs, and compare across seeds.
Exercises
Section titled “Exercises”- Hands-on. Implement the
trainfunction above for MNIST with the MLP and verify that the best checkpoint reloads to the same validation loss. - Hands-on. Intentionally make three of the listed bugs (missing
zero_grad, noeval()with dropout, softmax before cross-entropy). Record how each manifests in the curves. - Hands-on. Profile one training step of a small transformer (Attention & Transformers) with and without
torch.compileand bf16. Report step time and peak memory. - Conceptual. Why is
loss.item()inside the loop fine, buttotal_loss += loss(a tensor) a memory leak? - Hands-on. Write a custom
autograd.Functionfor a clipped ReLU, verify it withgradcheck, and compare against composingtorch.clamp.
Solutions (4)
total_loss += losskeeps each step’slossas a node withgrad_fn, so the entire graph of every iteration stays referenced and cannot be freed..item()extracts a Python float, breaking the link to the graph (and incurring a sync, so do it sparingly).
Further reading
Section titled “Further reading”- PyTorch documentation and tutorials (pytorch.org), especially “Autograd mechanics”, “Reproducibility”, “torch.compile”, and the Profiler recipe.
- Stevens, Antiga, Viehmann, Deep Learning with PyTorch (2020).
- Zhang, Lipton, Li, Smola, Dive into Deep Learning (d2l.ai), PyTorch edition.
- Paszke et al., “PyTorch: An Imperative Style, High-Performance Deep Learning Library” (2019); Ansel et al., “PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation” (2024).
- Karpathy, “A Recipe for Training Neural Networks” (blog, 2019).
- Li et al., “PyTorch Distributed: Experiences on Accelerating Data Parallel Training” (2020).
- Horace He, “Making Deep Learning Go Brrrr From First Principles” (blog, 2022).