Skip to content

PyTorch in Practice

What you’ll learn. The working knowledge needed to write and debug real PyTorch code: tensor shapes, dtypes, devices, and views; how autograd records and differentiates; building models with nn.Module; a complete, production-shaped training loop (validation, checkpointing, mixed precision, gradient accumulation, clipping, schedulers); efficient data loading; reproducibility; and profiling and speed-ups (torch.compile, memory tools). Examples target PyTorch 2.x.

A torch.Tensor is an n-dimensional array with a dtype, a device, and optionally a gradient history.

import torch
x = torch.tensor([[1., 2., 3.], [4., 5., 6.]]) # float32 by default for float literals
print(x.shape, x.dtype, x.device) # torch.Size([2, 3]) torch.float32 cpu
z = torch.zeros(2, 3, dtype=torch.bfloat16, device="cuda" if torch.cuda.is_available() else "cpu")

Creation: zeros, ones, full, arange, linspace, eye, rand, randn, randint, empty; torch.from_numpy(a) shares memory with the NumPy array, t.numpy() likewise (CPU only).

Shapes and views. view/reshape change shape; view requires compatible (contiguous) memory and returns a view sharing storage, so in-place edits affect both. transpose/permute return non-contiguous views; call .contiguous() before .view. unsqueeze/squeeze add or remove size-1 dims; expand broadcasts without copying.

Broadcasting aligns shapes from the right; dims must be equal or 1. Mistakes here are the most common silent bug:

a = torch.randn(5, 1); b = torch.randn(5)
print((a + b).shape) # (5, 5), not (5,): a (5,1) plus (5,) broadcasts to a matrix

Indexing: slicing is a view; advanced (integer/boolean) indexing copies. torch.gather/scatter, index_select, masked_fill, torch.where cover most needs.

Linear algebra and reductions: @/matmul (batched over leading dims), einsum (clear for attention-style ops: torch.einsum("bhtd,bhsd->bhts", q, k)), sum/mean/max/argmax(dim=, keepdim=). Remember (Linear Algebra) shape discipline: write the expected shape in a comment next to each line.

dtypes: float32 default; use bf16/fp16 under autocast (Training Deep Networks); integer index tensors must be torch.long for embeddings and cross-entropy targets. Mixed-dtype arithmetic follows promotion rules; explicit .to(dtype) avoids surprises.

Devices: tensors in an operation must live on the same device. .to("cuda") copies; create tensors directly on the device when possible (device= arg) to avoid host-to-device copies. Apple silicon uses "mps".

PyTorch builds the graph dynamically during the forward pass (Backpropagation). Any tensor with requires_grad=True (parameters, by default) records its operations.

w = torch.tensor([2.0, -1.0], requires_grad=True)
x = torch.tensor([1.0, 3.0])
loss = ((w @ x - 4.0) ** 2)
loss.backward() # populates w.grad
print(w.grad) # 2 (w.x - 4) x = 2*(-5)*[1, 3] -> tensor([-10., -30.])

Key facts:

  • .backward() on a scalar accumulates gradients into .grad of leaf tensors. Gradients accumulate across calls, so zero them each step (optimizer.zero_grad(set_to_none=True)). Accumulation is also the basis of gradient accumulation for large effective batches.
  • The graph is freed after backward(); pass retain_graph=True only when needed.
  • torch.no_grad() / torch.inference_mode() disable tracking (evaluation, inference) and save memory. .detach() returns a tensor cut from the graph (used for targets, EMA, logging; use .item() to log scalars, never accumulate loss tensors across steps or you keep graphs alive).
  • In-place ops on tensors needed for backward can raise “modified by an inplace operation” errors.
  • torch.autograd.grad(outputs, inputs, create_graph=True) gives higher-order derivatives (gradient penalties, meta-learning); torch.func (grad, vmap, jacrev, hessian) provides JAX-style functional transforms, including per-sample gradients.
  • Custom ops: subclass torch.autograd.Function with forward and backward, and verify with torch.autograd.gradcheck in float64.

A Module holds parameters (nn.Parameter) and submodules, defines forward, and is called as model(x) (never model.forward(x), which skips hooks).

import torch.nn as nn, torch.nn.functional as F
class MLP(nn.Module):
def __init__(self, d_in, d_hidden, n_classes, p=0.1):
super().__init__()
self.net = nn.Sequential(
nn.Linear(d_in, d_hidden), nn.GELU(), nn.Dropout(p),
nn.Linear(d_hidden, d_hidden), nn.GELU(), nn.Dropout(p),
nn.Linear(d_hidden, n_classes))
def forward(self, x):
return self.net(x) # logits; no softmax here
model = MLP(784, 256, 10)
print(sum(p.numel() for p in model.parameters() if p.requires_grad))

Important details:

  • Submodules and parameters are registered only if assigned as attributes; use nn.ModuleList/nn.ModuleDict for lists, not plain Python lists (otherwise parameters are invisible to the optimizer and .to(device)).
  • register_buffer stores non-trainable state that moves with the module and is saved in state_dict (e.g. BatchNorm running stats, causal masks).
  • model.train() / model.eval() toggle dropout and BatchNorm behavior: forgetting eval() during validation is a classic bug.
  • state_dict() is the recommended way to save weights; load_state_dict(strict=True) catches key mismatches.
  • Freeze layers by p.requires_grad_(False); pass only trainable parameters to the optimizer, or use parameter groups for different LRs/weight decay.
  • Initialization: override via self.apply(init_fn) (Training Deep Networks).
  • Hooks (register_forward_hook, register_full_backward_hook) help inspect activations and gradients.

Loss functions: F.cross_entropy(logits, targets) expects raw logits and integer class indices (it includes log-softmax; see Information Theory); F.binary_cross_entropy_with_logits for binary/multi-label; F.mse_loss; F.nll_loss after log_softmax.

import time, torch
from torch.utils.data import DataLoader
def evaluate(model, loader, device):
model.eval(); total, correct, loss_sum = 0, 0, 0.0
with torch.inference_mode():
for x, y in loader:
x, y = x.to(device, non_blocking=True), y.to(device, non_blocking=True)
logits = model(x)
loss_sum += F.cross_entropy(logits, y, reduction="sum").item()
correct += (logits.argmax(1) == y).sum().item(); total += y.numel()
return loss_sum / total, correct / total
def train(model, train_ds, val_ds, epochs=10, bs=128, lr=3e-4, wd=0.01,
accum=1, clip=1.0, device="cuda", ckpt="best.pt"):
model.to(device)
tl = DataLoader(train_ds, batch_size=bs, shuffle=True, num_workers=4,
pin_memory=True, drop_last=True, persistent_workers=True)
vl = DataLoader(val_ds, batch_size=bs * 2, num_workers=2, pin_memory=True)
decay = [p for n, p in model.named_parameters() if p.ndim >= 2]
no_decay = [p for n, p in model.named_parameters() if p.ndim < 2] # biases, norms
opt = torch.optim.AdamW([{"params": decay, "weight_decay": wd},
{"params": no_decay, "weight_decay": 0.0}], lr=lr)
total_steps = epochs * (len(tl) // accum)
sched = torch.optim.lr_scheduler.OneCycleLR(opt, max_lr=lr, total_steps=total_steps,
pct_start=0.05, anneal_strategy="cos")
use_amp = device.startswith("cuda")
best = float("inf")
for epoch in range(epochs):
model.train(); t0 = time.time(); run = 0.0
for i, (x, y) in enumerate(tl):
x, y = x.to(device, non_blocking=True), y.to(device, non_blocking=True)
with torch.autocast(device_type="cuda", dtype=torch.bfloat16, enabled=use_amp):
loss = F.cross_entropy(model(x), y) / accum
loss.backward()
run += loss.item() * accum
if (i + 1) % accum == 0:
gnorm = torch.nn.utils.clip_grad_norm_(model.parameters(), clip)
opt.step(); sched.step(); opt.zero_grad(set_to_none=True)
vloss, vacc = evaluate(model, vl, device)
print(f"epoch {epoch} train {run/len(tl):.4f} val {vloss:.4f} acc {vacc:.3f} "
f"gnorm {gnorm:.2f} lr {sched.get_last_lr()[0]:.2e} {time.time()-t0:.1f}s")
if vloss < best:
best = vloss
torch.save({"model": model.state_dict(), "opt": opt.state_dict(),
"epoch": epoch}, ckpt)

What this loop gets right: decoupled decay applied only to matrices; bf16 autocast (no loss scaler needed; for fp16 use torch.amp.GradScaler); gradient accumulation (divide loss by accum); clipping after accumulation; zero_grad(set_to_none=True); LR scheduler stepping once per optimizer step; best-checkpoint saving including optimizer state for resuming; eval() plus inference_mode() for validation; and logging of the gradient norm and LR.

Resuming: load model and opt state dicts, restore the scheduler and RNG states and epoch counter. Multi-GPU: DistributedDataParallel (launched with torchrun) for data parallelism; FSDP/FSDP2 shards parameters, gradients, and optimizer states for large models; higher-level wrappers (Hugging Face Accelerate, PyTorch Lightning) hide the boilerplate.

Don’t write your own loop for everything, but do write one at least once: knowing it helps you debug any framework. Keep experiment configuration in one place (dataclass or YAML) and log metrics to a tracker (TensorBoard, Weights & Biases, MLflow).

Dataset defines __len__ and __getitem__; DataLoader batches, shuffles, and parallelizes loading.

from torch.utils.data import Dataset
class CSVDataset(Dataset):
def __init__(self, X, y): self.X, self.y = torch.as_tensor(X, dtype=torch.float32), torch.as_tensor(y)
def __len__(self): return len(self.y)
def __getitem__(self, i): return self.X[i], self.y[i]

Performance levers:

  • num_workers > 0 loads in parallel worker processes; start at 4 to 8 per GPU and measure. pin_memory=True with non_blocking=True speeds host-to-GPU transfer; prefetch_factor and persistent_workers reduce stalls.
  • If GPU utilization is low, you are probably input-bound: decode/augment is slower than the model. Cache preprocessed data, use faster decoders (e.g. GPU JPEG decode, DALI), larger or sharded datasets in formats like WebDataset/Parquet/memory-mapped arrays instead of millions of small files.
  • Variable-length sequences need a collate_fn that pads and returns masks; bucket by length to cut padding waste.
  • For streaming or huge datasets use IterableDataset; handle per-worker sharding to avoid duplicated samples.
  • Split train/val/test correctly (group-aware, time-aware) before building datasets (ML Fundamentals); apply random augmentation to training only.
  • Random state in workers: PyTorch seeds each worker differently; with NumPy/random inside transforms use worker_init_fn or a Generator.

Exact bitwise reproducibility is expensive; aim for statistical reproducibility (same behavior across seeds) and controlled determinism when debugging.

import random, numpy as np, torch
def seed_everything(seed=0, deterministic=False):
random.seed(seed); np.random.seed(seed)
torch.manual_seed(seed); torch.cuda.manual_seed_all(seed)
if deterministic:
torch.use_deterministic_algorithms(True) # errors on nondeterministic ops
torch.backends.cudnn.benchmark = False
# also set CUBLAS_WORKSPACE_CONFIG=:4096:8 in the environment for CUDA matmul determinism
g = torch.Generator().manual_seed(0) # pass to DataLoader(generator=g) / random_split

Sources of nondeterminism: GPU atomics in some kernels (scatter/index_add backward), cuDNN autotuning (benchmark=True), different hardware or library versions, data-loader worker order, floating-point non-associativity across different parallel reductions. Record: seed, git commit, library versions (torch.__version__, CUDA), hardware, config, and data version. Report results over several seeds (Statistics & Estimation); seed-to-seed variation often exceeds small claimed improvements.

  1. Print shapes/dtypes/devices at each stage on one batch (torch.set_printoptions; or assert shapes).
  2. Overfit a single batch to near-zero loss; if impossible, suspect labels, loss, zero_grad, frozen params, or LR.
  3. torch.autograd.set_detect_anomaly(True) locates NaN-producing ops (slow; debug only). Check torch.isfinite(loss).
  4. After backward, inspect p.grad for None (param not used or detached), zeros, or huge norms.
  5. Confirm model.eval() for validation and that dropout/BN behave as intended.
  6. Compare to a trusted reference implementation or the same computation in NumPy on tiny inputs.
  7. Use torch.testing.assert_close for numerical comparisons.

Measure first. GPU kernels are asynchronous: wrap timing with torch.cuda.synchronize() or use CUDA events; otherwise you time only kernel launches.

from torch.profiler import profile, ProfilerActivity, schedule
with profile(activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA],
schedule=schedule(wait=1, warmup=2, active=3), record_shapes=True,
profile_memory=True) as prof:
for step, (x, y) in enumerate(loader):
loss = F.cross_entropy(model(x.cuda()), y.cuda()); loss.backward()
opt.step(); opt.zero_grad(); prof.step()
if step >= 6: break
print(prof.key_averages().table(sort_by="cuda_time_total", row_limit=10))

Typical wins, roughly in order of effort:

  1. Fix the data pipeline (workers, pinning, caching) if the GPU idles.
  2. Mixed precision (bf16 autocast): often 1.5-3x on tensor-core GPUs plus half the activation memory.
  3. Larger batch until memory or generalization limits it; avoid tiny kernels.
  4. torch.compile(model): captures the model into fused, optimized kernels (TorchInductor); commonly 1.2-2x for transformer-style models after a compilation warmup; watch for graph breaks (TORCH_LOGS=graph_breaks) and dynamic shapes causing recompilation.
  5. Fused attention via F.scaled_dot_product_attention (FlashAttention-style kernels) (Attention & Transformers); fused optimizers (AdamW(fused=True)).
  6. Avoid host-device syncs in the hot loop: .item(), print(tensor), and .cpu() force synchronization; log every N steps.
  7. Memory: torch.cuda.memory_summary() and max_memory_allocated(); activation checkpointing (torch.utils.checkpoint) trades compute for memory; gradient accumulation reduces peak memory; sharding (FSDP) for large models; 8-bit optimizers.
  8. Channels-last memory format for convolutions (CNNs and RNNs).

Remember arithmetic intensity: large matmuls are compute-bound, while elementwise ops, normalization, and attention’s softmax are memory-bandwidth-bound, which is why fusion matters. Report throughput (samples or tokens/s) and, for big runs, MFU.

  • Save weights with state_dict (or safetensors, a safe, fast format preferred over pickle-based torch.save for sharing; torch.load of untrusted files can execute code, so use weights_only=True or safetensors).
  • Export: torch.export / ONNX for portable graphs; TorchScript is legacy for new work. Quantization (int8/int4) and inference engines (vLLM for LLM serving, TensorRT, ONNX Runtime) for production.
  • The ecosystem: torchvision, torchaudio, Hugging Face transformers/datasets/peft/accelerate, timm, Lightning. Learn the core before relying on the wrappers.
  • Applying softmax before F.cross_entropy; passing one-hot targets where indices are expected (newer versions accept probabilities, but check).
  • Forgetting optimizer.zero_grad() or calling it between backward() and step().
  • Putting layers in a plain Python list; creating the optimizer before moving the model to the GPU (fine for modern PyTorch parameters, but historically a bug; be consistent).
  • Using loss tensors in logs/sums (retains graph); use .item().
  • Evaluating in train mode, or computing metrics on a different preprocessing path than training.
  • Data leakage via shuffling before splitting time-ordered or grouped data.
  • Benchmarking without warmup/synchronization, or counting compile time as step time.
  • Tensors have shape, dtype, and device; views share memory; broadcasting silently expands shapes, so check shapes constantly.
  • Autograd records ops dynamically; gradients accumulate on leaves, so zero them each step; use no_grad/inference_mode for evaluation.
  • nn.Module registers parameters and buffers; train()/eval() matter; cross_entropy takes logits.
  • A good loop has validation, bf16 autocast, accumulation, clipping, a scheduler, parameter groups, and checkpointing.
  • Throughput is usually limited by data loading or memory traffic before raw FLOPs; profile before optimizing.
  • Seed everything, log versions and configs, and compare across seeds.
  1. Hands-on. Implement the train function above for MNIST with the MLP and verify that the best checkpoint reloads to the same validation loss.
  2. Hands-on. Intentionally make three of the listed bugs (missing zero_grad, no eval() with dropout, softmax before cross-entropy). Record how each manifests in the curves.
  3. Hands-on. Profile one training step of a small transformer (Attention & Transformers) with and without torch.compile and bf16. Report step time and peak memory.
  4. Conceptual. Why is loss.item() inside the loop fine, but total_loss += loss (a tensor) a memory leak?
  5. Hands-on. Write a custom autograd.Function for a clipped ReLU, verify it with gradcheck, and compare against composing torch.clamp.
Solutions (4)
  1. total_loss += loss keeps each step’s loss as a node with grad_fn, so the entire graph of every iteration stays referenced and cannot be freed. .item() extracts a Python float, breaking the link to the graph (and incurring a sync, so do it sparingly).
  • PyTorch documentation and tutorials (pytorch.org), especially “Autograd mechanics”, “Reproducibility”, “torch.compile”, and the Profiler recipe.
  • Stevens, Antiga, Viehmann, Deep Learning with PyTorch (2020).
  • Zhang, Lipton, Li, Smola, Dive into Deep Learning (d2l.ai), PyTorch edition.
  • Paszke et al., “PyTorch: An Imperative Style, High-Performance Deep Learning Library” (2019); Ansel et al., “PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation” (2024).
  • Karpathy, “A Recipe for Training Neural Networks” (blog, 2019).
  • Li et al., “PyTorch Distributed: Experiences on Accelerating Data Parallel Training” (2020).
  • Horace He, “Making Deep Learning Go Brrrr From First Principles” (blog, 2022).