Data, Augmentation & Evaluation
What you’ll learn. How to scope and collect a vision dataset, design labeling workflows and measure annotation quality, split data without leakage, use augmentation (classical, mix-based, learned) and synthetic data, understand and test distribution shift and robustness, and choose metrics (classification, calibration, detection, segmentation, retrieval) with proper uncertainty. Most real-world vision failures are data failures, so this lecture is where the practical leverage is. Metrics specific to detection and segmentation are defined in Object Detection and Segmentation.
Data-centric thinking
Section titled “Data-centric thinking”For a fixed architecture, improving data quality, coverage and labels usually beats architecture tinkering, particularly with strong pretrained backbones (Self-Supervised & Foundation Vision Models). Ng’s “data-centric AI” framing (2021) is a useful stance: hold the model fixed, iterate on the data.
Building a dataset
Section titled “Building a dataset”Specify the task and the deployment distribution first
Section titled “Specify the task and the deployment distribution first”Write down: the exact output (labels, boxes, masks, attributes), what counts as correct, the input conditions (cameras, lighting, angles, resolution, motion blur, compression), the classes and “none of the above” handling, and the cost asymmetry of errors (missed defect versus false alarm). The training set must match the deployment distribution, so collect from the real sensor in the real environment whenever possible. A model trained on web images of “forklift” will not know your warehouse’s camera.
Sources and sampling
Section titled “Sources and sampling”- Own capture (best match, expensive), public datasets (ImageNet, COCO, OpenImages, LVIS, ADE20K, Cityscapes, medical sets; check licenses and consent), web scraping (license/copyright and privacy implications; filter), synthetic (below), and retrieval of similar images from large unlabeled pools using embeddings (CLIP/DINOv2 nearest neighbors) to find rare cases.
- Cover long-tail conditions deliberately: night, rain, occlusion, rare classes, unusual poses. Natural sampling underrepresents them; use stratified collection or targeted mining.
- Deduplicate (perceptual hashing, embedding similarity) to avoid near-duplicates across train/test.
Splits and leakage
Section titled “Splits and leakage”The classic bug: random frame-level splits of video or of multiple photos of the same object/patient/location put near-identical images into train and test, inflating metrics. Split by the independent unit: patient, session, scene, camera, or time period. For time-evolving problems, split by time (train on past, test on future). Hold out a final test set that you touch rarely; use a separate validation set for tuning. For small datasets, use grouped k-fold cross-validation.
from sklearn.model_selection import GroupKFoldimport numpy as np
# groups: e.g., one id per patient/scene so images from one group never straddle foldsX = np.arange(1000); y = np.random.randint(0, 2, 1000); groups = np.repeat(np.arange(100), 10)for tr, va in GroupKFold(n_splits=5).split(X, y, groups): assert set(groups[tr]).isdisjoint(groups[va])How much data?
Section titled “How much data?”There is no universal number. With a strong pretrained backbone, a few hundred well-chosen labeled images per class can give a useful classifier, and 500-5,000 images often suffice for fine-tuning a detector in a narrow domain; from-scratch training needs orders of magnitude more. Plot a learning curve (metric versus training-set fraction): if it is still steep, more data helps; if flat, fix labels, resolution or model capacity.
Labeling
Section titled “Labeling”Label types and cost
Section titled “Label types and cost”Image-level tags are cheapest; boxes cost roughly 10-30 seconds each; polygons/masks cost minutes; dense semantic labels cost far more (Cityscapes fine annotation reportedly took on the order of 1.5 hours per image). Choose the cheapest label that supports the task, or use model-assisted labeling: pre-label with a detector or open-vocabulary model, then human-correct. SAM-style click-to-mask tools make mask labeling much faster (Segmentation). Open-vocabulary detectors + SAM can auto-label, followed by human verification.
Guidelines and quality
Section titled “Guidelines and quality”- Write a labeling guide with examples and edge cases (what is a “defect”? tight boxes or loose? occluded objects included? truncated objects?). Ambiguity in the guide becomes noise in the data.
- Measure inter-annotator agreement: Cohen’s for two annotators, Fleiss’ or Krippendorff’s for more; mask/box IoU between annotators. Human agreement is the practical ceiling on achievable metric. Always have a gold subset labeled by experts.
- Label noise: typical benchmark labels contain a few percent errors (studies of ImageNet and other test sets found around 3-6%, and some classes ambiguous). Find noisy labels by training, then inspecting high-loss/low-confidence examples or using confident learning (cleanlab). Noisy test labels cap measurable accuracy and can mis-rank models.
- Tools: CVAT, Label Studio, Labelbox, Roboflow, FiftyOne (dataset inspection and error analysis, embeddings, filtering). Version datasets (DVC, Hugging Face datasets, lakeFS) so experiments are reproducible.
- Active learning: choose which images to label next by model uncertainty (entropy, margin, MC dropout, ensemble disagreement), diversity (core-set selection in embedding space), or a mix. It can reduce labeling cost substantially, but it needs a decent initial model and can bias the label distribution; keep a random-sample test set.
- Weak supervision, pseudo-labeling and self-training (label unlabeled data with the model and retrain on confident predictions) are effective when confirmation bias is controlled (confidence thresholds, noisy-student-style noise, an independent validation set).
Augmentation
Section titled “Augmentation”Augmentation injects invariances and increases effective data. It is the cheapest regularizer in vision.
Standard transforms
Section titled “Standard transforms”| Family | Examples | Notes |
|---|---|---|
| Geometric | random crop/resized crop, flip, rotation, affine, perspective | Only if label-preserving (no flips for digits/text; vertical flip wrong for street scenes; rotations fine for aerial/microscopy) |
| Photometric | brightness, contrast, saturation, hue, gamma, grayscale | Hue shifts can be label-destroying (ripe fruit, traffic lights) |
| Noise/blur | Gaussian noise, blur, JPEG compression, motion blur | Simulates sensor and codec shift |
| Occlusion | Cutout, random erasing | Robustness to occlusion |
| Mix-based | Mixup (, labels mixed with the same ), CutMix (paste a patch; labels mixed by area), Mosaic (4 images tiled; detection) | Regularize and calibrate; used in all modern training recipes (Vision Transformers) |
| Automated | AutoAugment, RandAugment (random ops at magnitude ), TrivialAugment | Few hyperparameters; strong baselines |
| Copy-paste | paste instance masks onto other backgrounds | Strong for instance segmentation and rare classes |
For detection/segmentation, geometry must be applied jointly to image and annotations (boxes, masks, keypoints); masks use nearest-neighbor interpolation. Use albumentations or torchvision.transforms.v2, which handle targets.
import torchfrom torchvision.transforms import v2from torchvision import tv_tensors
train_tf = v2.Compose([ v2.RandomResizedCrop(224, scale=(0.5, 1.0), antialias=True), v2.RandomHorizontalFlip(), v2.ColorJitter(0.3, 0.3, 0.3, 0.02), v2.RandAugment(num_ops=2, magnitude=9), v2.ToDtype(torch.float32, scale=True), v2.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),])# For detection, wrap targets so transforms move them together with the image:# img = tv_tensors.Image(img); target = {"boxes": tv_tensors.BoundingBoxes(b, format="XYXY", canvas_size=img.shape[-2:])}Guidelines: validate augmentations visually (dump 50 augmented samples and look); never augment validation/test data (apart from deterministic resize/center-crop); match augmentation strength to data size; consider test-time augmentation (average predictions over flips/scales) for a cheap accuracy gain at higher inference cost.
Learned and generative augmentation
Section titled “Learned and generative augmentation”Use diffusion models to create variations, backgrounds or inpainted objects (Diffusion & Image Generation); style transfer for domain randomization; GAN/diffusion-based data synthesis for rare cases. Validate carefully: generated images can carry artifacts the model overfits to, and synthetic-only training often underperforms real data.
Synthetic data
Section titled “Synthetic data”- Simulation and rendering (Blender, Unity, Unreal, NVIDIA Omniverse/Isaac Sim, BlenderProc): perfect labels (masks, depth, pose, normals) for free, controllable coverage of rare events. Challenge: the sim-to-real gap.
- Domain randomization: randomize textures, lighting, backgrounds, camera pose so the real world looks like “one more variation” (works well for robotics pose estimation).
- Domain adaptation / mixed training: mix synthetic with a modest set of real images; fine-tune on real; use style-matching or feature alignment.
- Generative synthetic data: text-to-image models plus auto-labeling (open-vocabulary detectors/SAM) produce labeled sets quickly; filter by CLIP score and human spot checks.
- Rule of thumb: synthetic data is best for coverage of rare cases and bootstrapping, with real data for the final calibration of the distribution. Always evaluate on real held-out data.
Distribution shift and robustness
Section titled “Distribution shift and robustness”Covariate shift: changes, same (new camera, weather). Label shift: class priors change. Concept drift: changes (definition of “defect” updates). Spurious correlations (shortcuts): models use background or watermarks (Waterbirds benchmark; hospital tokens in X-rays; “cows on grass”) instead of the object.
Evaluation tools:
- Corruption benchmarks: ImageNet-C (noise, blur, weather, digital corruptions at 5 severities), ImageNet-P.
- Natural shift sets: ImageNet-V2, ImageNet-R (renditions), ImageNet-Sketch, ImageNet-A (adversarially filtered natural images), ObjectNet (unusual poses/backgrounds), WILDS (real-world shifts across domains). Zero-shot CLIP showed unusually small drops on such shifts relative to supervised models trained on ImageNet (Self-Supervised & Foundation Vision Models). Subsequent work indicates much of “effective robustness” tracks training-data diversity and scale.
- Slice-based evaluation: report metrics by condition (lighting, class, size, camera, demographic group). Aggregate accuracy hides the failure slices, which are often what matters for safety and fairness.
- Adversarial robustness: small perturbations can flip predictions (FGSM, PGD attacks); adversarial training helps but costs accuracy and compute. Relevant for security-critical applications; for most products, natural shift and corruption robustness are the higher-frequency risks.
- Out-of-distribution (OOD) detection: flag inputs unlike training data. Baselines: max softmax probability, energy score, Mahalanobis distance on features, kNN distance in embedding space (strong with foundation features). Calibration and selective prediction (abstain and send to human) are often the practical solution.
Improving robustness: more diverse data (the biggest lever), strong augmentation (AugMix, RandAugment, mix-based), pretrained/foundation backbones with careful fine-tuning (full fine-tuning can erode robustness; WiSE-FT interpolates fine-tuned and zero-shot weights to retain it), test-time adaptation (adapt BN statistics or minimize entropy on test batches), and monitoring in production.
Metrics
Section titled “Metrics”Classification
Section titled “Classification”- Accuracy misleads with imbalance. Use the confusion matrix, per-class precision/recall, F1, macro versus micro averaging, balanced accuracy.
- ROC-AUC versus PR-AUC: with rare positives, PR curves reflect the experience of using the detector (precision at a given recall); ROC-AUC can look great while precision is poor.
- Choose a threshold from costs: e.g., recall at 99% precision, or minimize expected cost .
- Top-k accuracy for many classes.
- Calibration: does confidence 0.8 mean 80% correct? Expected Calibration Error: bin predictions and average weighted by bin size. Modern deep nets are overconfident; temperature scaling (divide logits by a scalar fit on validation data) is a simple, effective fix. Brier score and reliability diagrams complement ECE.
import numpy as npdef ece(conf, correct, bins=15): edges = np.linspace(0, 1, bins + 1); e = 0.0 for lo, hi in zip(edges[:-1], edges[1:]): m = (conf > lo) & (conf <= hi) if m.any(): e += m.mean() * abs(correct[m].mean() - conf[m].mean()) return eDetection, segmentation, others
Section titled “Detection, segmentation, others”- Detection: mAP and COCO variants, precision/recall at a chosen operating threshold, per-size AP (Object Detection).
- Segmentation: mIoU, Dice, boundary metrics, PQ (Segmentation).
- Retrieval/embedding: Recall@K, mAP, nDCG; verification: TAR@FAR.
- Regression (depth, counting, keypoints): MAE/RMSE, AbsRel and for depth, OKS for keypoints, relative errors for counts.
- Generation: FID, CLIP-alignment, human preference (Diffusion & Image Generation); VLMs: task accuracy on private sets (Vision-Language Models).
- System metrics: latency, throughput, cost (Efficient & Edge Vision).
Statistical honesty
Section titled “Statistical honesty”- Report confidence intervals (bootstrap resampling of test images, grouped by independent unit). With a test set of 500 images, an accuracy difference of 1% is within noise.
- Use paired comparisons (McNemar’s test; paired bootstrap) when comparing two models on the same test set.
- Avoid test-set overfitting: repeated tuning against the same test set leaks it. Even benchmark communities show slow overfitting to popular test sets, and ImageNet-V2 showed accuracy drops of several points for the same models.
- Seeds: report mean and standard deviation over several training seeds for small differences.
- Beware metric-goal mismatch: optimize what the business cares about (e.g., defects missed per 1000 units), not the convenient benchmark.
Error analysis workflow
Section titled “Error analysis workflow”- Compute per-class and per-slice metrics.
- View the worst errors (highest-confidence mistakes) and borderline cases; cluster errors using embeddings (FiftyOne, UMAP) to find systematic modes.
- Classify error causes: label error, ambiguous class definition, data gap, resolution too low, model limitation, preprocessing bug.
- Fix the cause: relabel, collect targeted data, adjust augmentation, change resolution, or change the model.
- Add failing cases to a regression test set; re-evaluate.
This loop is detailed with code in Building CV Systems in Practice.
Key takeaways
Section titled “Key takeaways”- Specify the deployment distribution first; split by independent unit (patient, scene, time) to avoid leakage; deduplicate.
- Label quality is a ceiling: write guidelines, measure agreement, clean noisy labels, and use model-assisted labeling and active learning to cut cost.
- Augmentation encodes the invariances you want; apply label-preserving transforms only, jointly to image and annotations, and never to eval data. Mix-based methods and RandAugment are strong defaults.
- Synthetic data is best for rare-case coverage; always validate on real held-out data.
- Evaluate on slices and shifted sets, calibrate confidence, and report uncertainty. Aggregate accuracy on an i.i.d. test set is not the deployment metric.
Exercises
Section titled “Exercises”- Conceptual. You have 20,000 frames from 40 hours of video from 10 cameras. Propose a train/val/test split and explain what leakage a random frame split causes.
- Conceptual. A binary classifier for a rare defect (0.5% prevalence) has 99% accuracy and 0.97 ROC-AUC. Why might it still be useless? Which metric would you report?
- Hands-on. Train a small CNN on CIFAR-10 with and without RandAugment+CutMix and evaluate on CIFAR-10-C (or add your own Gaussian noise/blur corruptions). Compare clean versus corrupted accuracy.
- Hands-on. Compute ECE for a model before and after temperature scaling by fitting on a validation split via minimizing NLL.
- Conceptual. Two annotators label bounding boxes with mean IoU 0.82 between them. What does that imply for interpreting a model’s AP@0.9?
Solutions (sketch)
- Split by camera or by hour-long session (e.g., hold out whole sessions/cameras for val and test), because adjacent frames are nearly identical, so random splits let the model memorize backgrounds and report inflated accuracy.
- A model predicting “no defect” always already gets 99.5% accuracy; ROC-AUC ignores class imbalance in precision. Report precision-recall (PR-AUC), recall at a fixed precision, and the confusion matrix at the operating point.
- Expect similar clean accuracy but a clear robustness gap favoring augmentation, especially under noise/blur.
- Optimize a single scalar with LBFGS on validation logits; ECE typically drops sharply, accuracy unchanged.
- Human-human agreement bounds meaningful localization precision; AP at IoU 0.9 largely measures annotation noise, so emphasize AP50-AP75 or fix the annotation protocol.
Further reading
Section titled “Further reading”- Deng et al., “ImageNet: A Large-Scale Hierarchical Image Database” (2009); Lin et al., “Microsoft COCO” (2014).
- Shorten and Khoshgoftaar, “A survey on Image Data Augmentation for Deep Learning” (2019).
- Zhang et al., “mixup” (2017); Yun et al., “CutMix” (2019); Cubuk et al., “RandAugment” (2019).
- Hendrycks and Dietterich, “Benchmarking Neural Network Robustness to Common Corruptions and Perturbations” (ImageNet-C, 2019).
- Recht et al., “Do ImageNet Classifiers Generalize to ImageNet?” (2019).
- Koh et al., “WILDS: A Benchmark of in-the-Wild Distribution Shifts” (2021).
- Guo et al., “On Calibration of Modern Neural Networks” (2017).
- Northcutt, Athalye, Mueller, “Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks” (2021).
- Wortsman et al., “Robust fine-tuning of zero-shot models” (WiSE-FT, 2022).
- Tobin et al., “Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World” (2017).