Skip to content

Image Fundamentals & Classical CV Recap

What you’ll learn. How an image is represented as an array, how color spaces and filtering work, how classical edge and keypoint detectors (Canny, Harris, SIFT) were built, how a camera maps 3D points to pixels, and why the classical recipe (hand-designed features plus a shallow classifier) hit a ceiling. This is deliberately a recap: enough to read old papers, debug preprocessing, and understand which ideas survived into deep networks (convolution, gradients, multi-scale pyramids, geometry).

A digital image is a sampled, quantized measurement of light. A grayscale image is a matrix I∈{0,…,255}H×WI \in \{0,\dots,255\}^{H\times W}; a color image adds a channel axis: H×W×3H \times W \times 3 (NumPy/OpenCV/PIL layout, “channels last”) or 3×H×W3 \times H \times W (PyTorch layout, “channels first”). Mixing these conventions is the single most common source of silent bugs.

import numpy as np
from PIL import Image
import torch
img = Image.open("cat.jpg").convert("RGB")
a = np.asarray(img) # (H, W, 3), uint8
t = torch.from_numpy(a).permute(2, 0, 1) # (3, H, W)
t = t.float() / 255.0 # scale to [0, 1]

Practical facts that matter later:

  • Bit depth. 8-bit per channel is standard for web photos; medical, satellite and RAW camera data are 12–16 bit. Normalizing blindly by 255 destroys information for those.
  • OpenCV loads BGR, not RGB. Pretrained models expect RGB. Convert explicitly.
  • Resolution vs. field of view. Downsampling discards high-frequency content; always low-pass (area/antialiased interpolation) before decimation, or you get aliasing. Use Image.resize(..., Image.BILINEAR) with antialiasing, or torch.nn.functional.interpolate(..., antialias=True). Train/test resize mismatches can cost real accuracy.
  • Gamma. Most 8-bit images are sRGB, a nonlinear encoding roughly vdisplay=vlinear1/2.2v_\text{display} = v_\text{linear}^{1/2.2}. Averaging pixel values in sRGB space is not physically averaging light. Most ML pipelines ignore this, and that is usually fine; graphics and HDR pipelines do not.

RGB is a device-oriented encoding. Other spaces separate quantities that are useful for specific tasks.

Space Components Typical use
RGB red, green, blue storage, display, CNN input
HSV / HSL hue, saturation, value color thresholding, augmentation
YCbCr luma + two chroma JPEG, video codecs; chroma is subsampled because eyes are less sensitive to it
CIE Lab lightness + two opponent axes perceptually more uniform distances
Grayscale luminance Y≈0.299R+0.587G+0.114BY \approx 0.299R + 0.587G + 0.114B

JPEG compresses by converting to YCbCr, subsampling chroma, splitting into 8×88\times8 blocks, applying a DCT, quantizing coefficients, and entropy coding. This leaves characteristic blocking artifacts that become a distribution-shift source for models trained on clean images (see Data, Augmentation & Evaluation).

The workhorse of classical vision is linear filtering. Cross-correlation of an image II with a kernel KK of size (2k+1)2(2k+1)^2:

(I⋆K)(x,y)=∑i=−kk∑j=−kkK(i,j) I(x+i, y+j).(I \star K)(x,y) = \sum_{i=-k}^{k}\sum_{j=-k}^{k} K(i,j)\, I(x+i,\, y+j).

Strictly, convolution flips the kernel; deep learning libraries implement cross-correlation and call it convolution. For learned filters the distinction is irrelevant.

Key kernels:

  • Box filter: average over a window. Cheap blur, but poor frequency response.
  • Gaussian Gσ(x,y)=12πσ2e−(x2+y2)/2σ2G_\sigma(x,y) = \frac{1}{2\pi\sigma^2} e^{-(x^2+y^2)/2\sigma^2}: the canonical blur. It is separable: a 2D Gaussian equals two 1D passes, reducing cost from O(k2)O(k^2) to O(k)O(k) per pixel. Blurring with σ\sigma then subsampling is the basis of image pyramids.
  • Sobel / Prewitt: finite-difference derivative approximations combined with smoothing.
  • Laplacian of Gaussian (LoG): second derivative; zero crossings mark edges; blob detector at the right scale.
  • Median filter: nonlinear; removes salt-and-pepper noise while preserving edges.
import numpy as np
from scipy.signal import convolve2d
sobel_x = np.array([[-1, 0, 1],
[-2, 0, 2],
[-1, 0, 1]], dtype=np.float32)
sobel_y = sobel_x.T
def gradients(gray):
gx = convolve2d(gray, sobel_x, mode="same", boundary="symm")
gy = convolve2d(gray, sobel_y, mode="same", boundary="symm")
mag = np.hypot(gx, gy)
ang = np.arctan2(gy, gx)
return mag, ang

The Fourier view helps intuition: convolution in space is multiplication in frequency. Gaussians are low-pass; derivative filters are high-pass. This is why CNNs trained on natural images learn first-layer filters that look like oriented edge detectors (Gabor-like) and color blobs, which are the same structures hand-designed 40 years earlier.

Edges are locations of rapid intensity change, i.e. large gradient magnitude ∥∇I∥\|\nabla I\|. Noise is amplified by differentiation, so smooth first. The Canny detector (1986) remains a standard:

  1. Gaussian smoothing.
  2. Gradient magnitude and direction (Sobel).
  3. Non-maximum suppression: keep a pixel only if its magnitude is a local maximum along the gradient direction, thinning edges to one pixel width.
  4. Hysteresis thresholding: two thresholds Thigh>TlowT_\text{high} > T_\text{low}. Pixels above ThighT_\text{high} are strong edges; pixels between are kept only if connected to a strong edge.

Note that step 3 resembles the NMS used in object detection (Object Detection): the same “suppress non-peak neighbors” idea reappears.

import cv2
gray = cv2.imread("scene.jpg", cv2.IMREAD_GRAYSCALE)
edges = cv2.Canny(gray, threshold1=50, threshold2=150)

Edges are ambiguous along their direction (the aperture problem). A corner is a point where the image changes in all directions, hence it can be localized precisely and matched across views.

Harris corner detector. Consider the change in appearance when shifting a window by (u,v)(u,v):

E(u,v)=∑x,yw(x,y) [I(x+u,y+v)−I(x,y)]2≈[uv]M[uv],M=∑w[Ix2IxIyIxIyIy2].E(u,v) = \sum_{x,y} w(x,y)\,[I(x+u,y+v) - I(x,y)]^2 \approx \begin{bmatrix} u & v \end{bmatrix} M \begin{bmatrix} u \\ v \end{bmatrix}, \quad M = \sum w \begin{bmatrix} I_x^2 & I_xI_y \\ I_xI_y & I_y^2 \end{bmatrix}.

If both eigenvalues of the structure tensor MM are large, the point is a corner; one large eigenvalue means an edge; none means a flat region. Harris uses the response R=det⁡M−κ (tr⁡M)2R = \det M - \kappa\,(\operatorname{tr} M)^2 to avoid computing eigenvalues.

A detector must find the same point when the object is closer or farther. Scale-space theory says: blur the image with a family of Gaussians of increasing σ\sigma, and look for extrema across space and scale.

SIFT (Lowe, 1999/2004) is the culmination of the hand-crafted local feature era. The idea:

  1. Detect keypoints as extrema of the Difference of Gaussians D(x,y,σ)=Gkσ⋆I−Gσ⋆ID(x,y,\sigma) = G_{k\sigma}\star I - G_\sigma\star I (an efficient LoG approximation) across a pyramid.
  2. Localize and filter: refine to sub-pixel/sub-scale, reject low-contrast points and edge-like responses.
  3. Orientation: compute a dominant gradient orientation in the neighborhood and rotate the frame to it, giving rotation invariance.
  4. Describe: in a 16×1616\times16 patch around the keypoint, split into 4×44\times4 cells, build an 8-bin gradient-orientation histogram in each: 4⋅4⋅8=1284\cdot4\cdot8 = 128 dimensions. Normalize (and clip) for illumination robustness.
  5. Match descriptors with nearest neighbor plus the ratio test (accept if best distance < ~0.75 × second-best).
import cv2
sift = cv2.SIFT_create()
kp1, d1 = sift.detectAndCompute(img1, None)
kp2, d2 = sift.detectAndCompute(img2, None)
matches = cv2.BFMatcher().knnMatch(d1, d2, k=2)
good = [m for m, n in matches if m.distance < 0.75 * n.distance]

Then RANSAC estimates a geometric model (homography or fundamental matrix) while rejecting outlier matches. SIFT-style pipelines (and descendants such as ORB for speed) are still the backbone of structure-from-motion software like COLMAP, though learned features (SuperPoint, LightGlue-style matchers) increasingly replace them.

A parallel lineage used descriptors for recognition: HOG (histograms of oriented gradients) with a linear SVM won pedestrian detection; bag of visual words (quantize SIFT descriptors, build a histogram) with SVMs dominated image classification until about 2012.

The pinhole camera maps a 3D point X=(X,Y,Z)⊤\mathbf{X} = (X,Y,Z)^\top in camera coordinates to the image plane:

x=f XZ+cx,y=f YZ+cy.x = f\,\frac{X}{Z} + c_x,\qquad y = f\,\frac{Y}{Z} + c_y.

In homogeneous coordinates with the intrinsic matrix KK and extrinsics [R ∣ t][R\,|\,\mathbf{t}] (world to camera):

λ[uv1]=K [R ∣ t][XwYwZw1],K=[fxscx0fycy001].\lambda \begin{bmatrix} u \\ v \\ 1 \end{bmatrix} = K\,[R\,|\,\mathbf{t}] \begin{bmatrix} X_w \\ Y_w \\ Z_w \\ 1 \end{bmatrix}, \qquad K = \begin{bmatrix} f_x & s & c_x \\ 0 & f_y & c_y \\ 0 & 0 & 1 \end{bmatrix}.

Consequences you will meet again:

  • Depth is lost by projection: all points along a ray map to one pixel. Recovering it needs a second view (stereo, motion), a prior (learned monocular depth, see Video, 3D & Spatial Vision), or an active sensor.
  • Lens distortion (radial/tangential) bends straight lines; calibration with a checkerboard estimates KK and distortion coefficients (cv2.calibrateCamera).
  • Epipolar geometry: for two views, a point in image 1 constrains its match in image 2 to a line, expressed by the fundamental matrix FF with x′⊤Fx=0\mathbf{x}'^\top F \mathbf{x} = 0 (or essential matrix E=K′⊤FKE = K'^\top F K when calibrated). Triangulation intersects the rays to recover 3D.
  • Homography: a planar scene (or pure rotation) is related between views by a 3×33\times3 matrix HH, the basis of panorama stitching.

NeRF and Gaussian splatting later in this section are, at heart, differentiable versions of this same projection model.

By 2010 the standard recipe was: hand-design features (SIFT, HOG, SURF), aggregate (bag of words, Fisher vectors, spatial pyramids), train an SVM. Each stage was engineered separately, and the pipeline’s information bottleneck was the features chosen by humans.

Three things changed:

  1. Data. ImageNet (about 1.2M training images, 1000 classes, from 2009/ILSVRC 2010–2012) gave enough labeled examples to fit millions of parameters.
  2. Compute. GPUs made convolution over large batches cheap.
  3. End-to-end learning with the right inductive bias. A convolutional network learns the filters, the aggregation, and the classifier jointly, optimizing the actual task loss. In 2012 AlexNet cut top-5 error from roughly 26% (best classical entry) to about 16%, a gap so large the field pivoted within two years. The story continues in CNN Architectures That Mattered; for the mechanics of convolutions see CNNs and RNNs.

What classical vision still offers:

  • Geometry (calibration, SfM, SLAM, multi-view stereo) is mostly not replaced, only augmented.
  • Fast, label-free preprocessing: resizing, denoising, color correction, morphological operations.
  • Interpretability and guarantees in industrial inspection (a Hough circle fit or template match can be better than a model when conditions are fixed).
  • A mental model: pyramids become feature pyramids (FPN), gradient histograms became learned filters, RANSAC remains in the loop for geometric verification.
  • Mixing BGR/RGB, uint8/float, HWC/CHW; double-normalizing.
  • Resizing with different interpolation at train and inference time.
  • Forgetting EXIF orientation: phone photos may be stored rotated with a tag. PIL.ImageOps.exif_transpose fixes it.
  • Treating pixel-space L2 distance as perceptual similarity.
  • An image is a sampled array; layout, dtype, color order and resize method are part of the model’s contract.
  • Linear filtering (convolution) with Gaussians and derivative kernels underlies edge detection; Canny adds NMS and hysteresis.
  • Local features (Harris, SIFT) detect repeatable points in scale space and describe them with gradient histograms; RANSAC verifies geometry.
  • The pinhole model λx=K[R∣t]X\lambda\mathbf{x} = K[R|\mathbf{t}]\mathbf{X} explains depth ambiguity, calibration, epipolar geometry and homographies, and still underlies modern 3D methods.
  • Deep learning won by learning features end-to-end given large data and GPUs; geometry and careful preprocessing remain valuable.
  1. Conceptual. Why is a Gaussian blur separable, and what is the speedup for a 15×1515\times15 kernel on an H×WH\times W image?
  2. Conceptual. Explain in terms of the eigenvalues of the structure tensor why a corner is easier to match than an edge point.
  3. Hands-on. Implement Sobel gradient magnitude in NumPy and compare against cv2.Sobel. Then add Gaussian noise (σ=20\sigma=20) and show how pre-smoothing changes the result.
  4. Hands-on. Take two photos of the same scene, match SIFT features with the ratio test, estimate a homography with cv2.findHomography(..., cv2.RANSAC) and report the inlier ratio.
  5. Conceptual. A pinhole camera has fx=fy=800f_x=f_y=800 px. An object 0.5 m wide appears 100 px wide. How far away is it?
Solutions (sketch)
  1. G(x,y)=g(x)g(y)G(x,y)=g(x)g(y), so convolve rows then columns. Cost per pixel drops from 152=22515^2=225 to 2⋅15=302\cdot15=30 multiply-adds, about 7.5x.
  2. At an edge one eigenvalue is near zero, so shifting along the edge leaves EE unchanged and the position along the edge is undetermined. At a corner both are large and EE increases in every direction.
  3. Check borders and dtype (use float32); the noisy gradient map should be much cleaner after smoothing.
  4. Inlier ratio depends on scene planarity; non-planar scenes give low homography inlier counts.
  5. Z=f X/x=800⋅0.5/100=4Z = f\,X/x = 800 \cdot 0.5 / 100 = 4 m.
  • Richard Szeliski, Computer Vision: Algorithms and Applications (2nd ed., 2022).
  • Forsyth and Ponce, Computer Vision: A Modern Approach.
  • Hartley and Zisserman, Multiple View Geometry in Computer Vision.
  • David Lowe, “Distinctive Image Features from Scale-Invariant Keypoints” (IJCV, 2004).
  • John Canny, “A Computational Approach to Edge Detection” (1986).
  • Krizhevsky, Sutskever, Hinton, “ImageNet Classification with Deep Convolutional Neural Networks” (2012).
  • OpenCV documentation and tutorials.