Image Fundamentals & Classical CV Recap
What you’ll learn. How an image is represented as an array, how color spaces and filtering work, how classical edge and keypoint detectors (Canny, Harris, SIFT) were built, how a camera maps 3D points to pixels, and why the classical recipe (hand-designed features plus a shallow classifier) hit a ceiling. This is deliberately a recap: enough to read old papers, debug preprocessing, and understand which ideas survived into deep networks (convolution, gradients, multi-scale pyramids, geometry).
Images as arrays
Section titled “Images as arrays”A digital image is a sampled, quantized measurement of light. A grayscale image is a matrix ; a color image adds a channel axis: (NumPy/OpenCV/PIL layout, “channels last”) or (PyTorch layout, “channels first”). Mixing these conventions is the single most common source of silent bugs.
import numpy as npfrom PIL import Imageimport torch
img = Image.open("cat.jpg").convert("RGB")a = np.asarray(img) # (H, W, 3), uint8t = torch.from_numpy(a).permute(2, 0, 1) # (3, H, W)t = t.float() / 255.0 # scale to [0, 1]Practical facts that matter later:
- Bit depth. 8-bit per channel is standard for web photos; medical, satellite and RAW camera data are 12–16 bit. Normalizing blindly by 255 destroys information for those.
- OpenCV loads BGR, not RGB. Pretrained models expect RGB. Convert explicitly.
- Resolution vs. field of view. Downsampling discards high-frequency content; always low-pass (area/antialiased interpolation) before decimation, or you get aliasing. Use
Image.resize(..., Image.BILINEAR)with antialiasing, ortorch.nn.functional.interpolate(..., antialias=True). Train/test resize mismatches can cost real accuracy. - Gamma. Most 8-bit images are sRGB, a nonlinear encoding roughly . Averaging pixel values in sRGB space is not physically averaging light. Most ML pipelines ignore this, and that is usually fine; graphics and HDR pipelines do not.
Color spaces
Section titled “Color spaces”RGB is a device-oriented encoding. Other spaces separate quantities that are useful for specific tasks.
| Space | Components | Typical use |
|---|---|---|
| RGB | red, green, blue | storage, display, CNN input |
| HSV / HSL | hue, saturation, value | color thresholding, augmentation |
| YCbCr | luma + two chroma | JPEG, video codecs; chroma is subsampled because eyes are less sensitive to it |
| CIE Lab | lightness + two opponent axes | perceptually more uniform distances |
| Grayscale | luminance |
JPEG compresses by converting to YCbCr, subsampling chroma, splitting into blocks, applying a DCT, quantizing coefficients, and entropy coding. This leaves characteristic blocking artifacts that become a distribution-shift source for models trained on clean images (see Data, Augmentation & Evaluation).
Filtering and convolution
Section titled “Filtering and convolution”The workhorse of classical vision is linear filtering. Cross-correlation of an image with a kernel of size :
Strictly, convolution flips the kernel; deep learning libraries implement cross-correlation and call it convolution. For learned filters the distinction is irrelevant.
Key kernels:
- Box filter: average over a window. Cheap blur, but poor frequency response.
- Gaussian : the canonical blur. It is separable: a 2D Gaussian equals two 1D passes, reducing cost from to per pixel. Blurring with then subsampling is the basis of image pyramids.
- Sobel / Prewitt: finite-difference derivative approximations combined with smoothing.
- Laplacian of Gaussian (LoG): second derivative; zero crossings mark edges; blob detector at the right scale.
- Median filter: nonlinear; removes salt-and-pepper noise while preserving edges.
import numpy as npfrom scipy.signal import convolve2d
sobel_x = np.array([[-1, 0, 1], [-2, 0, 2], [-1, 0, 1]], dtype=np.float32)sobel_y = sobel_x.T
def gradients(gray): gx = convolve2d(gray, sobel_x, mode="same", boundary="symm") gy = convolve2d(gray, sobel_y, mode="same", boundary="symm") mag = np.hypot(gx, gy) ang = np.arctan2(gy, gx) return mag, angThe Fourier view helps intuition: convolution in space is multiplication in frequency. Gaussians are low-pass; derivative filters are high-pass. This is why CNNs trained on natural images learn first-layer filters that look like oriented edge detectors (Gabor-like) and color blobs, which are the same structures hand-designed 40 years earlier.
Edges: from gradients to Canny
Section titled “Edges: from gradients to Canny”Edges are locations of rapid intensity change, i.e. large gradient magnitude . Noise is amplified by differentiation, so smooth first. The Canny detector (1986) remains a standard:
- Gaussian smoothing.
- Gradient magnitude and direction (Sobel).
- Non-maximum suppression: keep a pixel only if its magnitude is a local maximum along the gradient direction, thinning edges to one pixel width.
- Hysteresis thresholding: two thresholds . Pixels above are strong edges; pixels between are kept only if connected to a strong edge.
Note that step 3 resembles the NMS used in object detection (Object Detection): the same “suppress non-peak neighbors” idea reappears.
import cv2gray = cv2.imread("scene.jpg", cv2.IMREAD_GRAYSCALE)edges = cv2.Canny(gray, threshold1=50, threshold2=150)Corners and the idea of a local feature
Section titled “Corners and the idea of a local feature”Edges are ambiguous along their direction (the aperture problem). A corner is a point where the image changes in all directions, hence it can be localized precisely and matched across views.
Harris corner detector. Consider the change in appearance when shifting a window by :
If both eigenvalues of the structure tensor are large, the point is a corner; one large eigenvalue means an edge; none means a flat region. Harris uses the response to avoid computing eigenvalues.
Scale space and SIFT
Section titled “Scale space and SIFT”A detector must find the same point when the object is closer or farther. Scale-space theory says: blur the image with a family of Gaussians of increasing , and look for extrema across space and scale.
SIFT (Lowe, 1999/2004) is the culmination of the hand-crafted local feature era. The idea:
- Detect keypoints as extrema of the Difference of Gaussians (an efficient LoG approximation) across a pyramid.
- Localize and filter: refine to sub-pixel/sub-scale, reject low-contrast points and edge-like responses.
- Orientation: compute a dominant gradient orientation in the neighborhood and rotate the frame to it, giving rotation invariance.
- Describe: in a patch around the keypoint, split into cells, build an 8-bin gradient-orientation histogram in each: dimensions. Normalize (and clip) for illumination robustness.
- Match descriptors with nearest neighbor plus the ratio test (accept if best distance < ~0.75 × second-best).
import cv2sift = cv2.SIFT_create()kp1, d1 = sift.detectAndCompute(img1, None)kp2, d2 = sift.detectAndCompute(img2, None)matches = cv2.BFMatcher().knnMatch(d1, d2, k=2)good = [m for m, n in matches if m.distance < 0.75 * n.distance]Then RANSAC estimates a geometric model (homography or fundamental matrix) while rejecting outlier matches. SIFT-style pipelines (and descendants such as ORB for speed) are still the backbone of structure-from-motion software like COLMAP, though learned features (SuperPoint, LightGlue-style matchers) increasingly replace them.
A parallel lineage used descriptors for recognition: HOG (histograms of oriented gradients) with a linear SVM won pedestrian detection; bag of visual words (quantize SIFT descriptors, build a histogram) with SVMs dominated image classification until about 2012.
Camera geometry basics
Section titled “Camera geometry basics”The pinhole camera maps a 3D point in camera coordinates to the image plane:
In homogeneous coordinates with the intrinsic matrix and extrinsics (world to camera):
Consequences you will meet again:
- Depth is lost by projection: all points along a ray map to one pixel. Recovering it needs a second view (stereo, motion), a prior (learned monocular depth, see Video, 3D & Spatial Vision), or an active sensor.
- Lens distortion (radial/tangential) bends straight lines; calibration with a checkerboard estimates and distortion coefficients (
cv2.calibrateCamera). - Epipolar geometry: for two views, a point in image 1 constrains its match in image 2 to a line, expressed by the fundamental matrix with (or essential matrix when calibrated). Triangulation intersects the rays to recover 3D.
- Homography: a planar scene (or pure rotation) is related between views by a matrix , the basis of panorama stitching.
NeRF and Gaussian splatting later in this section are, at heart, differentiable versions of this same projection model.
Why deep learning won
Section titled “Why deep learning won”By 2010 the standard recipe was: hand-design features (SIFT, HOG, SURF), aggregate (bag of words, Fisher vectors, spatial pyramids), train an SVM. Each stage was engineered separately, and the pipeline’s information bottleneck was the features chosen by humans.
Three things changed:
- Data. ImageNet (about 1.2M training images, 1000 classes, from 2009/ILSVRC 2010–2012) gave enough labeled examples to fit millions of parameters.
- Compute. GPUs made convolution over large batches cheap.
- End-to-end learning with the right inductive bias. A convolutional network learns the filters, the aggregation, and the classifier jointly, optimizing the actual task loss. In 2012 AlexNet cut top-5 error from roughly 26% (best classical entry) to about 16%, a gap so large the field pivoted within two years. The story continues in CNN Architectures That Mattered; for the mechanics of convolutions see CNNs and RNNs.
What classical vision still offers:
- Geometry (calibration, SfM, SLAM, multi-view stereo) is mostly not replaced, only augmented.
- Fast, label-free preprocessing: resizing, denoising, color correction, morphological operations.
- Interpretability and guarantees in industrial inspection (a Hough circle fit or template match can be better than a model when conditions are fixed).
- A mental model: pyramids become feature pyramids (FPN), gradient histograms became learned filters, RANSAC remains in the loop for geometric verification.
Pitfalls
Section titled “Pitfalls”- Mixing BGR/RGB, uint8/float, HWC/CHW; double-normalizing.
- Resizing with different interpolation at train and inference time.
- Forgetting EXIF orientation: phone photos may be stored rotated with a tag.
PIL.ImageOps.exif_transposefixes it. - Treating pixel-space L2 distance as perceptual similarity.
Key takeaways
Section titled “Key takeaways”- An image is a sampled array; layout, dtype, color order and resize method are part of the model’s contract.
- Linear filtering (convolution) with Gaussians and derivative kernels underlies edge detection; Canny adds NMS and hysteresis.
- Local features (Harris, SIFT) detect repeatable points in scale space and describe them with gradient histograms; RANSAC verifies geometry.
- The pinhole model explains depth ambiguity, calibration, epipolar geometry and homographies, and still underlies modern 3D methods.
- Deep learning won by learning features end-to-end given large data and GPUs; geometry and careful preprocessing remain valuable.
Exercises
Section titled “Exercises”- Conceptual. Why is a Gaussian blur separable, and what is the speedup for a kernel on an image?
- Conceptual. Explain in terms of the eigenvalues of the structure tensor why a corner is easier to match than an edge point.
- Hands-on. Implement Sobel gradient magnitude in NumPy and compare against
cv2.Sobel. Then add Gaussian noise () and show how pre-smoothing changes the result. - Hands-on. Take two photos of the same scene, match SIFT features with the ratio test, estimate a homography with
cv2.findHomography(..., cv2.RANSAC)and report the inlier ratio. - Conceptual. A pinhole camera has px. An object 0.5 m wide appears 100 px wide. How far away is it?
Solutions (sketch)
- , so convolve rows then columns. Cost per pixel drops from to multiply-adds, about 7.5x.
- At an edge one eigenvalue is near zero, so shifting along the edge leaves unchanged and the position along the edge is undetermined. At a corner both are large and increases in every direction.
- Check borders and dtype (use float32); the noisy gradient map should be much cleaner after smoothing.
- Inlier ratio depends on scene planarity; non-planar scenes give low homography inlier counts.
- m.
Further reading
Section titled “Further reading”- Richard Szeliski, Computer Vision: Algorithms and Applications (2nd ed., 2022).
- Forsyth and Ponce, Computer Vision: A Modern Approach.
- Hartley and Zisserman, Multiple View Geometry in Computer Vision.
- David Lowe, “Distinctive Image Features from Scale-Invariant Keypoints” (IJCV, 2004).
- John Canny, “A Computational Approach to Edge Detection” (1986).
- Krizhevsky, Sutskever, Hinton, “ImageNet Classification with Deep Convolutional Neural Networks” (2012).
- OpenCV documentation and tutorials.