Skip to content

Video, 3D & Spatial Vision

What you’ll learn. How vision extends along time (video classification, action recognition, video SSL, tracking) and into 3D (depth estimation, NeRF, 3D Gaussian splatting, point clouds, human and object pose). For each: the core idea, the math that matters, a representative code sketch and the current state of practice. Prerequisites: camera geometry from Image Fundamentals, transformers from Vision Transformers.

A video is a tensor T×H×W×3T\times H\times W\times 3. Challenges: huge compute (a 10 s clip at 30 fps is 300 frames), temporal redundancy, and motion cues that single frames lack.

  • Two-stream networks: RGB frame stream plus optical-flow stream (motion), fused late.
  • C3D / I3D: 3D convolutions (kt×kh×kwk_t\times k_h\times k_w); I3D “inflates” pretrained 2D ImageNet kernels along time. SlowFast: a slow pathway (low frame rate, high channel capacity) captures appearance; a fast pathway (high frame rate, few channels) captures motion.
  • Factorized (2+1)D convolutions: spatial then temporal convolution, cheaper and often better.
  • Video transformers: tokenization extends ViT patches to space-time tubelets (e.g., 2×16×162\times16\times16). Full space-time attention is quadratic in T⋅NT\cdot N, so designs factorize: TimeSformer (divided space-time attention, alternating temporal and spatial attention), ViViT (factorized encoders), Video Swin (3D shifted windows), MViT (multiscale pooling attention).
  • Video self-supervision: VideoMAE / VideoMAE V2 mask 90-95% of space-time tubes (video is highly redundant) and reconstruct; V-JEPA predicts masked latents (Self-Supervised & Foundation Vision Models). Pretrained video backbones then fine-tune on Kinetics, Something-Something (which needs temporal reasoning) and so on.
  • Video-language models: sample frames (e.g., 8-64), encode each with an image encoder, compress tokens, feed to an LLM (Vision-Language Models). Token budget is the central constraint, so approaches include frame sampling by relevance, token merging across time, and long-context LLMs with streaming memory. Long-video understanding remains a weak spot compared with images.
# naive uniform frame sampling and tubelet embedding
import torch, torch.nn as nn
def sample_frames(video, n): # video: (T,H,W,3) uint8 tensor
idx = torch.linspace(0, video.size(0) - 1, n).long()
return video[idx]
class TubeletEmbed(nn.Module):
def __init__(self, dim=768, t=2, p=16):
super().__init__()
self.proj = nn.Conv3d(3, dim, kernel_size=(t, p, p), stride=(t, p, p))
def forward(self, x): # (B,3,T,H,W)
return self.proj(x).flatten(2).transpose(1, 2) # (B, N_tokens, dim)

Optical flow estimates per-pixel motion between frames. Classical: Horn-Schunck and Lucas-Kanade (brightness constancy Ixu+Iyv+It=0I_x u+I_y v+I_t=0 plus smoothness or local windows). Deep: FlowNet, RAFT (all-pairs correlation volume with iterative GRU refinement), followed by transformer-based refinements (GMFlow and others). Flow is used for stabilization, video interpolation and as a cue in tracking.

  • Single object tracking (SOT): given a box in frame 1, follow it. Siamese trackers (SiamRPN) matched template features; modern trackers (OSTrack, ViT-based; and SAM 2-based trackers) use joint attention between template and search region. SAM 2’s memory bank handles occlusion and re-appearance with mask prompts (Segmentation).
  • Multi-object tracking (MOT): tracking-by-detection. Detect per frame (Object Detection), associate with existing tracks via motion model (Kalman filter) and appearance embeddings; assignment via the Hungarian algorithm over a cost of IoU/embedding distance. SORT (Kalman + IoU), DeepSORT (adds appearance embeddings), ByteTrack (also associates low-confidence detections, which recovers occluded objects), BoT-SORT, OC-SORT. End-to-end query-based trackers (MOTR, TrackFormer) carry track queries across frames, DETR-style.
  • Point tracking: TAP-Vid / TAPIR / CoTracker track arbitrary points through video (dense correspondence in time), useful for robotics and motion capture; TrackAnything-style combos pair a segmenter with a tracker.
  • Metrics: MOTA, IDF1, HOTA (balances detection and association). Identity switches matter more than box accuracy in many applications.

Recall that projection discards depth (Image Fundamentals). A network can still infer plausible depth from learned priors (perspective cues, object sizes, texture gradients).

  • Metric vs relative depth. Metric depth is in meters (needs camera intrinsics or learned scale, accurate within a domain, e.g., KITTI driving, NYU indoors). Relative / affine-invariant depth is correct up to unknown scale and shift per image, which is enough for many graphics and perception tasks and generalizes across domains.
  • MiDaS trained on a mixture of datasets with a scale-and-shift-invariant loss. DPT applied ViT backbones with a dense decoder.
  • Depth Anything (v1/v2): trains on large amounts of pseudo-labeled images using a teacher, with DINOv2 encoders (Self-Supervised & Foundation Vision Models) and a DPT-like decoder; v2 emphasized synthetic data for fine detail and distillation into small variants. Metric-depth fine-tunes and models like ZoeDepth, Metric3D, UniDepth, Depth Pro (Apple) and others add scale; later generations (Depth Anything 3, MoGe-style geometry models, VGGT) predict depth, camera pose and point maps feed-forward from one or more images. Model names and rankings turn over quickly in this area.
  • Diffusion-based depth (Marigold) repurposes a text-to-image diffusion prior fine-tuned for depth, with high detail but slower (Diffusion & Image Generation).
  • Multi-view stereo / feed-forward reconstruction: classical stereo matching via cost volumes (plane sweep, SGM, learned: PSMNet, RAFT-Stereo); DUSt3R / MASt3R regress pointmaps from image pairs without known poses or intrinsics, and VGGT extends to many views in a single feed-forward pass, replacing parts of the SfM/MVS pipeline for some uses.
# relative depth with the HuggingFace pipeline (checkpoint name is an example; check current options)
from transformers import pipeline
from PIL import Image
depth = pipeline("depth-estimation", model="depth-anything/Depth-Anything-V2-Small-hf")
out = depth(Image.open("room.jpg"))
out["depth"].save("depth.png") # relative (inverse-depth-like) map for visualization

Goal: novel-view synthesis from posed photos. NeRF (Mildenhall et al., 2020) represents a scene as a function Fθ:(x,d)↦(c,σ)F_\theta:(\mathbf{x},\mathbf{d})\mapsto(\mathbf{c},\sigma) from 3D position and viewing direction to color and volume density, parameterized by an MLP.

Volume rendering. For a camera ray r(s)=o+sd\mathbf{r}(s)=\mathbf{o}+s\mathbf{d}, the expected color is

C(r)=∫snsfT(s) σ(r(s)) c(r(s),d) ds,T(s)=exp⁡(−∫snsσ(r(u)) du),C(\mathbf{r})=\int_{s_n}^{s_f}T(s)\,\sigma(\mathbf{r}(s))\,\mathbf{c}(\mathbf{r}(s),\mathbf{d})\,ds,\qquad T(s)=\exp\Big(-\int_{s_n}^{s}\sigma(\mathbf{r}(u))\,du\Big),

with transmittance T(s)T(s) the probability the ray reaches ss without being absorbed. In practice, sample NN points along the ray and use the quadrature

C^=∑i=1NTi αi ci,αi=1−e−σiδi,Ti=∏j<i(1−αj).\hat C=\sum_{i=1}^N T_i\,\alpha_i\,\mathbf{c}_i,\quad \alpha_i=1-e^{-\sigma_i\delta_i},\quad T_i=\prod_{j<i}(1-\alpha_j).

Train by minimizing photometric error ∥C^−Cgt∥2\|\hat C-C_\text{gt}\|^2 over sampled rays; everything is differentiable. Ingredients that made it work: positional encoding γ(p)=(sin⁡(2kπp),cos⁡(2kπp))k=0L−1\gamma(p)=(\sin(2^kπp),\cos(2^kπp))_{k=0}^{L-1} (to overcome the spectral bias of MLPs toward low frequencies), view-dependent color for specularity, and hierarchical (coarse-to-fine) sampling. Camera poses come from COLMAP (SfM).

def composite(sigmas, colors, deltas): # sigmas (R,N), colors (R,N,3), deltas (R,N)
alpha = 1 - torch.exp(-sigmas * deltas)
T = torch.cumprod(torch.cat([torch.ones_like(alpha[:, :1]), 1 - alpha + 1e-10], 1), 1)[:, :-1]
w = T * alpha # (R,N) weights
return (w.unsqueeze(-1) * colors).sum(1), w

Vanilla NeRF takes hours to days per scene and seconds per frame. Accelerations replaced the big MLP with spatial data structures: Instant-NGP (multiresolution hash grid + tiny MLP, training in minutes), Plenoxels, TensoRF; Mip-NeRF / Mip-NeRF 360 (anti-aliasing with cone tracing, unbounded scenes); Zip-NeRF. Variants handle dynamic scenes, relighting and surface extraction (NeuS, VolSDF; signed-distance parametrizations).

3D Gaussian Splatting (3DGS) (Kerbl et al., 2023) replaced implicit fields with an explicit set of millions of anisotropic 3D Gaussians, rendered by rasterization rather than ray marching, so it reaches real-time rendering (100+ fps at 1080p on a good GPU) with quality comparable or better than the best NeRFs and training in minutes.

Each Gaussian has: mean μ∈R3\boldsymbol\mu\in\mathbb{R}^3, covariance Σ=RSS⊤R⊤\Sigma=R S S^\top R^\top (rotation quaternion RR, scale SS for positive semi-definiteness), opacity α\alpha, and view-dependent color via spherical harmonics.

Rendering:

  1. Project each 3D Gaussian to a 2D Gaussian: Σ′=JWΣW⊤J⊤\Sigma' = J W \Sigma W^\top J^\top with viewing transform WW and the Jacobian JJ of the local affine approximation of perspective projection.
  2. Sort Gaussians by depth per screen tile (GPU radix sort).
  3. Alpha-composite front to back: C=∑ici αi∏j<i(1−αj)C=\sum_i c_i\,\alpha_i\prod_{j<i}(1-\alpha_j), the same compositing form as NeRF’s quadrature.

Optimization: initialize from the sparse SfM point cloud; optimize all parameters by gradient descent on an L1 + D-SSIM photometric loss; adaptive density control periodically clones Gaussians in under-reconstructed regions, splits large ones, and prunes near-transparent ones.

Strengths: speed, editability (explicit primitives), good fidelity. Weaknesses: large memory (hundreds of MB to GB per scene; compression research active), floaters and artifacts away from training views, geometry is less clean than SDF methods (2DGS, surface-regularized variants improve it), and it requires reasonably accurate camera poses. Extensions: dynamic 4D Gaussians, SLAM with Gaussians, feed-forward predictors (pixelSplat, MVSplat) that produce Gaussians from a few images without per-scene optimization, and generative 3D pipelines (text/image to 3D via diffusion priors, with Gaussians or meshes as output). Tools: Nerfstudio, gsplat, COLMAP, and the original reference implementation (check licenses before commercial use).

LiDAR and depth sensors produce unordered sets of points {(x,y,z,features)}\{(x,y,z,\text{features})\}. Networks must be permutation invariant (and ideally robust to sampling density and rigid transforms).

  • PointNet (2017): shared MLP per point followed by a symmetric function (max-pool) to get a global feature: f({xi})=γ(max⁡ih(xi))f(\{x_i\})=\gamma(\max_i h(x_i)). Simple but ignores local structure. PointNet++ adds hierarchical local grouping (sampling centroids with farthest point sampling, grouping neighbors, applying PointNet).
  • Graph/convolutional: DGCNN (EdgeConv on k-NN graphs), KPConv (kernel point convolution).
  • Voxel and sparse convolution: voxelize and use sparse 3D convolutions (MinkowskiEngine, spconv), efficient since most voxels are empty. VoxelNet/SECOND for detection; PointPillars (pillars to a pseudo-image processed by a 2D CNN, fast, widely deployed in autonomous driving), CenterPoint (center-based 3D detection and tracking).
  • Point transformers: Point Transformer V1-V3 (V3 serializes points along space-filling curves for efficient attention), and large-scale pretraining (Sonata and others) now apply SSL ideas to 3D.
  • BEV (bird’s-eye-view) perception for driving: fuse multi-camera images into a BEV grid (LSS, BEVFormer, BEVFusion with LiDAR), supporting detection, mapping and prediction.
  • Tools: Open3D, PyTorch3D, Kaolin; datasets: ScanNet, S3DIS (indoor), KITTI, nuScenes, Waymo Open (driving).
import torch, torch.nn as nn
class PointNetCls(nn.Module):
def __init__(self, k=10):
super().__init__()
self.mlp = nn.Sequential(nn.Conv1d(3, 64, 1), nn.BatchNorm1d(64), nn.ReLU(),
nn.Conv1d(64, 256, 1), nn.BatchNorm1d(256), nn.ReLU(),
nn.Conv1d(256, 1024, 1), nn.BatchNorm1d(1024), nn.ReLU())
self.head = nn.Sequential(nn.Linear(1024, 256), nn.ReLU(), nn.Dropout(0.3), nn.Linear(256, k))
def forward(self, pts): # (B, N, 3)
f = self.mlp(pts.transpose(1, 2)).max(dim=2).values # permutation-invariant pooling
return self.head(f)
  • 2D human pose: detect keypoints (COCO 17 joints). Top-down (detect people, then keypoints per crop: HRNet, ViTPose, RTMPose) versus bottom-up (detect all keypoints, group: OpenPose, HigherHRNet). Heatmap regression (Gaussian peaks per joint, argmax with sub-pixel refinement) remains the strongest formulation; direct regression and SimCC-style coordinate classification are faster. Metric: OKS-based AP.
  • 3D human pose and shape: lift 2D keypoints to 3D, or regress parametric body model parameters (SMPL) directly (HMR, SPIN, HMR 2.0, 4DHumans; whole-body and hand models like SMPL-X, MANO). Used in motion capture, sports analytics, AR.
  • Object 6-DoF pose (rotation + translation of a known object): correspondence-based (keypoints/dense coordinates + PnP + RANSAC, e.g., PVNet, GDR-Net), render-and-compare (MegaPose, FoundationPose), with foundation-model methods that generalize to unseen objects using a CAD model or a few reference images; evaluated with ADD(-S) and BOP challenge metrics.
  • Camera pose / localization: SfM (COLMAP), visual SLAM (ORB-SLAM3, DROID-SLAM, Gaussian-splatting SLAM), absolute pose regression, and learned feature matching (SuperPoint + LightGlue/LoFTR) connect back to Image Fundamentals.
  • Choose the 3D representation by output needs: measurement or robotics → metric depth/LiDAR + calibrated geometry; visualization or digital twin → 3DGS; mesh/CAD workflows → surface reconstruction (MVS, SDF-based NeRF, 2DGS, then meshing).
  • Capture quality dominates NeRF/3DGS results: sharp, well-overlapped images, fixed exposure/white balance, minimal motion blur, adequate camera pose accuracy (COLMAP failure on textureless scenes is common).
  • For video tasks, baseline with a strong image model on sampled frames plus temporal smoothing before reaching for expensive video transformers; many “video” problems are mostly appearance.
  • Throughput and memory budgets decide (frames per clip, resolution). See Efficient & Edge Vision.
  • Video models extend patches to tubelets and factorize or window attention to tame the quadratic cost; masked pretraining uses very high mask ratios because video is redundant. Long-video understanding remains hard.
  • Tracking is usually detection plus association (Kalman filter, Hungarian matching, appearance embeddings), with ByteTrack-style handling of low-score detections; promptable trackers built on SAM 2 are now common.
  • Monocular depth foundation models (Depth Anything-style) give robust relative depth; metric scale needs extra information or dedicated metric models.
  • NeRF = differentiable volume rendering of an MLP field; 3DGS = differentiable rasterization of explicit Gaussians, trading memory for real-time speed.
  • Point-cloud networks rely on permutation-invariant aggregation (PointNet) or sparse convolutions/serialized attention for scale.
  • Feed-forward geometry models (DUSt3R, VGGT, and kin) are rapidly displacing parts of classical SfM/MVS pipelines; verify against current work.
  1. Conceptual. Derive the NeRF quadrature weights wi=Tiαiw_i=T_i\alpha_i and show ∑iwi≤1\sum_i w_i\le1. When does equality hold?
  2. Conceptual. Why does PointNet use max-pooling rather than a concatenation of per-point features?
  3. Hands-on. Implement the composite function on a toy volume (a colored sphere with constant density) and render a depth map as ∑iwisi\sum_i w_i s_i.
  4. Hands-on. Run a depth foundation model on 5 indoor photos and compare against a ruler-measured distance to two objects, estimating the scale factor. How consistent is it across images?
  5. Conceptual. ByteTrack associates low-confidence detections in a second pass. What failure does this address, and what could go wrong?
Solutions (sketch)
  1. ∑iTiαi=1−∏i(1−αi)\sum_i T_i\alpha_i = 1-\prod_i(1-\alpha_i) by telescoping (Ti+1=Ti(1−αi)T_{i+1}=T_i(1-\alpha_i) so Tiαi=Ti−Ti+1T_i\alpha_i=T_i-T_{i+1}). Equals 1 only if the ray is fully absorbed (the remainder is background).
  2. Max-pool (or sum/mean) is invariant to point order; concatenation is order-dependent and has variable length.
  3. Sigma large inside the sphere, 0 outside; wiw_i concentrate at the front surface, so expected depth matches the sphere’s near surface.
  4. Relative models yield a different scale per image; one fixed scale factor will not work, so metric models or a reference are needed.
  5. Occluded objects get low scores and would otherwise lose their track (ID switches). Risk: associating clutter or false positives to tracks; mitigated by only matching to existing unmatched tracks.
  • Carreira and Zisserman, “Quo Vadis, Action Recognition?” (I3D, 2017); Feichtenhofer et al., “SlowFast Networks” (2019).
  • Bertasius et al., “Is Space-Time Attention All You Need for Video Understanding?” (TimeSformer, 2021); Tong et al., “VideoMAE” (2022).
  • Teed and Deng, “RAFT” (2020).
  • Zhang et al., “ByteTrack” (2022); Bewley et al., “Simple Online and Realtime Tracking” (2016).
  • Ranftl et al., “Towards Robust Monocular Depth Estimation” (MiDaS, 2020); Yang et al., “Depth Anything” (2024) and “Depth Anything V2” (2024).
  • Mildenhall et al., “NeRF” (2020); Müller et al., “Instant Neural Graphics Primitives” (2022); Barron et al., “Mip-NeRF 360” (2022).
  • Kerbl et al., “3D Gaussian Splatting for Real-Time Radiance Field Rendering” (2023).
  • Qi et al., “PointNet” (2017) and “PointNet++” (2017); Lang et al., “PointPillars” (2019).
  • Wang et al., “DUSt3R” (2024); Wang et al., “VGGT” (2025).
  • Hartley and Zisserman, Multiple View Geometry in Computer Vision.