← AI Terminology
Optical Flow
Optical flow is a computer vision technique that estimates the apparent motion of pixels or regions between consecutive video frames — producing a dense 2D vector field where each vector indicates how far and in what direction each pixel moved.
It is the foundational technique for video understanding and motion estimation.
It is the foundational technique for video understanding and motion estimation.
Why It Matters in AI
Understanding motion is fundamental to video analysis: action recognition requires knowing how a person's limbs move; autonomous vehicles need to track moving objects' trajectories; video compression (HEVC, AV1) uses motion estimation to encode only differences between frames. Classical optical flow (Lucas-Kanade, Farneback) was replaced by deep optical flow (FlowNet, RAFT) which achieves dramatically better accuracy on fast motion and occlusion.
Key Points
| Aspect | Description |
|---|---|
| RAFT | Recurrent All-Pairs Field Transforms — current state-of-the-art; iterative refinement approach |
| Output | Dense flow field: for each pixel (x,y) at frame t, vector (u,v) gives its position at frame t+1 |
| Classical | Lucas-Kanade (sparse, fast), Farneback (dense) — classic algorithms, still used on edge devices |
| Applications | Action recognition (two-stream networks), video stabilisation, super-resolution, video codecs |
| Deep learning | FlowNet (2015), PWCNet, RAFT (2020) — learns flow end-to-end; far better on large displacements |
| Sparse vs dense | Sparse: flow at keypoints (ORB, SIFT) — fast; Dense: flow at every pixel — slow but complete |
Simple Analogy
Time-lapse photography with motion arrows: for each point in a photo, draw an arrow showing where that point moved in the next photo. A dense grid of such arrows is optical flow — a complete picture of "where everything went" between two frames.
Common Usage Examples
- OpenCV:
cv2.calcOpticalFlowFarneback(prev_gray, curr_gray, None, 0.5, 3, 15, 3, 5, 1.2, 0) - Lucas-Kanade sparse:
pts_next, _, _ = cv2.calcOpticalFlowPyrLK(prev, curr, pts_prev, None) - RAFT:
from torchvision.models.optical_flow import raft_large; model = raft_large(pretrained=True) - Two-stream action recognition: spatial network (RGB frames) + temporal network (optical flow stacks)
- Video codec: HEVC motion estimation — block-level optical flow for P-frame encoding
Summary
In short: Optical flow estimates per-pixel motion between video frames — the foundational technique for video understanding, action recognition, and motion compensation in video codecs.