Looking at video one frame at a time, it isn't intuitively obvious which part of the image moved by how much. Optical Flow represents where each pixel's brightness moved to in the next frame, as a vector. It becomes a common language for any process that involves motion: collision prediction in self-driving cars, self-localization for drones, sports analytics, and video interpolation.
0. 30-Second Summary
- Flow is not "the velocity of the object itself" but its apparent motion in the image. Camera motion, object motion, and depth are all mixed together in it.
- The brightness-constancy equation gives only one equation per pixel, so it has to be solved by adding a local-window smoothness assumption, feature points, or regularization.
- Lucas–Kanade treats a small window as a single velocity — a sparse tracking method. Horn–Schunck uses smoothness across the whole image — a dense estimation method.
- Large displacements need an image pyramid; occlusion, reflections, and blur need confidence measures and outlier handling. Rolling shutter also requires correcting for the difference in row timing.
- Learning-based methods such as RAFT are highly accurate, but should be adopted only after checking GPU memory, out-of-distribution behavior, real-time performance, and licensing.
1. From Brightness Constancy to the Flow Constraint Equation
Figure 1 — A pyramid handles large displacement first, then refines a dense vector field at finer scales. Confidence and occlusion masks must travel with the vectors.
If a small, stationary pattern moves between frames, we can idealize it as having constant brightness.
A first-order Taylor expansion together with \Delta t\to0 gives
Since there are two unknown velocity components (u,v) but only one equation, this alone cannot be solved. On an edge, motion along the edge direction is invisible; in a flat region, there's no gradient at all. This is the aperture problem.
2. Lucas–Kanade and Horn–Schunck
Lucas–Kanade assumes the velocity is the same throughout a local window W, and minimizes the following squared error.
It uses only corners where the gradient matrix is sufficiently well-conditioned, and combines this with the same pyramid and iterative update used in feature-point tracking (see the preceding section). OpenCV's calcOpticalFlowPyrLK is an implementation from this family.
Horn–Schunck treats the flow field over the entire image as the unknown, and simultaneously minimizes the brightness constraint and the smoothness of velocity.
A larger \alpha yields a smoother flow field; a smaller one permits local discontinuities. Smoothing across an object boundary mixes the velocities of different objects, so robust losses or edge-preserving regularization are used instead.
3. Sparse Flow and Dense Flow
| Type | Estimated points | Representative methods | Strengths | Weaknesses |
|---|---|---|---|---|
| Sparse | Hundreds to thousands of points, e.g. corners | LK, KLT | Lightweight, feeds directly into pose estimation | Leaves gaps in low-texture regions |
| Semi-dense | Pixels with gradient | Direct VO, Hessian-based methods | Balances geometric information against compute cost | Doesn't fill the whole image |
| Dense | Nearly every pixel | Horn–Schunck, TV-L1, RAFT | Effective for moving objects, fluids, interpolation | Compute cost, ambiguity at occlusion boundaries |
For Visual Odometry, passing sparse correspondences into the geometric computation tends to be more stable. On the other hand, masking regions of moving pedestrians, or using per-pixel motion for video interpolation, requires dense flow. Deciding the required density up front, for the purpose at hand, is more effective than simply throwing more GPU at the problem.
4. Handling Large Displacements, Occlusion, and Brightness Change
A one-pixel differential approximation breaks down under large motion. A Gaussian pyramid is built by shrinking the image to 1/2, 1/4, 1/8 scale; large displacements are estimated at the coarse level, then upsampled to the fine level and refined iteratively. Too many pyramid levels makes small objects disappear; too few leaves an insufficient search range.
When lighting changes, brightness constancy breaks down, so local normalization, gradient direction, robust Charbonnier loss, or relative color difference are used instead. At the boundary of a moving object, a pixel visible in the previous frame may be hidden in the next (occlusion). Occlusion flags, forward-backward consistency, and visibility masks are used rather than forcing a track through it.
5. Learning-Based Methods: How to Read RAFT
RAFT (Recurrent All-Pairs Field Transforms) is known for computing correlation across all pixel pairs between two images and then refining the flow with an iterative update operator. Because it can draw on a much wider pool of correspondence candidates than the "local window" of classical methods, it can be strong in areas with repeating texture or under large displacements.
But a low average endpoint error (EPE) on a benchmark is not the same thing as being safe to use on a real robot in the field. If the camera's lens, exposure, rolling shutter, dust, or nighttime lighting differ from the training data, confidence drops. Evaluation should include inference time, input resolution, quantization error, GPU driver, and the model's license.
6. Separating Camera Motion from Dynamic Objects
Converting flow into camera motion requires the camera intrinsic matrix K and depth Z. In normalized coordinates of an image point \mathbf{x}, the flow due to the camera's translation \mathbf{t} and angular velocity \boldsymbol{\omega} can conceptually be written as
The translational component varies with 1/Z — nearer things move more than farther ones — while the rotational component does not depend on depth. Flow consistent with a single motion model, found via RANSAC, is treated as background; regions with large residuals become candidate dynamic objects. In scenes with many vehicles or pedestrians, object detection and semantic masks are used together with geometric estimation.
7. Evaluation Metrics and Reproducible Measurement
Given ground-truth flow (u^*,v^*), the average endpoint error is
Report not only the mean but the 95th percentile, error at occlusion boundaries, error in low-texture regions, and error broken out by velocity. Since ground truth is hard to obtain on real hardware, it's typically combined from motion capture, a robot arm's known trajectory, synthetic images, forward-backward consistency, and VO reprojection error.
Logs should retain camera timestamps, exposure, resolution, pyramid levels, window size, iteration count, GPU/CPU, temperature, and flow confidence. Even under the same algorithm name, results aren't comparable if these conditions differ.
8. Summary
Optical Flow constrains the apparent motion of pixels with equations, and solves it using local windows, whole-image smoothness, image pyramids, and learning-based correlation. Sparse LK suits self-localization; dense flow suits dynamic objects and video processing. Considering camera motion versus object motion, occlusion, lighting, and rolling shutter separately, and evaluating failure conditions rather than just the average error, helps avoid the wrong implementation choice.