If feature detection is the process of deciding "where in the image to use as a landmark," feature tracking is the process of finding "where that landmark moved to in the next frame." Stable tracking lets you estimate camera motion, object velocity, or a robot's self-localization. Conversely, getting a single correspondence wrong can cause the subsequent pose estimate and map to collapse all at once. This article treats the difference between detectors and descriptors as given, and handles both local optimization of pixel motion and descriptor matching as a single design problem.
0. 30-Second Summary
- There are two approaches to tracking: "search the next frame for the previous frame's surrounding patch," and "detect and compute descriptors in each frame and match them."
- The Lucas–Kanade method assumes brightness constancy, local motion, and shared velocity among neighboring pixels, and solves a 2×2 normal-equation system. Using a pyramid extends it to handle large displacements.
- Descriptor matching can re-search even across large frame gaps, but at the cost of more compute and more mismatches. Verify geometrically with the ratio test, mutual nearest neighbors, and RANSAC.
- Tracking quality should be judged not just by correspondence count, but by spatial distribution across the image, reprojection error, forward-backward consistency, and the recovery rate after occlusion.
- The hard cases are motion blur, low texture, reflections, dynamic objects, rolling shutter, and sudden scale change. A confidence score and a re-detection mechanism are essential.
1. Why Points on the Image Appear to Move
Figure 1 — A correspondence is a hypothesis until its motion agrees with neighboring points and the camera model. Spatial coverage matters as much as the raw match count.
A 3D point X projects to \mathbf{x}=(x,y) on the image under camera motion. With frame interval \Delta t, the goal of tracking is to find the displacement \mathbf{d}_k in
When the camera translates, apparent speed varies with depth; when it rotates, the whole image flows in the same direction. Simply subtracting consecutive images is fragile against lighting change and exposure noise, so local patch structure is used instead.
2. Lucas–Kanade: Solving a Small Window All at Once
The brightness-constancy assumption is
Linearizing for small displacement (u,v) gives the optical-flow constraint equation
At a single pixel, this is one equation for two unknowns (the aperture problem). So the pixels within a window W are pooled, and least squares
is solved. The bracketed term is the feature point's local structure matrix — the more gradient there is in two directions, at a corner, the more stably invertible it is. On a flat wall or a single edge, motion can't be uniquely determined.
To handle large displacements, the displacement is propagated from the coarse level of a downscaled pyramid down to the fine level. At each level, several iterations are run before recomputing at the next position. In implementation, you tune pyramid depth, window size, termination criteria, minimum eigenvalue, and forward-backward tracking error.
3. Choosing Between This and Descriptor Matching
Descriptors like ORB or SIFT convert the surrounding patch into a vector or bit string, and take the closest-distance candidate as the correspondence. For small motion between consecutive frames, Lucas–Kanade is fast, but descriptor re-matching becomes valuable when recovering from occlusion, skipping frames, or when the camera moves substantially.
| Method | Input | Strengths | Weaknesses | Typical use |
|---|---|---|---|---|
| LK tracking | Previous frame's point and next frame's image | Fast, subpixel accuracy | Weak to large displacement, occlusion, low texture | VO, real-time tracking |
| ORB matching | Descriptors from 2 images | Lightweight, handles rotation | Mismatches from reflection/blur | SLAM initialization/re-search |
| SIFT matching | Descriptors from 2 images | Robust to scale/rotation | Compute, memory | SfM, image retrieval |
| Learning-based | Points, descriptors, matcher | Potentially robust to large appearance change | Outside training data, GPU load | Hard environments, research |
Taking only the closest-distance descriptor candidate lets in mismatches from similar patterns. Requiring d_1/d_2<\tau (the ratio test) between the nearest distance d_1 and the second-nearest d_2, and further checking mutual nearest neighbors from A→B and B→A, helps. Finally, verify with RANSAC using the reprojection error of a fundamental matrix, homography, or PnP estimated from the correspondences.
4. Quantifying Tracking Confidence
In implementation, judging "tracking succeeded" as merely "a point was returned" isn't enough. Logging the following lets you isolate the cause of a breakdown:
- LK's minimum eigenvalue and residual
- The difference between forward and backward tracking (forward-backward error)
- The distribution of descriptor ratio and distance
- RANSAC inlier ratio and reprojection error
- The spatial distribution of points on the image (are they clustered only in the center?)
- Mean inter-frame displacement, blur metric, exposure/gain
Even with a high RANSAC inlier ratio, if all points cluster in one corner of the image, pose estimation degenerates. Capping the maximum point count per grid cell and spreading feature points across the whole field of view improves the observability of rotation and translation. Sometimes keeping a small number of correspondences spread across different directions and distances is better than simply increasing point count.
5. Moving Objects and Rolling Shutter
Visual Odometry assumes a static environment to estimate camera motion. When many pedestrians, cars, or a spinning fan appear in frame, their correspondences become outliers inconsistent with the camera-motion model. When there are too many dynamic objects for RANSAC alone to remove, combine semantic masking, optical-flow clustering, background modeling, and depth consistency.
A CMOS camera's rolling shutter exposes the image row by row, from top to bottom, at slightly different times. Under fast rotation or vibration, camera pose changes row by row even within a single frame, breaking the assumption of a single projection model. Mitigations include row-timing correction using IMU angular velocity, global shutter, short exposure, and calibrating the readout time.
6. Flow of a Minimal Implementation
- Calibrate the camera together with intrinsics, distortion, and timestamps.
- Detect FAST/ORB or Shi–Tomasi in the initial frame, and equalize spatially via a grid.
- On each new frame, track with pyramidal LK, checking forward-backward error and image boundaries.
- Discard low-confidence points, and detect new points in under-populated grid cells.
- Run descriptor matching at the necessary interval, removing outliers with RANSAC.
- Pass the remaining correspondences to an Essential Matrix, PnP, or IMU fusion step.
- If points are lost continuously, re-initialize, and log the tracking state and cause.
7. Conclusion
Feature tracking is a technology that treats detector, local optimization, descriptor matching, and geometric verification as a single confidence-design problem. LK smoothly links consecutive frames, descriptors recover from large changes, and RANSAC sieves out mismatches geometrically. Rather than depending on any single one of these, keeping logs that include point distribution, time synchronization, dynamic objects, and rolling shutter substantially improves the reproducibility of Visual SLAM and VIO.