Feature Tracking Primer — How to Avoid Losing the Same Point Between Frames

If feature detection is the process of deciding "where in the image to use as a landmark," feature tracking is the process of finding "where that landmark moved to in the next frame." Stable tracking lets you estimate camera motion, object velocity, or a robot's self-localization. Conversely, getting a single correspondence wrong can cause the subsequent pose estimate and map to collapse all at once. This article treats the difference between detectors and descriptors as given, and handles both local optimization of pixel motion and descriptor matching as a single design problem.

0. 30-Second Summary

• There are two approaches to tracking: "search the next frame for the previous frame's surrounding patch," and "detect and compute descriptors in each frame and match them."
• The Lucas–Kanade method assumes brightness constancy, local motion, and shared velocity among neighboring pixels, and solves a 2×2 normal-equation system. Using a pyramid extends it to handle large displacements.
• Descriptor matching can re-search even across large frame gaps, but at the cost of more compute and more mismatches. Verify geometrically with the ratio test, mutual nearest neighbors, and RANSAC.
• Tracking quality should be judged not just by correspondence count, but by spatial distribution across the image, reprojection error, forward-backward consistency, and the recovery rate after occlusion.
• The hard cases are motion blur, low texture, reflections, dynamic objects, rolling shutter, and sudden scale change. A confidence score and a re-detection mechanism are essential.

1. Why Points on the Image Appear to Move

Diagram 1 · Use the button to switch views

Figure 1 — A correspondence is a hypothesis until its motion agrees with neighboring points and the camera model. Spatial coverage matters as much as the raw match count.

A 3D point X projects to \mathbf{x}=(x,y) on the image under camera motion. With frame interval \Delta t, the goal of tracking is to find the displacement \mathbf{d}_k in

When the camera translates, apparent speed varies with depth; when it rotates, the whole image flows in the same direction. Simply subtracting consecutive images is fragile against lighting change and exposure noise, so local patch structure is used instead.

2. Lucas–Kanade: Solving a Small Window All at Once

The brightness-constancy assumption is

Linearizing for small displacement (u,v) gives the optical-flow constraint equation

At a single pixel, this is one equation for two unknowns (the aperture problem). So the pixels within a window W are pooled, and least squares

is solved. The bracketed term is the feature point's local structure matrix — the more gradient there is in two directions, at a corner, the more stably invertible it is. On a flat wall or a single edge, motion can't be uniquely determined.

To handle large displacements, the displacement is propagated from the coarse level of a downscaled pyramid down to the fine level. At each level, several iterations are run before recomputing at the next position. In implementation, you tune pyramid depth, window size, termination criteria, minimum eigenvalue, and forward-backward tracking error.

3. Choosing Between This and Descriptor Matching

Descriptors like ORB or SIFT convert the surrounding patch into a vector or bit string, and take the closest-distance candidate as the correspondence. For small motion between consecutive frames, Lucas–Kanade is fast, but descriptor re-matching becomes valuable when recovering from occlusion, skipping frames, or when the camera moves substantially.

Taking only the closest-distance descriptor candidate lets in mismatches from similar patterns. Requiring d_1/d_2\lt\tau (the ratio test) between the nearest distance d_1 and the second-nearest d_2, and further checking mutual nearest neighbors from A→B and B→A, helps. Finally, verify with RANSAC using the reprojection error of a fundamental matrix, homography, or PnP estimated from the correspondences.

4. Quantifying Tracking Confidence

In implementation, judging "tracking succeeded" as merely "a point was returned" isn't enough. Logging the following lets you isolate the cause of a breakdown:

• LK's minimum eigenvalue and residual
• The difference between forward and backward tracking (forward-backward error)
• The distribution of descriptor ratio and distance
• RANSAC inlier ratio and reprojection error
• The spatial distribution of points on the image (are they clustered only in the center?)
• Mean inter-frame displacement, blur metric, exposure/gain

Even with a high RANSAC inlier ratio, if all points cluster in one corner of the image, pose estimation degenerates. Capping the maximum point count per grid cell and spreading feature points across the whole field of view improves the observability of rotation and translation. Sometimes keeping a small number of correspondences spread across different directions and distances is better than simply increasing point count.

5. Moving Objects and Rolling Shutter

Visual Odometry assumes a static environment to estimate camera motion. When many pedestrians, cars, or a spinning fan appear in frame, their correspondences become outliers inconsistent with the camera-motion model. When there are too many dynamic objects for RANSAC alone to remove, combine semantic masking, optical-flow clustering, background modeling, and depth consistency.

A CMOS camera's rolling shutter exposes the image row by row, from top to bottom, at slightly different times. Under fast rotation or vibration, camera pose changes row by row even within a single frame, breaking the assumption of a single projection model. Mitigations include row-timing correction using IMU angular velocity, global shutter, short exposure, and calibrating the readout time.

6. Flow of a Minimal Implementation

1. Calibrate the camera together with intrinsics, distortion, and timestamps.
2. Detect FAST/ORB or Shi–Tomasi in the initial frame, and equalize spatially via a grid.
3. On each new frame, track with pyramidal LK, checking forward-backward error and image boundaries.
4. Discard low-confidence points, and detect new points in under-populated grid cells.
5. Run descriptor matching at the necessary interval, removing outliers with RANSAC.
6. Pass the remaining correspondences to an Essential Matrix, PnP, or IMU fusion step.
7. If points are lost continuously, re-initialize, and log the tracking state and cause.

Deciding when to resume tracking

When occlusion or exposure change temporarily reduces the number of points, extrapolating every old position can turn a wrong correspondence into a stable-looking track. Check frame number, timestamps, image boundaries, and forward–backward error first, then remove candidates outside the operating envelope. Even a sufficient count is unsafe when all surviving points occupy one small image region; stop pose estimation, re-detect by grid cell, and separate static background from moving objects before refitting the geometric model. Recording this as a state machine with “tracking” and “reinitializing” states makes missing data distinguishable from an incorrect match.

In implementation, attach a frame number and observation timestamp to every correspondence so processing delay cannot mix an old point into a newer frame. When the camera moves abruptly, check whether the displacement still fits the coarsest pyramid level and window. If it does not, switching to descriptor re-search is safer than widening the window until it converges on another pattern. Re-searched candidates must still pass epipolar or homography residual checks and forward–backward consistency. If moving objects are tracked deliberately, keep camera-motion points and object-tracking points in separate sets; do not reuse one set's outlier decisions for the other. Replay logs with lighting changes, occlusion, and communication delay to test recovery at these boundaries.

7. Conclusion

Feature tracking is a technology that treats detector, local optimization, descriptor matching, and geometric verification as a single confidence-design problem. LK smoothly links consecutive frames, descriptors recover from large changes, and RANSAC sieves out mismatches geometrically. Rather than depending on any single one of these, keeping logs that include point distribution, time synchronization, dynamic objects, and rolling shutter substantially improves the reproducibility of Visual SLAM and VIO.

References

• OpenCV — Optical Flow / Lucas-Kanade
• OpenCV — Feature matching
• Bouguet, Pyramidal implementation of the affine Lucas Kanade feature tracker
• ORB-SLAM3 paper
• VINS-Mono paper

Track these features between two frames in the optical flow Lab.