Epipolar Geometry — Reading Depth and Camera Motion from Two Images

Move a camera slightly sideways and shoot the same scene, and nearby objects shift more against the background than distant ones do. This parallax lets you recover 3D shape and camera motion from 2D images. But simply matching "the same physical point" across images isn't enough. In real images full of mismatches, lens distortion, pure rotation, planes, and moving objects, you need to determine which pairs of points are consistent with a single camera motion. Epipolar geometry is the common language for doing exactly that.

This isn't just about stereo measurement. Structure from Motion (SfM), Visual Odometry, Visual SLAM, AR plane tracking, robot self-localization, and COLMAP's sparse reconstruction all rest on correspondences and projective geometry. This article keeps coordinate frames unambiguous throughout, connecting what each matrix means, which estimator to choose, and when you shouldn't trust the result.

Example of a stereo-camera vehicle

Image: Subaru WRX S4 2.0GT-S EyeSight (Tokumeigakarinoaoshima, CC BY-SA 4.0), Wikimedia Commons. An exterior view, not a close-up of the camera internals.

0. 30-Second Summary

• The plane formed by two camera centers and one 3D point is called the epipolar plane. This plane cuts each image as a line, so a corresponding point in one image can only appear on that line — the epipolar line — in the other image.
• For uncalibrated images, the Fundamental Matrix F satisfies \mathbf{x}'^\mathsf{T}F\mathbf{x}=0. In normalized coordinates with known intrinsics, the Essential Matrix E=[\mathbf{t}]_\times R is used. E recovers rotation R and the direction of translation, but a single monocular two-view pair cannot recover the absolute scale of translation.
• The normalized 8-point algorithm is an easy-to-implement linear initial estimate; the 5-point algorithm is a minimal solver needing fewer correspondences for a calibrated camera. Both are fragile against mismatches, so in practice you remove outliers with RANSAC/USAC and evaluate with reprojection error.
• Triangulation finds the intersection of two lines of sight, but depth becomes unstable when parallax is small, the baseline is short, or image noise is large. After estimation, bundle adjustment jointly refines camera poses and 3D points.
• For a scene that's planar only, or a camera that's nearly pure-rotating, a homography H describes the images well, and translation/depth recovery via F/E degenerates. Model selection shouldn't rely on inlier count alone — check residuals, parallax, spatial distribution, and cheirality together.

1. Writing Two-View Projection from Coordinates

Let a world-coordinate point be the homogeneous coordinate \mathbf{X}=(X,Y,Z,1)^\mathsf{T}. Pinhole camera projection, up to scale, is written

Here \tilde{\mathbf{x}}=(u,v,1)^\mathsf{T} is the homogeneous image coordinate, K is the intrinsic matrix, and R\in SO(3) and \mathbf{t} are the world-to-camera extrinsic pose. Typically

where f_x,f_y are the focal lengths in pixel units, (c_x,c_y) is the principal point, and s is skew. Once distortion is corrected, the normalized image coordinate is \mathbf{x}=K^{-1}\tilde{\mathbf{x}}. From here on, take the left camera as reference with P_1=K[I\mid\mathbf{0}] and the right camera as P_2=K[R\mid\mathbf{t}].

In the figure, C,C' are the camera centers, and the segment CC' is the baseline. The plane defined by point X and the two centers cuts the left image plane as the epipolar line l, and the right image plane as l'. Once you've found the corresponding point \mathbf{x} in the left image, the 2D search region in the right image collapses to a single line. For a rectified stereo pair, that line is horizontal, and correspondence search becomes a 1D search along the same scanline.

2. The Essential Matrix and Fundamental Matrix

Focusing purely on extrinsic pose, consider calibrated, normalized coordinates (\mathbf{x},\mathbf{x}'). The line-of-sight direction from the left camera to the point is \mathbf{x}, and in the right camera's frame it's R\mathbf{x}. The fact that the translation vector \mathbf{t} and the two lines of sight lie in the same plane can be written as a zero scalar triple product:

Here [\mathbf{t}]_\times is the skew-symmetric matrix form of the cross product.

This E=[\mathbf{t}]_\times R is called the Essential Matrix. E is not an arbitrary 3\times3 matrix — it has rank 2, with the constraint that its two nonzero singular values are equal. Projecting via SVD to the form E=U\operatorname{diag}(s,s,0)V^\mathsf{T} recovers this physical constraint.

For the uncalibrated case using raw pixel coordinates directly,

and F is the Fundamental Matrix. F\tilde{\mathbf{x}} gives the epipolar line l' in the right image, and F^\mathsf{T}\tilde{\mathbf{x}}' gives the line l in the left image. Because F absorbs the intrinsic parameters, it's convenient for geometric verification of image pairs, but interpreting pose in metric units requires calibration.

What the Epipole Tells You

The point where the right camera's center projects into the left image is the left epipole \mathbf{e}, satisfying F\mathbf{e}=0. Likewise F^\mathsf{T}\mathbf{e}'=0. If the epipole lies inside the image, the epipolar lines converge radially, indicating the camera moved roughly forward or backward. If it lies at infinity, the lines are nearly parallel, indicating something close to sideways motion. This is a useful diagnostic, but a bad estimate alone can also produce an unnatural epipole position, so you should never determine motion from this alone.

3. Estimating the Matrix from Correspondences: The 8-Point Algorithm

A single correspondence \tilde{\mathbf{x}}=(u,v,1)^\mathsf{T} and \tilde{\mathbf{x}}'=(u',v',1)^\mathsf{T} gives one linear constraint on F's nine entries. With \mathbf{f}=\operatorname{vec}(F), for example

Stacking eight or more correspondences into matrix A, the 8-point algorithm takes the smallest singular vector of A\mathbf{f}=0. The name comes from eight correspondences satisfying the degrees of freedom, but in the real, noisy case, many more points are used with least squares.

Solving with raw pixel coordinates leads to ill-conditioning from the magnitude of the coordinate values. Hartley's normalized 8-point algorithm normalizes each image's point set with similarity transforms T,T' so the centroid is zero and the mean distance is \sqrt{2}, solves in that space, and finally recovers

Further, taking the SVD of the resulting F and zeroing the smallest singular value enforces rank 2. This looks like a minor implementation detail, but it strongly affects solution stability.

If calibrated, the same idea builds an initial estimate of E from normalized correspondences. But the linear solution from 8 points doesn't automatically satisfy the Essential Matrix's stronger singular-value constraint. You compute E=U\operatorname{diag}(\sigma_1,\sigma_2,\sigma_3)V^\mathsf{T} and replace it with \operatorname{diag}((\sigma_1+\sigma_2)/2,(\sigma_1+\sigma_2)/2,0) to project it.

4. The 5-Point Algorithm: Shrinking the Minimal Sample When Calibrated

The Essential Matrix has 5 degrees of freedom. The 5-point algorithm is a minimal solver that finds a finite set of E candidates from 5 correspondences; Nistér's method substitutes the null space into polynomial constraints, enumerating up to 10 real solution candidates. Both derivation and implementation are more complex than the 8-point algorithm, but the benefit of needing only 5 points per RANSAC hypothesis is very large.

Given inlier ratio w, the probability that a single draw is all inliers is w^s, failure probability p, and minimal sample size s, the required-iteration estimate is

For w=0.5,p=0.01: s=8 needs about 1177 iterations, while s=5 needs about 145. In practice this isn't a straightforward comparison, since methods like PROSAC draw samples in order of match quality and terminate adaptively. Still, the value of the 5-point algorithm is clear in low-inlier-ratio environments.

OpenCV's findEssentialMat provides RANSAC/LMEDS along with 5-point-family implementations, and recoverPose handles candidate decomposition and the cheirality check. For implementers, confirming whether the input is undistorted/normalized and which coordinate units the threshold uses matters more than the fact that "we called the 5-point algorithm."

5. RANSAC: Using Geometry While Assuming Outliers

Matchers like SIFT, ORB, SuperPoint, and LoFTR produce mismatches from repeated textures, reflections, repetitive grids, and occlusion. Fitting F to all correspondences by least squares lets a small number of errors wreck the whole matrix. RANSAC repeats the following:

1. Randomly select a minimal set of correspondences and build an F or E hypothesis.
2. Compute the residual for every correspondence, marking those within a threshold as inliers.
3. Keep the hypothesis with the most support, or the best robust score.
4. Re-estimate using all final inliers, and refine with nonlinear optimization if needed.

You shouldn't threshold the epipolar constraint using algebraic error \mathbf{x}'^\mathsf{T}F\mathbf{x} alone, since it depends on F's scale. In practice, the Sampson distance

is commonly used instead. It's a first-order approximation of geometric error — a normalized measure of the distance from each correspondence to its epipolar line. The threshold in pixel coordinates depends on image resolution, keypoint localization accuracy, residual distortion, and blur. There is no universal "1 px." You tune it by visualizing the residual histogram and the spatial distribution of inliers on the image.

Current OpenCV also offers USAC-family robust estimation. Combining quality-ordered sampling, local optimization, and degeneracy checks, it can be faster and more stable than plain RANSAC. However, statistical outlier rejection can never get past the assumption that "the majority follows a single, static rigid-body motion." If most of the frame is a moving vehicle or a person, you need to add other information such as semantic masks, motion segmentation, IMU, or depth.

6. Decomposing E into Pose and Choosing the Right Candidate

For a corrected E=U\operatorname{diag}(s,s,0)V^\mathsf{T}, using

gives rotation candidates R=UWV^\mathsf{T} or UW^\mathsf{T}V^\mathsf{T}, and translation-direction candidates \pm U_{:,3}. There are 4 combinations of sign and rotation. What matters here is that the two-view constraint alone makes all of them algebraically consistent with the same E.

Selection uses cheirality (positive depth). For each candidate, triangulate a small number of inliers and pick whichever gives Z>0 for the most points in both camera frames. Additionally check whether the rotation matrix's determinant is +1, whether reprojection error is small, and whether there's sufficient parallax. One limitation worth remembering in particular: \mathbf{t} can only be recovered up to direction. Scaling \mathbf{t} and all 3D points by the same factor leaves the projection unchanged. A known stereo baseline, wheel odometry, IMU, an object of known size, or GNSS can supply the scale.

7. Triangulation: From Two Rays to a 3D Point

The projection equation \mathbf{x}\times(P\mathbf{X})=\mathbf{0} produces two independent equations per view. DLT triangulation solves the linear system A\mathbf{X}=0, stacked from two views, via SVD; it's simple, and OpenCV's triangulatePoints is close to this form. For example, letting \mathbf{p}_{ij}^\mathsf{T} be the j-th row of P_i, a point (u_i,v_i) gives

Before dividing by the homogeneous component at the end, check that w isn't extremely small.

For a rectified horizontal stereo pair, this is more intuitive. With disparity d=u_L-u_R (the horizontal-coordinate difference between left and right), focal length f, and baseline B,

Depth error is roughly \delta Z\simeq \frac{Z^2}{fB}\delta d. The farther away, and the shorter the focal length or baseline, the larger the depth error from the same 1-pixel disparity error. So rather than "it matched, so add it to the point cloud," use triangulation angle, disparity, reprojection error, and positive depth as quality gates.

Linear triangulation is only an initial estimate — it doesn't correctly minimize image noise. Bundle adjustment, which jointly optimizes camera poses P_i and points \mathbf{X}_j, solves

where \rho is a robust loss such as Huber or Cauchy, and \pi is the perspective division. This is why reconstructions using COLMAP, Theia, or Ceres Solver gain accuracy. To fix the gauge freedom, place the first camera at the origin, and if needed, fix one known scale.

8. Choosing Between Epipolar Geometry and Homography

When every point in the scene lies on a single plane \pi, or the camera undergoes pure rotation, the correspondence between images is well described by a 3×3 homography \tilde{\mathbf{x}}'\sim H\tilde{\mathbf{x}}. If calibrated, with the plane's normal \mathbf{n} and distance d,

Under pure rotation, the translation term vanishes and H=KRK^{-1}. For a poster, a desk, a building facade, or footage panning across a distant scene, H becomes an excellent model, and it's the natural first choice for AR planar anchors and image stitching.

However, estimating F/E from planar-only data can leave you with many apparent inliers while unable to stably separate 3D structure from translation. Conversely, forcing a general, non-planar scene into a single H leaves near and far objects warping inconsistently. In implementation, estimate both F/E and H with RANSAC and compare residuals, the number of points explained, point distribution, and parallax after reconstruction. If you decide whether to accept a model based on match count alone, you'll get pulled toward a large planar wall or a plane dominating the image center.

9. Calibration Isn't a Preprocessing Step — It's Part of the Model

Shoot a checkerboard, Charuco, or AprilTag grid at multiple distances, tilts, and image positions to estimate K and distortion coefficients. Brown–Conrady radial distortion is roughly expressed, for normalized radius r^2=x^2+y^2, as

For wide-angle and fisheye lenses, don't force a standard pinhole distortion model — choose OpenCV's fisheye model or one matched to the lens in use. Even when calibration's mean reprojection error is small, the error structure can shift at the image edges, at different focal lengths, with temperature, with focus, or with resolution changes.

Before entering two-view processing, confirm the calibration values were obtained at the same resolution, crop, and digital-zoom conditions as your current capture. It's easy to conflate estimating E from points normalized via undistortPoints with estimating F from undistorted images. Always read whether an API uses focal length, principal point, and distortion internally, or expects already-corrected coordinates. For a stereo rig, in addition to both cameras' intrinsics, find the relative pose with stereoCalibrate and rectify epipolar lines to horizontal with stereoRectify.

10. A Minimal Pipeline in OpenCV

Below is the skeleton for obtaining relative pose and a quality-filtered sparse 3D point set from two frames of a calibrated monocular camera. It uses ORB for features, but this can be swapped for SIFT or a learning-based matcher depending on capture conditions. In practice you'd also log exposure, moving objects, and time synchronization.

import cv2 as cv
import numpy as np

# K, dist are values calibrated for this capture resolution and lens
orb = cv.ORB_create(nfeatures=3000)
kp1, des1 = orb.detectAndCompute(img1, None)
kp2, des2 = orb.detectAndCompute(img2, None)
matches = cv.BFMatcher(cv.NORM_HAMMING).knnMatch(des1, des2, k=2)
good = [m for m, n in matches if m.distance < 0.75 * n.distance]

p1 = np.float32([kp1[m.queryIdx].pt for m in good])
p2 = np.float32([kp2[m.trainIdx].pt for m in good])
# threshold is in pixel units. Decide it from the residual distribution, not an initial guess.
E, mask = cv.findEssentialMat(p1, p2, K, method=cv.USAC_MAGSAC,
                              prob=0.999, threshold=1.0)
in1, in2 = p1[mask.ravel() != 0], p2[mask.ravel() != 0]
count, R, t, pose_mask = cv.recoverPose(E, in1, in2, K)

# P1, P2 are for normalized coordinates. Scale is arbitrary, so t's length is not a physical unit.
n1 = cv.undistortPoints(in1.reshape(-1, 1, 2), K, dist).reshape(-1, 2)
n2 = cv.undistortPoints(in2.reshape(-1, 1, 2), K, dist).reshape(-1, 2)
P1 = np.hstack([np.eye(3), np.zeros((3, 1))])
P2 = np.hstack([R, t])
X4 = cv.triangulatePoints(P1, P2, n1.T, n2.T)
X = (X4[:3] / X4[3]).T

# Further filter by positive depth in both views, reprojection error, and triangulation angle.
z1 = X[:, 2]
z2 = (R @ X.T + t).T[:, 2]
valid = (z1 > 0) & (z2 > 0) & np.isfinite(X).all(axis=1)

This example passes raw pixels and K directly into findEssentialMat, but if distortion isn't negligible, first pass normalized points from undistortPoints and switch to the corresponding API form. It's also a mistake to treat the \mathbf{t} returned by recoverPose as a "distance traveled." Applications that need scale must constrain it with a known baseline, VIO, wheel odometry, a depth sensor, or similar.

COLMAP implements feature extraction, matching, geometric verification, incremental mapping, and bundle adjustment as one connected pipeline. For small datasets, you can inspect the camera model and reconstruction through the GUI. On the command line, the choice of camera model, how EXIF focal length is handled, the matching strategy (exhaustive/sequential/vocab tree), and the time interval between image pairs govern both accuracy and compute cost. After reconstruction, check not the point count but the number of registered images, mean reprojection error, the observation count per image, and gaps in the point cloud.

11. Common Failure Conditions and How to Diagnose Them

Small Parallax, No Baseline

With forward motion, a distant scene, or a short frame interval, you may still get correspondences without getting depth. If epipolar lines look reasonable but the triangulation angle is near zero, hold off on updating depth rather than forcing it. The fundamental fixes are to space keyframes farther apart, capture observations with sideways motion, or use a stereo rig with a known baseline.

Pure Rotation or Planar Degeneracy

In panning shots or a field of view containing only a wall, H has explanatory power. A high inlier count for E doesn't necessarily mean translation was observed. Log the competition between H and E, and gate on the positive-depth rate and median parallax after triangulation. In AR poster tracking, this isn't a failure — it's the correct model selection.

Mismatches, Repetitive Patterns, Reflections

Windows, tiles, bookshelves, LCD screens, and water surfaces produce similar local descriptors. Layer the ratio test, mutual nearest-neighbor matching, and geometric RANSAC, and check whether inlier points are spread across the whole image. Mirror images and transparent objects break the rigid-body, Lambertian-reflectance assumption itself, so no amount of threshold tuning will save you.

Dynamic Objects and Multiple Motions

RANSAC only picks the single largest motion. If the background is the minority, it may end up estimating a car's motion instead. Depending on your application, choose from semantically excluding people/vehicles, clustering optical flow, running multi-model estimation, or aligning with depth/IMU.

Lens Distortion, Rolling Shutter, Asynchrony

Using an uncorrected wide-angle edge leaves systematic curvature in epipolar lines. With rolling shutter during fast motion, attitude changes within a single frame, so a single E is only an approximation. Even a slight offset in left/right exposure timing for a stereo pair produces spurious disparity for moving objects. Consider global shutter, short exposure, a row-timing model, IMU-based correction, and hardware synchronization.

Numerical and Coordinate-Frame Accidents

Mixing pixel and normalized coordinates, confusing world-to-camera versus camera-to-world for R,\mathbf{t}, swapping the left/right point order, and forgetting to update K after resizing an image are all common mistakes. Don't take estimated values at face value — overlay correspondences and epipolar lines, and automate checks for positive depth in both cameras, reprojection error, \det R=1, and R^\mathsf{T}R\simeq I.

12. Practical Evaluation Metrics and a Design Checklist

Don't call two-view estimation a success just because "a matrix came back." Match count is skewed by texture amount, and average error alone can hide behind a few good points. Saving the following per frame lets you later distinguish where in the sensor/matcher/pose-estimation chain something broke:

• Detection count, ratio-test pass count, RANSAC inlier count/ratio, distribution across image grid cells
• Median and upper-percentile Sampson distance and reprojection error, positive-depth rate, triangulation-angle distribution
• Support count and robust score for H versus E/F, and the reason a model was accepted or rejected
• Magnitude of the estimated rotation, temporal continuity of translation direction, consistency with a scaled external sensor
• Exposure time, gain, IMU angular velocity, left/right time offset, blur metric, image mask ratio

For research or product evaluation with ground truth available, report relative rotation error, translation-direction error, trajectory ATE/RPE, and absolute/relative depth error separately. Since monocular two-view translation is scale-ambiguous, state clearly whether the error is after normalization or after Sim(3) alignment. Rather than excluding failed frames from the average, noting which degeneracy or visual condition each failure occurred under communicates the system's limits more honestly.

13. Recent Developments: Has Learning Replaced Geometry?

Learning-based keypoints and descriptors (SuperPoint), coarse-to-fine matchers (LoFTR), and general-purpose correspondence estimation (LightGlue and similar) can produce more candidate matches than classical descriptors under low texture or viewpoint change. But correspondences returned by a network can still be wrong, and the physical ambiguities of camera motion, planes, rolling shutter, and scale don't disappear. In practical SfM/SLAM, a hybrid setup that verifies a learned matcher's output through robust estimation of E/F/H plus bundle adjustment remains the practical choice.

At a broader level, neural/explicit scene representations such as NeRF and 3D Gaussian Splatting also exploit consistency across multiple views. These enable attractive novel-view synthesis, but are sensitive to the quality of camera pose and observation geometry, and many implementations initialize with poses derived from COLMAP. Research continues on jointly estimating correspondence, depth, segmentation, inertial data, and timing models for large-scale, dynamic, and reflective environments.

So the decision to adopt a newer model shouldn't rest only on "did the match count go up compared to ORB" — it should also factor in the post-estimation inlier distribution, pose error, compute latency, GPU requirements, breakdown outside the training conditions, and licensing. Geometry isn't an outdated preprocessing step; it remains the verifier that checks a learned model's output against real 3D structure.

14. Conclusion

Epipolar geometry is the framework that elevates correspondences between two images from "points that look most similar" to "points explainable by a single camera motion." If intrinsics are known, you proceed to relative pose via E=[\mathbf{t}]_\times R; if not, you verify epipolar lines and correspondences with F. The 8-point algorithm is the foundation for understanding and initialization, the 5-point algorithm is an efficient minimal solver for robust estimation, RANSAC is the mechanism that assumes outliers, and triangulation plus bundle adjustment are the bridge into 3D.

However, when there's no parallax, only a plane, pure rotation, many moving objects, or heavy distortion/asynchrony, a returned matrix does not guarantee physically meaningful depth or translation. Designing model selection against homography, managing calibration conditions, checking reprojection error and positive depth, and fusing with external scale into a single pipeline is what leads to reproducible computer vision.

References (Primary Sources and Official Documentation)

• Hartley & Zisserman, Multiple View Geometry in Computer Vision (author's official page)
• Longuet-Higgins, A computer algorithm for reconstructing a scene from two projections (1981, Royal Society)
• Hartley, In Defense of the Eight-Point Algorithm (IEEE TPAMI, 1997)
• Nistér, An Efficient Solution to the Five-Point Relative Pose Problem (IEEE TPAMI, 2004)
• Fischler & Bolles, Random Sample Consensus (Communications of the ACM, 1981)
• OpenCV — Epipolar Geometry tutorial
• OpenCV — calib3d: findEssentialMat / recoverPose
• COLMAP official documentation
• Schönberger & Frahm, Structure-from-Motion Revisited (CVPR 2016)
• Sarlin et al., LightGlue (ICCV 2023)