What Structure from Motion recovers is a sparse 3D point cloud connecting nothing but feature points. You can make out a building's outline or the corners of its texture, but walls and curved surfaces are left with almost no points at all, and the result can't be used as "shape" as it stands. Multi-View Stereo (MVS) takes the camera poses already known from SfM or camera calibration as given, and estimates depth for nearly every pixel in the image, filling it out into a dense point cloud or mesh. The division of labor — pose estimation is SfM's job, dense shape recovery is MVS's job — is the starting point for understanding how these two technologies relate.

0. 30-Second Summary

1. What Does It Take as Input, and What Does It Solve For?

MVS's input is the following information, already found via SfM or camera calibration:

The output is a dense depth map \{D_i\} for each image (or a selected set of reference images), or the point cloud/mesh obtained by integrating them. If SfM's output — the sparse point cloud and camera poses — is the "skeleton," MVS is the process that puts "flesh" on it. You can't recover dense shape with unknown poses — MVS always sits downstream of SfM or calibration, and it's worth fixing this ordering in mind from the start.

2. Why Isn't a Sparse Point Cloud Enough?

The reason SfM doesn't directly output a dense point cloud is that its input depends on feature-point matching. As we saw in the Feature Detection Primer, only "distinctive" pixels — corners, edges — can be stably detected and matched. A uniformly textured region like a plain wall has no feature points at all, leaving a gaping hole in SfM's 3D point cloud.

MVS, on the other hand, can exploit the strong constraint that poses are already known, so it doesn't need feature points at all. For any pixel, it can directly evaluate "does this depth candidate stay consistent across the other images?" This opens up the possibility of estimating depth even for a texture-poor wall, as long as there's at least some pattern or shading to work with (a completely featureless surface remains a weak point, as discussed below).

3. The Core Principle: Photo-Consistency

Nearly every MVS method is built on the assumption of photo-consistency. Suppose the depth of the 3D point corresponding to pixel \mathbf{u} in a reference image is d; that 3D point can be recovered as

\mathbf{X}(\mathbf{u}, d) = \pi_{\text{ref}}^{-1}(\mathbf{u}, d)

and the photo-consistency assumption is that reprojecting it into another image k, at pixel \mathbf{u}' = \pi_k(\mathbf{X}(\mathbf{u},d)), should give a color and brightness close to I_{\text{ref}}(\mathbf{u}). This is easiest to understand as a generalization, from 2 viewpoints to N viewpoints, of stereo-camera disparity search — the process, covered in How Depth Cameras Work and How Stereo Cameras Work, of finding corresponding pixels between left and right images by matching brightness. In fact, depth for a two-eye stereo camera is found by the simple formula

Z = \frac{fB}{d_{\text{disp}}}

using focal length f, baseline length B, and disparity d_{\text{disp}} — and MVS is exactly this operation of "search for disparity and convert to depth," extended to any number of cameras in any arrangement.

A typical implementation uses a small window W around the pixel and measures this agreement with normalized cross-correlation (NCC).

\mathrm{NCC}(\mathbf{u}, d) = \frac{\sum_{\mathbf{x}\in W}\left(I_{\text{ref}}(\mathbf{x})-\bar I_{\text{ref}}\right)\left(I_k(\mathbf{x}')-\bar I_k\right)} {\sqrt{\sum_{\mathbf{x}\in W}\left(I_{\text{ref}}(\mathbf{x})-\bar I_{\text{ref}}\right)^2}\sqrt{\sum_{\mathbf{x}\in W}\left(I_k(\mathbf{x}')-\bar I_k\right)^2}}

\mathbf{x}' is the corresponding point obtained by mapping \mathbf{x} into image k, assuming a local plane at depth candidate d. \mathrm{NCC} is robust to brightness scale and offset changes, so it functions even with some exposure or lighting difference between images. Computing this agreement score for every image pair and every depth candidate, and choosing the depth with the best score, forms the computational skeleton of MVS.

4. The Basic Pipeline

The basic MVS pipeline A diagram showing the flow from a set of images with known camera poses, through estimating each image's dense depth map via either Plane-Sweep or Patch-based methods, to fusing multi-view depth maps into a dense point cloud or mesh. Image set +known camera poses Plane-Sweepsweep depth candidates,evaluate photo-consistency Patch-basedexpand patches,filter by visibility Dense depth mapfor each image Depth mapfusion Point cloud/mesh

The classical MVS basic form is a two-stage structure: choose either Plane-Sweep or Patch-based to find each image's dense depth map first, then fuse them into one consistent 3D shape in the second stage. Modern deep-learning-based methods largely follow this same two-stage structure, while replacing the internals of depth estimation with a neural network.

5. The Plane-Sweep Method

The Plane-Sweep method traces back to a space-scanning multi-image matching approach Collins proposed at CVPR 1996. It lines up virtual planes, spaced at regular intervals, perpendicular to the reference camera's optical axis (or oriented according to the scene) within the reference camera's view frustum, and evaluates as it sweeps depth from shallow to deep.

Assuming a plane at some depth d, points on that plane can be mapped from the reference image to another image via a homography transform. Using a transform of the form H = K_k(R+\mathbf{t}\mathbf{n}^\mathsf{T}/d)K_{\text{ref}}^{-1}, covered in the Homography Primer, the other image I_k is warped into the reference viewpoint. Photo-consistency (such as the NCC from the previous section) is computed at every pixel between the warped image and the reference image, and cost is accumulated for each depth candidate.

d^*(\mathbf{u}) = \arg\min_{d\in\mathcal{D}} \sum_{k} \rho\left(1-\mathrm{NCC}_k(\mathbf{u},d)\right)

Once cost has been computed for every depth candidate, the minimum-cost depth is chosen per pixel. This is a discrete depth search, which pairs well with GPU parallelization, evaluating many depth hypotheses at once. Many implementations combine this with semi-local cost aggregation (regularization similar to Semi-Global Matching), smoothly interpolating depth from neighboring information even in texture-poor regions. COLMAP's dense-reconstruction module also adopts an approach close to Plane-Sweep, optimizing per-pixel view selection (Pixelwise View Selection) — Schönberger et al.'s ECCV 2016 paper is a representative example.

6. The Patch-based Method (PMVS)

Rather than sweeping depth pixel by pixel, the Patch-based method directly generates and expands a set of small rectangular patches covering the scene surface. A representative example is PMVS (Patch-based Multi-View Stereo), published by Furukawa and Ponce in IEEE TPAMI in 2010.

Processing repeats three stages: "match, expand, filter."

  1. Match: first generate a small number of initial patches from correspondence points easy to detect as feature points, such as SIFT or Harris corners. Each patch carries a center position, a normal direction, and the set of images that see that point (visibility).
  2. Expand: propagate new patches into the neighborhood of the initial patches, widening the covered area to surrounding pixels. The position and normal of each propagated patch are locally optimized to maximize photo-consistency with the surrounding images.
  3. Filter: remove patches with visibility contradictions (such as a case where a patch is supposedly visible despite being behind another patch) or low photo-consistency.

Unlike Plane-Sweep, which determines depth independently per pixel, PMVS carries the extra information of a patch normal, so it tends to have higher reconstruction accuracy for oblique surfaces. On the other hand, because it's built on iteratively expanding and filtering, expansion doesn't progress well in regions with few initial patches or poor texture, and reconstruction tends to be left with holes.

7. Deep-Learning-Based Methods: the Cost-Volume Idea

In recent years, methods that represent per-depth-candidate agreement not with a hand-designed metric (like NCC) but with features learned by a convolutional neural network and a cost volume have become mainstream. A representative example is MVSNet, published by Yao et al. at ECCV 2018.

MVSNet obtains a feature map from each image via a trained feature extractor, assumes discrete depth planes within the reference camera's view frustum, and aligns each image's feature map to the reference viewpoint via a differentiable homography warp. It combines the variance across multiple images' feature maps into a single cost volume, regularizes it with 3D convolution, and then regresses depth via a softmax along the depth direction. Its basic skeleton follows Plane-Sweep's idea of "sweep depth candidates and evaluate," but the difference from classical methods is that the computation of photo-consistency itself becomes learnable.

Learning-based methods tend to behave more robustly under conditions where hand-crafted photo-consistency metrics struggle — repetitive patterns, weak texture — as long as the training data includes similar situations. On the other hand, performance can degrade in scenes far outside the distribution of the training dataset (unfamiliar materials, extreme lighting).

8. Depth-Map Fusion and Meshing

Because multi-view depth maps are each estimated independently, simply overlaying them as 3D points as-is leaves contradictions from noise and occlusion (slightly offset points at the same location piling up in several layers, or depth disagreeing between viewpoints). Fusion is the process that consolidates these depth maps into a single, consistent representation.

9. Comparing Representative Algorithms

Aspect Plane-Sweep Patch-based (PMVS) Learning-based (MVSNet family)
Principle Sweeps depth planes, evaluates photo-consistency per pixel Iteratively expands and filters small patches Regularizes a cost volume with a CNN and regresses depth
Accuracy Depends on depth resolution and cost-aggregation design; moderate to high Tends to be accurate for oblique or complex local shapes High accuracy under conditions close to the training data
Computational cost Easily GPU-parallelized, fast Tends to be slower than Plane-Sweep due to iterative processing Fast inference after training; training cost is separate
Robustness Weak in texture-poor regions Tends to leave holes in regions with few initial patches Comparatively robust to weak texture or repetitive patterns
Implementation difficulty Moderate (homography warping and cost aggregation) High (visibility management and iterative expansion design) High (requires training data and network design)
Representative implementations COLMAP dense, many commercial photogrammetry tools PMVS/CMVS MVSNet, subsequent learning-based methods

10. Relationship to Stereo and Depth Cameras

MVS shares its underlying principle — "find depth from correspondence across multiple viewpoints" — with stereo cameras and depth cameras, but they occupy different positions.

Depending on the application, robots or AR that need real-time performance suit stereo/depth cameras, while offline, high-precision 3D models for cultural-heritage documentation, architectural surveying, or photogrammetry suit MVS.

11. Difficult Conditions and Common Failure Cases

12. Practical Choices

13. Summary

Multi-View Stereo takes camera poses already known from SfM or calibration as given, and recovers dense depth using photo-consistency as its clue. The two classical approaches, Plane-Sweep and Patch-based, have different tradeoffs, and in recent years, MVSNet-family methods that learn a cost volume are pushing accuracy and robustness further. The whole flow runs through to fusing and meshing the resulting depth maps, and understanding the split — MVS sacrificing real-time performance for accuracy, versus stereo/depth cameras prioritizing real-time performance — is the foundation for making the right practical choice.

References

#Multi-View Stereo #MVS #3D Reconstruction #Plane Sweep #PMVS #MVSNet #Computer Vision