Contents — find the section you need

Neural 3D methods learn a scene representation from multiple images and camera poses, then render an image from a viewpoint that was not captured. Whereas SfM primarily estimates poses and sparse points, and MVS primarily reconstructs dense geometry, neural 3D also models how a scene looks from a position and viewing direction.

In brief

  • NeRF represents a scene as a function that returns density and radiance from position and viewing direction; 3D Gaussian Splatting optimizes many explicit 3D Gaussian elements.
  • Reproducing training views well does not prove that unseen views or geometry are correct.
  • Separate image metrics such as PSNR/SSIM from geometric evaluation of depth, points, normals, and poses. Novel-view synthesis is not a survey-accuracy proof.
Diagram 1 · Use the button to switch views
Neural 3D training and evaluation

Representation choices

NeRF evaluates (\sigma,\mathbf{c})=F_\theta(\mathbf{x},\mathbf{d}) at samples along camera rays and uses volume rendering to form an image. Training minimizes the difference between rendered and input images. 3D Gaussian Splatting projects Gaussians with position, covariance, opacity, and color, and optimizes these elements starting from sparse points.

Representation Strength Caveat
NeRF family Continuous radiance field and view-dependent appearance Many ray samples; training and rendering depend on setup
3D Gaussian Splatting Explicit projected elements can render quickly Validate density, unseen geometry, and dynamic objects separately
Mesh / MVS Surface geometry is explicit and easy to inspect Sensitive to reflections, occlusions, holes, and weak texture

Split training and evaluation views

Rendering every input image used for training only measures reconstruction of observed views. It does not establish interpolation into unseen regions. Split images or a camera trajectory into train, validation, and held-out views, and fix the view spacing, overlap, moving objects, and camera-pose source. If poses are estimated, record their uncertainty too.

PSNR, SSIM, and LPIPS can measure image similarity, but they measure appearance. Geometric evaluation should separately compare depth, points, normals, camera poses, and reprojection error. A blurred representation may improve an image score while placing a wall or thin structure incorrectly.

Common failure conditions

  • Mirrors, transparent surfaces, reflections, and changing illumination break the assumption that a point has consistent appearance.
  • Regions outside the training trajectory or hidden by occlusion may look plausible without strong evidence.
  • Moving people, vehicles, or plants treated as static geometry can create ghosts and duplicate surfaces.
  • If SfM poses or scale are unstable, rendered images alone cannot identify the cause.

Summary

Neural 3D learns a view-dependent scene representation from images and poses. NeRF and 3D Gaussian Splatting use different representations and rendering strategies, but both require a clear separation between training-view reconstruction and held-out-view generalization. Record visual quality, geometry, pose error, dynamic content, and view splits independently before interpreting a result.

Check your understanding
Does a high training-view PSNR prove that the 3D geometry is correct?

No. Check held-out views, depth or point-cloud metrics, and camera-pose error separately.

References

What to read next

Review camera poses and sparse points from input imagesStructure from Motion Primer — Recovering 3D and Camera Positions Together From an Unordered Set of PhotosReview classical dense reconstruction from multiple viewsMulti-View Stereo Primer — Filling a Sparse Point Cloud Into Dense 3D ShapeReview the backgroundMonocular Depth Estimation — From Relative Order to Metric Distance
#Neural 3D #NeRF #3D Gaussian Splatting #Novel View Synthesis #Computer Vision