Camera Calibration Primer — Recovering Lens Distortion and Intrinsic Parameters

Visual SLAM, Visual Odometry, epipolar geometry, and PnP all take for granted a correct mapping from pixel coordinates into 3D space. But light passing through a lens never forms an image the way a textbook-ideal pinhole would. What is the focal length, in pixel units? How far is the principal point offset from the image center? How much does a straight line bow on the image? Camera calibration is the process of solving for these quantities numerically. Skimp on calibration, and no matter how sophisticated the SLAM or SfM algorithm stacked on top of it is, a systematic error propagates outward from the very foundation.

0. 30-Second Summary

• Camera calibration is the process of estimating intrinsic parameters (focal length, principal point, skew) and distortion coefficients from known correspondences between 3D points and image points. Extrinsic parameters (the camera pose at each shot) are recovered at the same time.
• Lenses have both radial and tangential geometric distortion. The standard representation is the Brown–Conrady model, expressing radial distortion as k_1,k_2,k_3 and tangential distortion as p_1,p_2. Wide-angle and fisheye lenses call for a different model.
• Zhang's method — shooting a planar pattern (such as a checkerboard) from several poses, linearly solving for the intrinsic parameters from each view's homography, then refining including distortion via nonlinear optimization — is the standard calibration method in practice.
• Calibration quality is judged by reprojection error. Looking only at the mean value isn't enough — you also need to check the spatial distribution of error within the image and the variance across poses, or you'll miss localized distortion that wasn't fully captured.
• Calibration isn't a constant, valid forever once found. Zoom, focus, temperature, shock, and changes in resolution or cropping can all change the intrinsic parameters. Downstream Visual SLAM, VO, and PnP quietly degrade once this assumption breaks.

1. Revisiting the Pinhole Model From Coordinates

A world-coordinate point \mathbf{X}=(X,Y,Z,1)^\mathsf{T} (homogeneous coordinates) is projected onto the image, up to scale, in the pinhole model as

R\in SO(3) and \mathbf{t} are the rotation and translation from world coordinates to camera coordinates (the extrinsic parameters), and K is the intrinsic matrix

Here f_x,f_y are the focal lengths in pixel units, (c_x,c_y) is the principal point (where the optical axis intersects the image plane), and s is the skew (nearly zero on most modern sensors). Calibration is the inverse problem of recovering this K, the distortion coefficients, and each shot's R,\mathbf{t}, purely from observed data.

As the diagram shows, a single shot alone can't separate the intrinsic parameters from the extrinsic ones. Only by collecting several observations at different poses (tilt, distance) can K be pinned down nearly uniquely — and this is exactly the core idea behind Zhang's method, covered next.

2. A Lens Isn't an Ideal Pinhole: the Distortion Model

Real lenses have radial distortion and tangential distortion. Writing the normalized image coordinates as (x,y)=(X_c/Z_c,\,Y_c/Z_c) and the radius as r^2=x^2+y^2, the Brown–Conrady distortion model can be written as follows.

k_1,k_2,k_3 are the radial distortion coefficients, and p_1,p_2 are the tangential distortion coefficients. Radial distortion is a phenomenon where the image shrinks or bulges depending on distance from the lens center, showing up as the pronounced "barrel" distortion seen with wide-angle lenses, or the "pincushion" distortion seen with telephoto lenses. Tangential distortion is a smaller, asymmetric component arising from implementation imperfections where the lens group and the image sensor aren't perfectly parallel. The final pixel coordinates are obtained as \tilde{\mathbf{x}}_{px}=K(x_d,y_d,1)^\mathsf{T}.

For lenses with an extremely wide field of view, like fisheye lenses, the Brown–Conrady polynomial model tends to diverge near the edges and isn't practical. It's common to use an angle-based equidistant-projection approximation instead, like OpenCV's fisheye model. Rather than "memorizing one distortion model," it's important to keep in mind that the model should be chosen based on the lens's field of view and optical design.

3. Zhang's Method: Calibrating With a Planar Pattern

The most widely implemented method today is the planar-pattern calibration method Zhengyou Zhang published in 2000. No special 3D calibration rig is needed — all it requires is photographing a planar pattern, such as a printed checkerboard, from several poses, moving the camera or the pattern itself.

The mapping from the pattern plane (taking Z=0) at some pose i to the image can be written as a homography using the rotation matrix's first and second columns \mathbf{r}_1,\mathbf{r}_2 and the translation \mathbf{t}.

Here, each pose's H_i can be estimated linearly from the known grid points on the pattern and their image correspondences, using the DLT method covered in the Homography Primer. Using the constraint that the rotation matrix's columns are orthonormal — \mathbf{r}_1^\mathsf{T}\mathbf{r}_2=0,\ \|\mathbf{r}_1\|=\|\mathbf{r}_2\| — gives a linear equation in B=K^{-\mathsf{T}}K^{-1}:

which gives two such equations per pose (\mathbf{h}_1,\mathbf{h}_2 are H_i's first and second columns). Since B is a symmetric matrix with 6 degrees of freedom, given 3 or more poses, B can be found via linear least squares, and then K found in closed form through a procedure equivalent to Cholesky decomposition. This is why Zhang's method only needs "a few shots of a plane." However, if all the poses are nearly parallel to the image plane (fronto-parallel), the equations degenerate, so poses at several different tilts are required.

The closed-form solution is only an initial value that ignores distortion. From here, the standard two-stage approach in practice refines all parameters, including the distortion coefficients, via nonlinear optimization (usually Levenberg–Marquardt) using the reprojection error described in the next section as the objective function.

4. Reprojection Error and Parameter Optimization

Calibration's objective function is to minimize, over every pose and every grid point, the difference between the observed pixel and the projected position computed with the estimated parameters. Writing the observation at pose i, point j as \mathbf{u}_{ij}, and the corresponding known 3D point on the plane as \mathbf{X}_j,

is what gets minimized. \boldsymbol{\kappa}=(k_1,k_2,k_3,p_1,p_2) is the distortion coefficient vector, and \pi_d is the projection function including distortion. The unknowns are large — K (4–5 degrees of freedom), \boldsymbol{\kappa} (3–5 degrees of freedom), and R_i,\mathbf{t}_i per pose (6 degrees of freedom × number of poses) — but with enough observed points, the problem is well constrained. This formulation can be seen as a special case of the "jointly optimize camera pose and 3D structure" framework covered in the Bundle Adjustment Primer. Because the 3D-point coordinates are known and fixed in calibration, it's an easier subproblem than ordinary bundle adjustment.

It's common to report the root mean square (RMS) of the reprojection error \left\|\mathbf{u}_{ij}-\hat{\mathbf{u}}_{ij}\right\| as "calibration accuracy," but it's risky to call calibration done just by looking at the mean value. Always check the following as well:

• Whether there's variance across each pose's mean error (a pose with a particularly large error at a specific angle suggests pattern warping, motion blur, or uneven lighting)
• The spatial distribution of error within the image (systematic error remaining at the edges suggests the distortion model's order or type may be insufficient)
• Sub-pixel accuracy of grid-point detection (if corner detection itself is unstable, no amount of optimization can distinguish that from model error)

5. Implementation Skeleton in OpenCV

The typical single-camera calibration procedure looks like this. Fixing the shooting conditions (resolution, zoom, focus), and photographing the pattern at the image's four corners, center, and several tilts, accounts for 90% of a good calibration.

import cv2 as cv
import numpy as np

pattern_size = (9, 6)  # number of internal corners
objp = np.zeros((pattern_size[0]*pattern_size[1], 3), np.float32)
objp[:, :2] = np.mgrid[0:pattern_size[0], 0:pattern_size[1]].T.reshape(-1, 2)
objp *= square_size_m  # measured side length of one square [m]

obj_points, img_points = [], []
for gray in calibration_images:  # multiple frames shot at different poses
    found, corners = cv.findChessboardCorners(gray, pattern_size)
    if found:
        corners = cv.cornerSubPix(gray, corners, (11, 11), (-1, -1),
                                   (cv.TERM_CRITERIA_EPS + cv.TERM_CRITERIA_MAX_ITER, 30, 0.001))
        obj_points.append(objp)
        img_points.append(corners)

ret, K, dist, rvecs, tvecs = cv.calibrateCamera(
    obj_points, img_points, gray.shape[::-1], None, None)
# ret is the RMS reprojection error [px]. Also compute per-pose error separately to check.

For wide-angle or fisheye lenses, use the cv.fisheye.calibrate family of APIs rather than the ordinary calibrateCamera. For a stereo rig, use cv.stereoCalibrate to find the relative pose in addition to both cameras' intrinsic parameters, and cv.stereoRectify to align the left and right images to horizontal epipolar lines. This procedure connects directly to the baseline design covered in How Stereo Cameras Work and How Depth Cameras Work.

6. Why Calibration Drift Breaks Downstream Systems

Much of the math in Visual SLAM, VO, and PnP is written under the assumption that calibration values stay constant as long as shooting conditions don't change. Essential Matrix estimation in the Epipolar Geometry Primer assumes normalized coordinates K^{-1}\tilde{\mathbf{x}}; the reprojection-error minimization in the PnP Primer also treats K as known. If K or the distortion coefficients no longer match the actual optical system, every one of these calculations is effectively being carried out with "the wrong ruler."

Specifically, calibration values quietly degrade for reasons such as these:

• Zoom/focus changes: with a lens whose focal length changes, f_x,f_y change from shot to shot. Allowing autofocus to operate means the intrinsic parameters at calibration time and at runtime won't match.
• Temperature and mechanical shock: a slight positional shift in the lens barrel or sensor mounting moves the principal point and distortion coefficients. This is particularly non-negligible for outdoor, automotive, and drone applications.
• Changes in resolution, cropping, or digital zoom: since K is a pixel-unit parameter, if the image is resized or cropped, f_x,f_y,c_x,c_y must also be updated according to the scale. Forgetting to do this is an extremely common mistake.
• Deformation of a left/right camera rig: in stereo setups, if the relative pose (extrinsic calibration) — not just the intrinsic parameters — shifts slightly over time, a systematic error rides on the depth computed from disparity.

These errors are hard to notice in a single, isolated pose estimate. But in systems that sequentially integrate pose over time, like VO or SLAM, systematic reprojection error accumulates as drift, producing a distorted map that loop closure can't fully correct. In production systems, it's advisable to monitor calibration drift online using fixed, known 3D features (straight lines on a building, signs of known size), or to build in a routine of periodic recalibration.

7. Common Failures and Countermeasures

8. Summary

Camera calibration is the first step that turns an image from "just a 2D array" into "a geometrically interpretable observation." The pinhole model's intrinsic and extrinsic parameters, together with the radial and tangential distortion coefficients, are recovered from observations at several poses — as in Zhang's planar-pattern method — and finished off with nonlinear optimization of the reprojection error. The quality of this calibration doesn't show up in any single algorithm's own accuracy metric, but it's the premise underlying every equation in every downstream stage — epipolar geometry, PnP, Visual SLAM, SfM, bundle adjustment. Calibration isn't a one-time ritual — it's part of a pipeline that should keep being monitored as shooting conditions change.

References

• Zhang, A Flexible New Technique for Camera Calibration (IEEE TPAMI, 2000)
• Heikkilä & Silvén, A Four-step Camera Calibration Procedure with Implicit Image Correction (CVPR 1997)
• Hartley & Zisserman, Multiple View Geometry in Computer Vision (authors' official page)
• OpenCV — Camera Calibration and 3D Reconstruction
• OpenCV — Camera Calibration tutorial
• OpenCV — fisheye calibration module
• Kalibr (camera/IMU calibration tool, official repository)