Multiple View Geometry

You can calibrate a camera, correct its lens, recover its pose from known points or from matches to a second view, and turn two images into a metric point cloud by rectifying, matching, and triangulating every pixel.

11 parts, in reading order · 29 interactive figures

Start with part 1

  1. The Pinhole Camera — A Pixel Is a Ray

    Part 1 · Read · Oct 2026 · 4 interactive figures

    • A point's pixel comes from similar triangles: offset equals f times X over Z.
    • The 3×3 matrix K holds focal length f and principal point (cx, cy) in pixels.
    • Pixels count from the top-left corner, so add cx and cy to the triangle offset.
    • Homogeneous coordinates let K multiply first, then divide by the last number.

  2. Where Is the Camera? — R, C and t

    Part 2 · Read · Oct 2026 · 2 interactive figures

    • A camera is placed by its centre C̃ and rotation R; R's rows are the camera's right, down, and forward axes in world coordinates.
    • The translation vector t = −RC̃ is the world origin expressed in the camera's axes, not the camera's position.
    • The full projection chain combines R, t, and K into one 3×4 matrix P = K[R | t] applied to homogeneous world points.
    • A point is in front of the camera when its third coordinate after R(X − C̃) is positive, behind when negative.

  3. Points at Infinity — Horizons, Vanishing Points and the Floor

    Part 3 · Read · Oct 2026 · 3 interactive figures

    • Intersect any two image lines with one cross product and one division, no special cases for vertical or parallel lines.
    • Parallel image lines meet at a point at infinity (w=0) whose (u,v) gives their common direction.
    • Vanishing points of world directions are the first three columns of P = K[R|t]; translation t drops out completely.
    • Map any floor pixel back to its ground (X,Z) coordinates with the 3x3 homography H = first, second, and fourth columns of P.

  4. Finding the Camera From Known Points — P by Least Squares

    Part 4 · Read · Oct 2026 · 3 interactive figures

    • Two linear equations per known point build a 2n×12 matrix A whose null vector is the camera matrix P.
    • The DLT solves Ap=0 by SVD: the right singular vector for the smallest singular value gives P up to scale.

  5. Lens Distortion — Bending Straight Lines Back

    Part 5 · Read · Oct 2026 · 3 interactive figures

    • The lens bends points after dividing by depth and before K, moving them along their radius from the principal point.
    • OpenCV models radial distortion with even powers: x_d = x_u(1 + k1 r^2 + k2 r^4 + k3 r^6), where k1 = -0.2 for this lens.
    • Barrel distortion folds back at r = 1/sqrt(-3 k1); for k1 = -0.2 the fold radius is 1.291, safely outside the image corner at r = 0.8.

  6. What calibrateCamera Actually Does

    Part 6 · Read · Oct 2026 · 3 interactive figures

    • One photo of a flat board gives one 3×3 homography H found from corner pixels alone.
    • Each homography yields two linear equations on the 3×3 matrix ω, the protractor for pixels.
    • Three photos at different angles give enough equations to solve for ω and recover K.
    • A final polish refines K, lens distortion, and every photo's pose together.

  7. Epipolar Geometry — A Point Becomes a Line

    Part 7 · Read · Oct 2026 · 2 interactive figures

    • One camera loses depth; a second camera recovers it if we find the matching pixel.
    • The match for any pixel in image 1 must lie on a single epipolar line in image 2, reducing search from 640×480 to one line.
    • The essential matrix E = [t]×R is built from the triple product of the baseline, camera 1's ray, and camera 2's rotated ray all lying in one plane.
    • The fundamental matrix F = K⁻ᵀ E K⁻¹ maps a pixel x₁ directly to its epipolar line l' = F x₁ in pixel coordinates, requiring undistorted pixels.

  8. Triangulation — Two Rays That Almost Meet

    Part 8 · Read · Oct 2026 · 2 interactive figures

    • Four linear rows from two pixels give a 4×4 system whose smallest singular vector is the 3D point.
    • Noise across the epipolar line makes rays skew and miss; noise along it keeps them meeting but shifts depth.
    • Triangulation accuracy is set by the angle between the two rays, a better guide than the baseline alone.
    • OpenCV's triangulatePoints returns the same homogeneous 4-vector; divide by W and check its sign before use.

  9. From Matches to Motion — Four Cameras, One Right Answer

    Part 9 · Read · Oct 2026 · 3 interactive figures

    • Eight matches give one linear equation each to solve for F up to scale using the eight-point method.
    • Normalising pixel coordinates before fitting prevents large coordinate products from dominating the least-squares solution.
    • Forcing the fitted F to rank 2 by zeroing its smallest singular value restores a single epipole for all epipolar lines.

  10. Stereo Depth — Rectification and Z = fB/d

    Part 10 · Read · Oct 2026 · 2 interactive figures

    • Rectification turns camera 2 by 10° so every match lands on the same row in both pictures.
    • Disparity d = u1 - u2 is the slide along the row, here 125 px for the sample point.
    • Depth is Z = fB/d, with f = 500 px, B = 1 m, giving Z = 4 m for d = 125 px.
    • Error in Z grows with the square of distance, so depth gets worse fast as points move farther away.

  11. From Two Photos to a Point Cloud

    Part 11 · Read · Oct 2026 · 2 interactive figures

    • One pre-computed map undoes lens distortion and applies the rectifying turn for both cameras in a single backward lookup.
    • Stereo matching slides a window along each rectified row, computes SAD cost for every disparity, and picks the lowest cost as the match.
    • Smoothness regularization adds small penalty P1 for 1-pixel disparity steps and larger penalty P2 for jumps, aggregated along multiple scanline paths.
    • The 4x4 reprojection matrix Q converts each (u, v, d) into 3D via homogeneous division, implementing Z = fB/d with f=500 px and B=1 m.

Other series