Lens Distortion — Bending Straight Lines Back
- 1The Pinhole Camera — A Pixel Is a Ray
- 2Where Is the Camera? — R, C and t
- 3Points at Infinity — Horizons, Vanishing Points and the Floor
- 4Finding the Camera From Known Points — P by Least Squares
- 5Lens Distortion — Bending Straight Lines Back
- 6What calibrateCamera Actually Does
- 7Epipolar Geometry — A Point Becomes a Line
- 8Triangulation — Two Rays That Almost Meet
- 9From Matches to Motion — Four Cameras, One Right Answer
- 10Stereo Depth — Rectification and Z = fB/d
- 11From Two Photos to a Point Cloud
Summary
Real lenses bend straight lines, most of all at the edges of the picture. This read puts the bend where it belongs, between dividing by depth and K; builds the radial and tangential terms OpenCV's distCoeffs hold; and shows why undoing them takes a loop of guesses. After it you can read a distCoeffs vector, predict where a corner pixel moves, and say what undistort does to a whole picture.
Table of Contents
- 1. Where the Lens Bends2
- 2. Barrel, Pincushion and Mustache2
- 3. A Lens That Sits Crooked2
- 4. Undoing It Takes a Few Guesses3
- 5. Undistorting a Whole Picture3
- 6. In OpenCV4
- 7. What Came Next4
- 8. References4
- 9. Connections4
Calibrate a camera and print its residual: corners near the edges are off by more than corners in the middle. The pinhole is fine. The glass in front of it is not.
Everything so far in this series assumed a perfect pinhole: a point, its ray, one straight line through the centre, and a pixel where that line meets the image plane. A real camera has a lens there instead, and a lens does not keep straight lines straight. The book says it plainly: the cheaper the lens and the shorter its focal length, the worse it gets (Hartley & Zisserman, Ch. 7 Computation of the Camera Matrix P, §7.4, p. 189).
Look through the peephole of a door. The frame of the corridor wall bows outward, the middle of the view looks normal, and the closer you look to the rim, the more everything curls. That is barrel distortion, and it is strongest at the edge of the view. Keep the peephole in mind; every equation below is that picture in numbers.
Brush up on these first ;)
- The Pinhole Camera — A Pixel Is a Ray: the normalised point , a pixel written in units of the focal length, with (0, 0) on the principal axis. The lens lives exactly there.
- Finding the Camera From Known Points — P by Least Squares: the reprojection error, the pixel distance between where a point should land and where it was seen. Distortion is why that error grows toward the edges.
1. Where the Lens Bends
Camera: focal length and principal point
The grey grid is what a pinhole would photograph: nine straight columns and seven straight rows across the 640 × 480 picture. The blue grid is what our lens delivers. Read it before any symbol:
- The middle row and middle column stay straight. Every other line bows outward, its middle further from the centre than its ends, the outer lines most.
- The ring at the top-left corner is the pixel (0, 0). The lens moves it along the diagonal, toward the centre, to the rust dot.
- The dashed circles are distances from the principal point (320, 240), in units of the focal length. Points on one circle all move by the same fraction.
That last bullet is the whole idea. The lens does not care about rows, columns or pixels. It cares about one thing: how far out from the axis a point is. So the bend is applied to the normalised point, after the divide by depth and before K, where "how far out" has a meaning that does not depend on the sensor (H&Z Ch. 7 Computation of the Camera Matrix P, §7.4, p. 190).
Let's write the book's version first. Take the ideal normalised point , the one a pinhole would produce, and its distance from the axis . The lens moves it along its own radius:
- : the undistorted point, the pinhole's answer, in units of f (no unit).
- : the distorted point, where the lens actually puts it.
- : the distortion factor, a function of the radius only, with : the axis never moves (H&Z Ch. 7 Computation of the Camera Matrix P, eq. 7.7, p. 190). The book expands it as a power series in with coefficients (H&Z Ch. 7 Computation of the Camera Matrix P, p. 191).
OpenCV writes the same idea with even powers only, which is the form you will actually type (outside the book, [OPEN24], after [BROW66]):
- : the radial coefficients, no unit. The lens of this series has and nothing else.
Then K turns the bent point into pixels, exactly as in read 1. Only the input to K has changed:
Numbers are always satisfying, so let's push the corner pixel through all three equations:
- The pixel (0, 0) is , so .
- , so , and .
- The bent point is , and K sends it to (40.96, 30.72).
- The corner is seen 51.2 px toward the centre, 12.8 % of its distance from the centre. That is the peephole, in pixels.
Now walk inward. The half-width point (640, 240) has and ; it moves 26.21 px. The worked pixel of read 1, (357.5, 215), has ; it moves 0.073 px, less than a tenth of a pixel. Distortion lives at the edges.
Why it exists. The pinhole puts every corner where straight lines say, and a real lens does not. Leave the bend out and K tries to absorb it: in the book's own grid calibration the focal length came out 1673.3 px without the correction and 1580.5 px with it, and the residual fell from 0.365 to 0.179 px (H&Z Ch. 7 Computation of the Camera Matrix P, Table 7.3, p. 192).
What it assumes. - The bend depends on the radius only: every point on one dashed circle moves by the same fraction, straight toward or away from the centre (H&Z Ch. 7 Computation of the Camera Matrix P, eq. 7.7, p. 190). - The radius is measured in units of f with square pixels; if the aspect ratio is not 1, it has to be corrected before r is computed (H&Z Ch. 7 Computation of the Camera Matrix P, p. 191). - The centre of the bend is the principal point (the NOTE below says where the book differs). In the book's own 640 × 480 example the fitted centre is (321.87, 241.18), close to but not on the image centre (H&Z Ch. 7 Computation of the Camera Matrix P, ex. 7.3, p. 191).
Where it breaks. - A lens that sits crooked moves points sideways as well, not only along the radius. That is Section 3. - A very wide lens. The normalised point is , so r is the tangent of the angle off the axis, and at 90° it is infinite: no series in r reaches a fisheye's rim. OpenCV has a separate fisheye model for it (Table. 1). - A barrel series runs out before that: our folds back at r = 1.291, 52.2° off the axis (Section 2).
NOTE: The book's series has odd powers as well (, ) and no tangential terms; OpenCV's has even powers only, plus the tangential terms of Section 3. The idea is the same, a factor that depends on the radius, but every OpenCV formula in this read is outside the book. The book also allows the centre of the bend to sit away from the principal point (H&Z Ch. 7 *Computation of the Camera Matrix P*, p. 191); OpenCV puts it at the principal point. And the book fits its series in pixels as a correction, applied to the measured point to get the ideal one (H&Z Ch. 7 *Computation of the Camera Matrix P*, p. 191); OpenCV's coefficients bend the ideal point into the measured one. A κ from the book is not a k under another name.
Try this: drag the ring from the corner to the centre. 51 px of bend falls to almost nothing.
2. Barrel, Pincushion and Mustache
Camera: focal length and principal point
The same grid, with flipped to +0.2. Then press mustache: negative and positive.
The sign of decides the look:
- Barrel (): , the edges are pulled in, straight lines bow outward like the sides of a barrel. Our lens, the peephole.
- Pincushion (): , the edges are pushed out, straight lines bow in toward the centre and the corners stretch away like a pincushion.
- Mustache (, , and turning back up inside the picture): barrel near the middle, pincushion toward the edges, so a straight line near the edge wiggles like a mustache.
How strong can get before the formula breaks? Look at the radius alone. A point at radius lands at . For barrel distortion, first grows with , then stops growing, then shrinks: two different points would land on the same pixel. The turn is where the slope is zero:
- : the distorted radius, , in units of f.
- : the radius where the lens folds back on itself. Beyond it the map is no longer one-to-one.
For , 1.291, safely outside the picture's corner at . The figure draws the fold as a red dashed circle whenever it comes inside the picture; with set too, the condition becomes . Press a lens that folds to see it.
Why it exists. A fitted is just a number. This says up to which radius it still describes a lens: one ideal point in, one pixel out.
What it assumes. - A barrel lens, . With the slope never reaches 0: pincushion does not fold. - alone. With the fold moves, by the condition above.
Where it breaks. - Past two ideal points land on one pixel, so nothing can undo the lens there, and the guessing loop of Section 4 stops converging at exactly this radius. For a lens that folds (, ) the fold is at r = 0.661, inside the corner's 0.8. Try this: press a lens that folds and watch the red circle cut off the corners. - Coefficients that put the fold inside the picture are the failed calibration of the NOTE below.
NOTE: OpenCV's documentation says the same thing from the other side: radial distortion of a real lens is monotonic, and a calibration that returns a non-monotonic factor should be treated as a failure, even when it looks fine near the centre [OPEN24]. If your fitted coefficients put the fold inside the picture, do not trust them at the corners.
3. A Lens That Sits Crooked
Camera: focal length and principal point
Here is 0 and is 0.01.
If the glass is not centred on the sensor, or sits a little tilted, the bend is no longer the same in every direction. Brown called this decentering distortion [BROW66]; OpenCV calls its two coefficients tangential. The shift is no longer along the radius: the grid shears a little, more on one side than the other. The two terms are added to the radial result (outside the book, [BROW66], [OPEN24]):
- : the tangential coefficients, no unit. They are usually far smaller than .
- On the right, are the radial result of Section 1; the arrow means "add the tangential terms to it".
At the corner, with alone and no radial bend, the shift is (0.00614, 0.01101) in normalised units, or (3.07, 5.50) px after K. Small next to 51.2 px, but not zero, and it is not along the diagonal: that is the "crooked" part.
Why it exists. Radial terms can only move a point along its radius. Glass that is off centre or tilted also pushes points sideways, and with 's alone that shift has nowhere to go but the residual.
What it assumes. - The lens is only slightly off: the shift is quadratic in the point, 0 on the axis and largest at the corners, and are small.
Where it breaks.
- On a well-centred lens have nothing to describe, and fitting them only hands the noise two more knobs. Calibration can switch them off (CALIB_ZERO_TANGENT_DIST, What calibrateCamera Actually Does).
- The shift is computed from the ideal point, which is unknown when you go back, so undoing it needs the guessing loop of Section 4 too.
- The coefficients in the wrong slots: the NOTE below.
NOTE, the classic bug: OpenCV stores the coefficients as
distCoeffsin the order (k₁, k₂, p₁, p₂, k₃), with k₃ last, after the tangential pair [OPEN24]. It is a vector, not a matrix. Our lens is (−0.2, 0, 0, 0, 0). Write (k₁, k₂, k₃, p₁, p₂) by habit and a radial lens turns into a crooked one.
4. Undoing It Takes a Few Guesses
Camera: focal length and principal point
This is the top-left corner of the picture, zoomed in. The rust dot is where the lens put the corner, (40.96, 30.72). The hollow ring is where it belongs, (0, 0). The blue dots are guesses, and the small plot is the distance left after each one.
- The first guess lands 13.591 px from the true corner; the second, 3.884 px.
- Each guess sits on the same diagonal: the lens only ever moved the point along its radius.
- The gaps shrink by nearly the same factor every time, which is why the curve in the plot drops like a stone.
Why guess at all? Going forward is one formula: give me , and Section 1 gives you . Going back has no formula, because depends on the radius of , the very thing we are trying to find. Even with alone, undoing it means solving a cubic in . So guess, correct, repeat:
- : the seen point in normalised units, fixed throughout.
- : the radius of the current guess. Each round recomputes there and divides again.
The first round uses the radius of itself, which is a little too small, so it gives and the guess : back through K, 13.591 px short of the corner. Then:
- after 1–8 guesses the error is 13.591, 3.884, 1.132, 0.331, 0.097, 0.029, 0.008 and 0.002 px;
- five guesses get under 0.1 px.
Why does the error fall by the same factor each time? Write one round as a function of the radius, . Near the answer, a guess that is off by comes back off by :
- : the slope of one round at the answer , no unit.
That is 0.2936: each guess divides the error by about 3.41. A stronger lens has a larger slope and needs more rounds; press a stronger lens and watch the plot flatten out.
Why it exists. Every use of a calibrated camera goes from pixels back to rays, and the lens formula only runs forward. This loop is how a seen pixel gets its pinhole point back.
What it assumes. - The seen point is a good first guess: the bend is small. - Each round shrinks the error: at the answer.
Where it breaks.
- In general . For a barrel lens , so is positive and reaches 1 exactly where : the slope of Section 2. The loop converges inside the fold, slower the nearer it gets, and not beyond it.
- A fixed count suits a mild lens only. Five guesses leave 0.097 px for ours, 3.699 px for a stronger lens (, slope 0.5773) and 8.525 px at . Set the stopping rule yourself with undistortPointsIter [OPEN24]. Try this: press a stronger lens and set the guesses to five.
OpenCV's undistortPoints runs this loop: the same division, starting from the seen point, five rounds by default, with undistortPointsIter when you want to set the stopping rule yourself [OPEN24]. One more thing it does that surprises people: without a P argument it returns , not pixels. You get (−0.64, −0.48) back for our corner, the normalised point of read 1, at work. Pass the camera matrix as P and you get the pixel (0, 0) instead.
Try this: press play. Each guess lands about 3.4 times closer.
5. Undistorting a Whole Picture
Camera: focal length and principal point
Camera 1: move it and turn it
Camera 1 looks at a doorframe and a long board: eight edges, all straight in the world. Through our lens they bow (blue). The rust curves are the same edges corrected with the guessed .
- At a guess of 0 nothing is corrected, and the rust lies on the blue.
- As the guess moves toward −0.2 the bow flattens; past it, the edges bow the other way.
- The straightness error reaches 0 px at exactly −0.2, the true lens.
That is the method the book calls "a simple and more general approach": straight lines in the scene must image as straight lines, so choose the coefficients (and the centre) that make them straight again. No calibration pattern, only lines you know are straight (H&Z Ch. 7 Computation of the Camera Matrix P, §7.4, p. 191). Photogrammetry calls it plumb-line correction (H&Z Ch. 7 Computation of the Camera Matrix P, §7.5.1, p. 193). The other way the book gives is to fold the into the same refinement that fits P, using the reprojection error of read 4 (H&Z Ch. 7 Computation of the Camera Matrix P, p. 191). That is what the next read does.
Once you know the lens, how do you straighten a whole photograph? Not by pushing each source pixel forward; that leaves holes. You run the map the other way, per output pixel:
- for each pixel of the straight output picture, take its normalised point, bend it with the lens formula of Section 1 (forward, so no guessing), and look up the colour at that spot in the original;
- OpenCV builds that lookup table once with
initUndistortRectifyMapand applies it withremap;undistortdoes both in one call [OPEN24].
And then a choice appears. Switch the first figure to after undistortion. The rust outline is where the border of the original photo lands once it is straightened. For our barrel lens it lands outside the frame all the way round, furthest out at the corners and least at the middle of each edge, so the straightened picture no longer fills a rectangle:
- keep the old K and the 640 × 480 frame, and every output pixel has a source, but the corners of the original fall off the edge;
- shrink the picture to keep every source pixel, and the gaps between the rust outline and the rectangle have nothing to show: black borders.
OpenCV leaves that choice to you as a new camera matrix, from getOptimalNewCameraMatrix(K, D, size, alpha): alpha 0 keeps only valid pixels (a crop), alpha 1 keeps every source pixel (black borders), and anything between is a blend. It also returns the rectangle of all-valid pixels [OPEN24].
NOTE: You do not always need to warp the picture. If all you want is points (corners, matches, a residual), undistort the points and leave the pixels alone. The book says detecting features in the original picture is often the better choice, because warping averages the noise and can alias fine detail; linking edges into straight lines is the exception, best done after warping (H&Z Ch. 7 *Computation of the Camera Matrix P*, pp. 192–193).
6. In OpenCV
| Here | OpenCV | Shape | Series value |
|---|---|---|---|
| (k₁, k₂, p₁, p₂, k₃) | distCoeffs | 1×5 (or 4, 8, 12, 14) | (−0.2, 0, 0, 0, 0) |
| x_d → x_u (normalised) | undistortPoints(src, K, D) | N×1, 2 channels | (40.96, 30.72) → (−0.64, −0.48) |
| … back in pixels | undistortPoints(src, K, D, P=K) | N×1, 2 channels | → (0, 0) |
| control the loop | undistortPointsIter(src, K, D, R, P, criteria) | — | five guesses: 0.097 px (also the default count) |
| a whole picture | undistort(img, K, D, newCameraMatrix), or initUndistortRectifyMap + remap | image | — |
| the new K | getOptimalNewCameraMatrix(K, D, size, alpha) | 3×3, plus a rectangle | alpha 0 crops, 1 keeps black borders |
| very wide lenses | the cv::fisheye functions (a different lens formula) | — | — |
Table. 1 This read in OpenCV 4.x [OPEN24]. With five coefficients OpenCV's full formula (which also allows k₄–k₆ and thin-prism terms) reduces to Sections 1 and 3.
7. What Came Next
So far each piece came from its own experiment: K and R from known 3-D points in read 4, from lines we knew were straight here. Nobody calibrates a camera like that. In practice one function takes a pile of checkerboard photos and hands back K, the five coefficients and a pose for every photo. Who does all of it at once, from a printed checkerboard, and what is that function actually solving? That is What calibrateCamera Actually Does.
8. References
- [HART04] R. Hartley, A. Zisserman, Multiple View Geometry in Computer Vision, 2nd ed., Cambridge University Press, 2004. Chapters used in this read: Ch. 7 Computation of the Camera Matrix P.
- [OPEN24] OpenCV 4.x documentation, Camera Calibration and 3D Reconstruction (calib3d): the distortion formula,
distCoeffs,undistortPoints,undistort,initUndistortRectifyMap,getOptimalNewCameraMatrix,cv::fisheye. - [BROW66] D. C. Brown, "Decentering Distortion of Lenses", Photogrammetric Engineering 32(3), 1966.
9. Connections
What this page builds on, cites or links to, and what points back at it. A plain box is a “builds on”; “cited” and “linked in the text” are the lighter links. Hover a box for a preview.
Related
- Read · this builds on it (earlier in the series)
- Read · builds on this (next in the series)
- Read · same series: Multiple View Geometry
- Read · this builds on it (earlier in the series)
- Read · same series: Multiple View Geometry
- Read · builds on this (next in the series)
Go deeper
-
Start withRead
-
Read
-
Read
-
Read
-
You are hereLens Distortion — Bending Straight Lines BackRead
-
ThenRead
-
Read
-
Read
-
Read
-
Read
-
Read
Go broader
- Sits under
- SeriesMultiple View Geometrypart 5 of 11 · start from the first part
- Its tags on the map
- Ask the mapWhat connects Lens distortion and …type the other page's name; the map answers with the chain of links between them