Lens Distortion — Bending Straight Lines Back

  1. 1The Pinhole Camera — A Pixel Is a Ray
  2. 2Where Is the Camera? — R, C and t
  3. 3Points at Infinity — Horizons, Vanishing Points and the Floor
  4. 4Finding the Camera From Known Points — P by Least Squares
  5. 5Lens Distortion — Bending Straight Lines Back
  6. 6What calibrateCamera Actually Does
  7. 7Epipolar Geometry — A Point Becomes a Line
  8. 8Triangulation — Two Rays That Almost Meet
  9. 9From Matches to Motion — Four Cameras, One Right Answer
  10. 10Stereo Depth — Rectification and Z = fB/d
  11. 11From Two Photos to a Point Cloud

Summary

Real lenses bend straight lines, most of all at the edges of the picture. This read puts the bend where it belongs, between dividing by depth and K; builds the radial and tangential terms OpenCV's distCoeffs hold; and shows why undoing them takes a loop of guesses. After it you can read a distCoeffs vector, predict where a corner pixel moves, and say what undistort does to a whole picture.

Table of Contents

  1. 1. Where the Lens Bends2
  2. 2. Barrel, Pincushion and Mustache2
  3. 3. A Lens That Sits Crooked2
  4. 4. Undoing It Takes a Few Guesses3
  5. 5. Undistorting a Whole Picture3
  6. 6. In OpenCV4
  7. 7. What Came Next4
  8. 8. References4
  9. 9. Connections4
TrivediInformational[Page 1]
S. TrivediOct 2026

Calibrate a camera and print its residual: corners near the edges are off by more than corners in the middle. The pinhole is fine. The glass in front of it is not.

Everything so far in this series assumed a perfect pinhole: a point, its ray, one straight line through the centre, and a pixel where that line meets the image plane. A real camera has a lens there instead, and a lens does not keep straight lines straight. The book says it plainly: the cheaper the lens and the shorter its focal length, the worse it gets (Hartley & Zisserman, Ch. 7 Computation of the Camera Matrix P, §7.4, p. 189).

Look through the peephole of a door. The frame of the corridor wall bows outward, the middle of the view looks normal, and the closer you look to the rim, the more everything curls. That is barrel distortion, and it is strongest at the edge of the view. Keep the peephole in mind; every equation below is that picture in numbers.

one camera: a world point becomes a pixelworld pointX (metres)camera frameR X + tdivide by depth(x/z, y/z)lens bends itk₁ k₂ p₁ p₂into pixelsKpixel(u, v)two cameras: two pixels become a point againtwo pixelsx₁ ↔ x₂on one linel′ = F x₁camera 2R, t from Ecross raysX from x₁ x₂turn to rowsv₁ = v₂depthZ = fB/dpoint cloudevery pixelthis read: the one stage that is not a matrix
Fig. 1 Where this read sits: the lens bends the point after dividing by depth and before K turns it into pixels. It is the one stage of the chain that is not a matrix.

Brush up on these first ;)

1. Where the Lens Bends

-0.200
Camera: focal length and principal point
500320240
What the lens does to a straight grid. The grey grid is what a pinhole would see; the blue grid is what the lens delivers. Each point slides along its own radius by a factor that depends only on how far out it is, so the middle barely moves and the corners move most. Drag the ring to pick a pixel; switch to "after undistortion" to see where the picture's own border ends up. Open the camera panel to change the focal length and the principal point.

The grey grid is what a pinhole would photograph: nine straight columns and seven straight rows across the 640 × 480 picture. The blue grid is what our lens delivers. Read it before any symbol:

  • The middle row and middle column stay straight. Every other line bows outward, its middle further from the centre than its ends, the outer lines most.
  • The ring at the top-left corner is the pixel (0, 0). The lens moves it along the diagonal, toward the centre, to the rust dot.
  • The dashed circles are distances from the principal point (320, 240), in units of the focal length. Points on one circle all move by the same fraction.

That last bullet is the whole idea. The lens does not care about rows, columns or pixels. It cares about one thing: how far out from the axis a point is. So the bend is applied to the normalised point, after the divide by depth and before K, where "how far out" has a meaning that does not depend on the sensor (H&Z Ch. 7 Computation of the Camera Matrix P, §7.4, p. 190).

Let's write the book's version first. Take the ideal normalised point (xu,yu), the one a pinhole would produce, and its distance from the axis r. The lens moves it along its own radius:

(xdyd)=L(r)(xuyu),r=xu2+yu2L(r)=1+κ1r+κ2r2+κ3r3+…(1)
  • (xu,yu): the undistorted point, the pinhole's answer, in units of f (no unit).
  • (xd,yd): the distorted point, where the lens actually puts it.
  • L(r): the distortion factor, a function of the radius only, with L(0)=1: the axis never moves (H&Z Ch. 7 Computation of the Camera Matrix P, eq. 7.7, p. 190). The book expands it as a power series in r with coefficients κi (H&Z Ch. 7 Computation of the Camera Matrix P, p. 191).

OpenCV writes the same idea with even powers only, which is the form you will actually type (outside the book, [OPEN24], after [BROW66]):

xd=xu(1+k1r2+k2r4+k3r6)yd=yu(1+k1r2+k2r4+k3r6)r2=xu2+yu2(2)
  • k1,k2,k3: the radial coefficients, no unit. The lens of this series has k1=−0.2 and nothing else.

Then K turns the bent point into pixels, exactly as in read 1. Only the input to K has changed:

(uv1)=[50003200500240001](xdyd1)(3)

Numbers are always satisfying, so let's push the corner pixel through all three equations:

  • The pixel (0, 0) is 𝐱^=K−1(0,0,1)⊤, so (xu,yu)=(−0.64,−0.48).
  • r2=0.64, so r=0.8, and L=1+(−0.2)(0.64)=0.872.
  • The bent point is (xd,yd)=(−0.55808,−0.41856), and K sends it to (40.96, 30.72).
  • The corner is seen 51.2 px toward the centre, 12.8 % of its distance from the centre. That is the peephole, in pixels.

Now walk inward. The half-width point (640, 240) has r=0.64 and L=0.9181; it moves 26.21 px. The worked pixel of read 1, (357.5, 215), has r2=0.008125; it moves 0.073 px, less than a tenth of a pixel. Distortion lives at the edges.

Why it exists. The pinhole puts every corner where straight lines say, and a real lens does not. Leave the bend out and K tries to absorb it: in the book's own grid calibration the focal length came out 1673.3 px without the correction and 1580.5 px with it, and the residual fell from 0.365 to 0.179 px (H&Z Ch. 7 Computation of the Camera Matrix P, Table 7.3, p. 192).

What it assumes. - The bend depends on the radius only: every point on one dashed circle moves by the same fraction, straight toward or away from the centre (H&Z Ch. 7 Computation of the Camera Matrix P, eq. 7.7, p. 190). - The radius is measured in units of f with square pixels; if the aspect ratio is not 1, it has to be corrected before r is computed (H&Z Ch. 7 Computation of the Camera Matrix P, p. 191). - The centre of the bend is the principal point (the NOTE below says where the book differs). In the book's own 640 × 480 example the fitted centre is (321.87, 241.18), close to but not on the image centre (H&Z Ch. 7 Computation of the Camera Matrix P, ex. 7.3, p. 191).

Where it breaks. - A lens that sits crooked moves points sideways as well, not only along the radius. That is Section 3. - A very wide lens. The normalised point is (X/Z,Y/Z), so r is the tangent of the angle off the axis, and at 90° it is infinite: no series in r reaches a fisheye's rim. OpenCV has a separate fisheye model for it (Table. 1). - A barrel series runs out before that: our k1=−0.2 folds back at r = 1.291, 52.2° off the axis (Section 2).

NOTE: The book's series has odd powers as well (κ1r, κ3r3) and no tangential terms; OpenCV's has even powers only, plus the tangential terms of Section 3. The idea is the same, a factor that depends on the radius, but every OpenCV formula in this read is outside the book. The book also allows the centre of the bend to sit away from the principal point (H&Z Ch. 7 *Computation of the Camera Matrix P*, p. 191); OpenCV puts it at the principal point. And the book fits its series in pixels as a correction, applied to the measured point to get the ideal one (H&Z Ch. 7 *Computation of the Camera Matrix P*, p. 191); OpenCV's coefficients bend the ideal point into the measured one. A κ from the book is not a k under another name.

Try this: drag the ring from the corner to the centre. 51 px of bend falls to almost nothing.

2. Barrel, Pincushion and Mustache

0.200
Camera: focal length and principal point
500320240
What the lens does to a straight grid. The grey grid is what a pinhole would see; the blue grid is what the lens delivers. Each point slides along its own radius by a factor that depends only on how far out it is, so the middle barely moves and the corners move most. Drag the ring to pick a pixel; switch to "after undistortion" to see where the picture's own border ends up. Open the camera panel to change the focal length and the principal point.

The same grid, with k1 flipped to +0.2. Then press mustache: k1 negative and k2 positive.

The sign of k1 decides the look:

  • Barrel (k1<0): L<1, the edges are pulled in, straight lines bow outward like the sides of a barrel. Our lens, the peephole.
  • Pincushion (k1>0): L>1, the edges are pushed out, straight lines bow in toward the centre and the corners stretch away like a pincushion.
  • Mustache (k1<0, k2>0, and L turning back up inside the picture): barrel near the middle, pincushion toward the edges, so a straight line near the edge wiggles like a mustache.

How strong can k1 get before the formula breaks? Look at the radius alone. A point at radius r lands at rd=r(1+k1r2). For barrel distortion, rd first grows with r, then stops growing, then shrinks: two different points would land on the same pixel. The turn is where the slope is zero:

drddr=1+3k1r2=0rfold=1−3k1(4)
  • rd: the distorted radius, xd2+yd2, in units of f.
  • rfold: the radius where the lens folds back on itself. Beyond it the map is no longer one-to-one.

For k1=−0.2, rfold= 1.291, safely outside the picture's corner at r=0.8. The figure draws the fold as a red dashed circle whenever it comes inside the picture; with k2 set too, the condition becomes 1+3k1r2+5k2r4=0. Press a lens that folds to see it.

Why it exists. A fitted k1 is just a number. This says up to which radius it still describes a lens: one ideal point in, one pixel out.

What it assumes. - A barrel lens, k1<0. With k1>0 the slope 1+3k1r2 never reaches 0: pincushion does not fold. - k1 alone. With k2 the fold moves, by the condition above.

Where it breaks. - Past rfold two ideal points land on one pixel, so nothing can undo the lens there, and the guessing loop of Section 4 stops converging at exactly this radius. For a lens that folds (k1=−0.4, k2=−0.5) the fold is at r = 0.661, inside the corner's 0.8. Try this: press a lens that folds and watch the red circle cut off the corners. - Coefficients that put the fold inside the picture are the failed calibration of the NOTE below.

NOTE: OpenCV's documentation says the same thing from the other side: radial distortion of a real lens is monotonic, and a calibration that returns a non-monotonic factor should be treated as a failure, even when it looks fine near the centre [OPEN24]. If your fitted coefficients put the fold inside the picture, do not trust them at the corners.

3. A Lens That Sits Crooked

000.01
Camera: focal length and principal point
500320240
What the lens does to a straight grid. The grey grid is what a pinhole would see; the blue grid is what the lens delivers. Each point slides along its own radius by a factor that depends only on how far out it is, so the middle barely moves and the corners move most. Drag the ring to pick a pixel; switch to "after undistortion" to see where the picture's own border ends up. Open the camera panel to change the focal length and the principal point.

Here k1 is 0 and p1 is 0.01.

If the glass is not centred on the sensor, or sits a little tilted, the bend is no longer the same in every direction. Brown called this decentering distortion [BROW66]; OpenCV calls its two coefficients tangential. The shift is no longer along the radius: the grid shears a little, more on one side than the other. The two terms are added to the radial result (outside the book, [BROW66], [OPEN24]):

xd←xd+2p1xuyu+p2(r2+2xu2)yd←yd+p1(r2+2yu2)+2p2xuyu(5)
  • p1,p2: the tangential coefficients, no unit. They are usually far smaller than k1.
  • On the right, xd,yd are the radial result of Section 1; the arrow means "add the tangential terms to it".

At the corner, with p1=0.01 alone and no radial bend, the shift is (0.00614, 0.01101) in normalised units, or (3.07, 5.50) px after K. Small next to 51.2 px, but not zero, and it is not along the diagonal: that is the "crooked" part.

Why it exists. Radial terms can only move a point along its radius. Glass that is off centre or tilted also pushes points sideways, and with k's alone that shift has nowhere to go but the residual.

What it assumes. - The lens is only slightly off: the shift is quadratic in the point, 0 on the axis and largest at the corners, and p1,p2 are small.

Where it breaks. - On a well-centred lens p1,p2 have nothing to describe, and fitting them only hands the noise two more knobs. Calibration can switch them off (CALIB_ZERO_TANGENT_DIST, What calibrateCamera Actually Does). - The shift is computed from the ideal point, which is unknown when you go back, so undoing it needs the guessing loop of Section 4 too. - The coefficients in the wrong slots: the NOTE below.

NOTE, the classic bug: OpenCV stores the coefficients as distCoeffs in the order (k₁, k₂, p₁, p₂, k₃), with k₃ last, after the tangential pair [OPEN24]. It is a vector, not a matrix. Our lens is (−0.2, 0, 0, 0, 0). Write (k₁, k₂, k₃, p₁, p₂) by habit and a radial lens turns into a crooked one.

TrivediInformational[Page 1]
S. TrivediOct 2026

4. Undoing It Takes a Few Guesses

-0.22
Camera: focal length and principal point
500320240
Undoing the lens, one guess at a time. Zoomed in on the top-left corner of the picture. The lens moved the corner pixel to the rust dot; each blue guess divides by the bend at the previous guess, and the guesses close in on the true corner. The small plot tracks the distance left after each guess. Open the camera panel to change the focal length and the principal point.

This is the top-left corner of the picture, zoomed in. The rust dot is where the lens put the corner, (40.96, 30.72). The hollow ring is where it belongs, (0, 0). The blue dots are guesses, and the small plot is the distance left after each one.

  • The first guess lands 13.591 px from the true corner; the second, 3.884 px.
  • Each guess sits on the same diagonal: the lens only ever moved the point along its radius.
  • The gaps shrink by nearly the same factor every time, which is why the curve in the plot drops like a stone.

Why guess at all? Going forward is one formula: give me 𝐱u, and Section 1 gives you 𝐱d. Going back has no formula, because L depends on the radius of 𝐱u, the very thing we are trying to find. Even with k1 alone, undoing it means solving a cubic in r. So guess, correct, repeat:

𝐱u←𝐱dL(‖𝐱u‖),start from 𝐱u=𝐱d(6)
  • 𝐱d: the seen point in normalised units, fixed throughout.
  • ‖𝐱u‖: the radius of the current guess. Each round recomputes L there and divides again.

The first round uses the radius of 𝐱d itself, which is a little too small, so it gives L=0.90267 and the guess (−0.61825,−0.46369): back through K, 13.591 px short of the corner. Then:

  • after 1–8 guesses the error is 13.591, 3.884, 1.132, 0.331, 0.097, 0.029, 0.008 and 0.002 px;
  • five guesses get under 0.1 px.

Why does the error fall by the same factor each time? Write one round as a function of the radius, g(r)=rd/L(r). Near the answer, a guess that is off by ε comes back off by |g′|ε:

|g′(r)|=2|k1|r2L(r)=2(0.2)(0.64)0.872(7)
  • g′(r): the slope of one round at the answer r=0.8, no unit.

That is 0.2936: each guess divides the error by about 3.41. A stronger lens has a larger slope and needs more rounds; press a stronger lens and watch the plot flatten out.

Why it exists. Every use of a calibrated camera goes from pixels back to rays, and the lens formula only runs forward. This loop is how a seen pixel gets its pinhole point back.

What it assumes. - The seen point is a good first guess: the bend is small. - Each round shrinks the error: |g′|<1 at the answer.

Where it breaks. - In general g′=−rL′(r)/L(r). For a barrel lens L′<0, so g′ is positive and reaches 1 exactly where L+rL′=0: the slope drd/dr=0 of Section 2. The loop converges inside the fold, slower the nearer it gets, and not beyond it. - A fixed count suits a mild lens only. Five guesses leave 0.097 px for ours, 3.699 px for a stronger lens (k1=−0.35, slope 0.5773) and 8.525 px at k1=−0.4. Set the stopping rule yourself with undistortPointsIter [OPEN24]. Try this: press a stronger lens and set the guesses to five.

OpenCV's undistortPoints runs this loop: the same division, starting from the seen point, five rounds by default, with undistortPointsIter when you want to set the stopping rule yourself [OPEN24]. One more thing it does that surprises people: without a P argument it returns 𝐱^, not pixels. You get (−0.64, −0.48) back for our corner, the normalised point of read 1, K−1𝐱 at work. Pass the camera matrix as P and you get the pixel (0, 0) instead.

Try this: press play. Each guess lands about 3.4 times closer.

5. Undistorting a Whole Picture

0
Camera: focal length and principal point
500320240
Camera 1: move it and turn it
0.00.00.00.00.0
Straight lines tell you the lens. Camera 1 looks at a doorframe and a long board, all straight edges. Through the lens they bow; the rust curves are the same edges corrected with the guessed k₁. Play the guess and watch the bow flip from one side to the other: the edges come out straight at one value only. Drag the background to turn the scene. Open the camera panels to move the camera, or change the focal length and the principal point.

Camera 1 looks at a doorframe and a long board: eight edges, all straight in the world. Through our lens they bow (blue). The rust curves are the same edges corrected with the guessed k1.

  • At a guess of 0 nothing is corrected, and the rust lies on the blue.
  • As the guess moves toward −0.2 the bow flattens; past it, the edges bow the other way.
  • The straightness error reaches 0 px at exactly −0.2, the true lens.

That is the method the book calls "a simple and more general approach": straight lines in the scene must image as straight lines, so choose the coefficients (and the centre) that make them straight again. No calibration pattern, only lines you know are straight (H&Z Ch. 7 Computation of the Camera Matrix P, §7.4, p. 191). Photogrammetry calls it plumb-line correction (H&Z Ch. 7 Computation of the Camera Matrix P, §7.5.1, p. 193). The other way the book gives is to fold the κi into the same refinement that fits P, using the reprojection error of read 4 (H&Z Ch. 7 Computation of the Camera Matrix P, p. 191). That is what the next read does.

Once you know the lens, how do you straighten a whole photograph? Not by pushing each source pixel forward; that leaves holes. You run the map the other way, per output pixel:

  • for each pixel of the straight output picture, take its normalised point, bend it with the lens formula of Section 1 (forward, so no guessing), and look up the colour at that spot in the original;
  • OpenCV builds that lookup table once with initUndistortRectifyMap and applies it with remap; undistort does both in one call [OPEN24].

And then a choice appears. Switch the first figure to after undistortion. The rust outline is where the border of the original photo lands once it is straightened. For our barrel lens it lands outside the frame all the way round, furthest out at the corners and least at the middle of each edge, so the straightened picture no longer fills a rectangle:

  • keep the old K and the 640 × 480 frame, and every output pixel has a source, but the corners of the original fall off the edge;
  • shrink the picture to keep every source pixel, and the gaps between the rust outline and the rectangle have nothing to show: black borders.

OpenCV leaves that choice to you as a new camera matrix, from getOptimalNewCameraMatrix(K, D, size, alpha): alpha 0 keeps only valid pixels (a crop), alpha 1 keeps every source pixel (black borders), and anything between is a blend. It also returns the rectangle of all-valid pixels [OPEN24].

NOTE: You do not always need to warp the picture. If all you want is points (corners, matches, a residual), undistort the points and leave the pixels alone. The book says detecting features in the original picture is often the better choice, because warping averages the noise and can alias fine detail; linking edges into straight lines is the exception, best done after warping (H&Z Ch. 7 *Computation of the Camera Matrix P*, pp. 192–193).

TrivediInformational[Page 2]
S. TrivediOct 2026

6. In OpenCV

HereOpenCVShapeSeries value
(k₁, k₂, p₁, p₂, k₃)distCoeffs1×5 (or 4, 8, 12, 14)(−0.2, 0, 0, 0, 0)
x_d → x_u (normalised)undistortPoints(src, K, D)N×1, 2 channels(40.96, 30.72) → (−0.64, −0.48)
… back in pixelsundistortPoints(src, K, D, P=K)N×1, 2 channels→ (0, 0)
control the loopundistortPointsIter(src, K, D, R, P, criteria)—five guesses: 0.097 px (also the default count)
a whole pictureundistort(img, K, D, newCameraMatrix), or initUndistortRectifyMap + remapimage—
the new KgetOptimalNewCameraMatrix(K, D, size, alpha)3×3, plus a rectanglealpha 0 crops, 1 keeps black borders
very wide lensesthe cv::fisheye functions (a different lens formula)——

Table. 1 This read in OpenCV 4.x [OPEN24]. With five coefficients OpenCV's full formula (which also allows k₄–k₆ and thin-prism terms) reduces to Sections 1 and 3.

7. What Came Next

So far each piece came from its own experiment: K and R from known 3-D points in read 4, k1 from lines we knew were straight here. Nobody calibrates a camera like that. In practice one function takes a pile of checkerboard photos and hands back K, the five coefficients and a pose for every photo. Who does all of it at once, from a printed checkerboard, and what is that function actually solving? That is What calibrateCamera Actually Does.

8. References

  1. [HART04] R. Hartley, A. Zisserman, Multiple View Geometry in Computer Vision, 2nd ed., Cambridge University Press, 2004. Chapters used in this read: Ch. 7 Computation of the Camera Matrix P.
  2. [OPEN24] OpenCV 4.x documentation, Camera Calibration and 3D Reconstruction (calib3d): the distortion formula, distCoeffs, undistortPoints, undistort, initUndistortRectifyMap, getOptimalNewCameraMatrix, cv::fisheye.
  3. [BROW66] D. C. Brown, "Decentering Distortion of Lenses", Photogrammetric Engineering 32(3), 1966.

9. Connections

What this page builds on, cites or links to, and what points back at it. A plain box is a “builds on”; “cited” and “linked in the text” are the lighter links. Hover a box for a preview.

POINTS AT THIS PAGEREADWhat calibrateCamera Actually DoesREADEpipolar Geometry — A Point Becomes aLineREADFrom Two Photos to a Point CloudREAD · LINKED IN THE TEXTThe Pinhole Camera — A Pixel Is a RayREAD · LINKED IN THE TEXTWhere Is the Camera? — R, C and tREAD · LINKED IN THE TEXTPoints at Infinity — Horizons, VanishingPoints and the FloorREAD · LINKED IN THE TEXTFinding the Camera From Known Points — Pby Least SquaresREAD · LINKED IN THE TEXTTriangulation — Two Rays That AlmostMeetREAD · LINKED IN THE TEXTFrom Matches to Motion — Four Cameras,One Right AnswerREAD · LINKED IN THE TEXTStereo Depth — Rectification and Z =fB/dThis pageTHIS PAGE POINTS ATREADThe Pinhole Camera — A Pixel Is a RayREADFinding the Camera From Known Points — Pby Least SquaresREFERENCE · CITEDMultiple View Geometry in ComputerVisionREFERENCE · CITEDCamera Calibration and 3D Reconstruction(calib3d)REFERENCE · CITEDDecentering Distortion of LensesREAD · LINKED IN THE TEXTWhat calibrateCamera Actually Does

Related

Go deeper

  1. Start with
    Read
  2. Read
  3. Read
  4. Read
  5. You are here
    Lens Distortion — Bending Straight Lines Back
    Read
  6. Then
    Read
  7. Read
  8. Read
  9. Read
  10. Read
  11. Read

Go broader

<- Older [Page 4] Newer ->