Finding the Camera From Known Points — P by Least Squares

Every read so far was handed the camera matrix P and asked where a point lands. Real life runs the other way. You have a target whose corners you measured with a ruler, you have a photo of it, and you want the camera: its focal length, which way it points and where it stood. That is the problem of this read, and the whole pipeline hangs on three steps:

• write two linear equations per known point and its pixel;
• solve them all at once for the twelve numbers of P;
• take P apart into K, R and the centre C̃.

A hiker is lost on a mountain. She can see three peaks she can also find on her map, and she sights the direction to each with a compass. The directions alone tell her where she stands: draw the three sight lines back from the peaks on the map and they meet at her feet. Surveyors call this resection, and it is exactly the word the book uses for finding a camera from known points [HART04]. Our compass is the pixel, our peaks are the corners of a target.

Brush up on:

• Where Is the Camera? — R, C and t: P = K[R | t], a 3 × 4 matrix that has 12 numbers but only 11 degrees of freedom, because its scale is free. Camera 2 of the series stands at C̃₂ = (1, 0, 0) and is turned 10° toward camera 1.
• Points at Infinity — Horizons, Vanishing Points and the Floor: a plane maps to the picture by a 3 × 3 homography H, which has eight degrees of freedom and needs four points. Section 6 comes back to it.

One tool from outside the series is needed, and it is the only new one: the singular vector with the smallest singular value is the best answer to Ap = 0. Section 2 says what that means and why it works.

1. Two Equations per Point

The target for this read is an L: two 4 × 4 grids of corners with 0.25 m spacing, one on the wall z = 4 m and one on the floor y = 1 m (y points down, so the floor is 1 m below the cameras). That is 32 corners, and camera 2 sees them between u = 189.5 and 329.8 px and v = 240 and 403.2 px. The figure starts with six of them and 0.5 px of noise on every pixel.

corner noise σ (px)

0.5

corners used

6

angle between the faces (°)

90

normalisedraw pixels

Camera: focal length and principal point

focal length f (px)

500

principal point cₓ (px)

320

principal point c_y (px)

240

Camera 2: move it and turn it

camera 2 sideways x (m)

1.0

camera 2 height (m, up is −)

0.0

camera 2 forward z (m)

0.0

camera 2 turn left/right (°)

10

camera 2 tilt up/down (°)

0.0

the worked rigall 32 cornersraw pixelsfold it flatone bad cornerno noiseReset

• The rust camera is the true camera 2. The grey frustum marked "found" is the camera rebuilt from the six black corners and their noisy pixels, and nothing else.
• In camera 2's picture, filled dots are the measured corners and rings are where the found camera puts them back. The short lines join the two, drawn 20 times longer so that you can see them.
• The found camera is 0.442 m from the true one, even though its rings sit almost on top of the dots. Keep that in mind until Section 4 ;)

Why six? Each known point gives two equations, and P has 11 unknowns, so 5½ points pin it down; we use six. Let's devise the equations. The pixel x_i = (u_i, v_i, 1) and the projected point P X_i are the same ray, so they are parallel and their cross product is zero. Writing P's three rows as P¹, P², P³ (each a 4-vector), two of the three components of that cross product are independent (Hartley & Zisserman, Ch. 7 Computation of the Camera Matrix P, eqs. 7.1–7.2, pp. 178–179):

• ${𝐗}_i=(X,Y,Z,1{)}^{\top }$ is the known corner in metres, with a 1 appended.
• $(u_i,v_i)$ is its measured pixel in px.
• Both rows are linear in P's twelve numbers. Stack them for n points and you get a 2n × 12 matrix A with $A𝐩=\mathbf{0}$, where $𝐩$ lists P row by row.

For our six points A is 12 × 12. Here they are with their true pixels in camera 2 (the figure adds the noise on top):

Table. 1 The six corners the figure starts with: three on the wall z = 4 m, three on the floor y = 1 m, and their exact pixels through camera 2's P₂.

Why it exists. A pixel is PX divided by its last number, and that division is not linear in P. The cross product removes it, so the camera comes out of a linear solve. Without these two rows per point there is no straight-line estimate at all.

What it assumes. - Each pixel belongs to its corner: every match is right. - The pinhole chain holds: straight lines stay straight (H&Z Ch. 7 Computation of the Camera Matrix P, §7.4, p. 189). - The points are not laid out degenerately: not all on one plane (section 6), not on a twisted cubic through the camera centre, and not on a plane plus one line through the centre (H&Z Ch. 7 Computation of the Camera Matrix P, pp. 179–180).

Where it breaks. - One wrong match: least squares spreads it over all twelve numbers. "one bad corner" moves one pixel by (25, −15) px, and the six-corner camera lands 1.433 m from the truth with f = 95.9 px; with 32 corners it is still 0.216 m off with f = 413.1 px. Throw such points out with RANSAC (H&Z Ch. 4 Estimation – 2D Projective Transformations, §4.7, pp. 116–122) before the DLT. - Close to a plane: a far camera over a scene with little depth is close to the planar case and gives a poor P (H&Z Ch. 7 Computation of the Camera Matrix P, p. 180). Section 6 shows it. - Try this: press "one bad corner", then slide "corners used" up to 32.

Try this: slide "corners used" up to 32 and watch the 64 × 12 matrix pull the grey camera onto the rust one.

2. Least Squares Without a Right-Hand Side

corner noise σ (px)

0

corners used

6

angle between the faces (°)

90

normalisedraw pixels

Camera: focal length and principal point

focal length f (px)

500

principal point cₓ (px)

320

principal point c_y (px)

240

Camera 2: move it and turn it

camera 2 sideways x (m)

1.0

camera 2 height (m, up is −)

0.0

camera 2 forward z (m)

0.0

camera 2 turn left/right (°)

10

camera 2 tilt up/down (°)

0.0

the worked rigall 32 cornersraw pixelsfold it flatone bad cornerno noiseReset

With no noise the rings land exactly on the dots, the reprojection error (dot to ring) falls to 8.8e−14 px, and the found camera is the true one: K comes back as 500 px, 500 px, (320, 240). With noise there is no p that makes Ap exactly zero, so we need "as close to zero as possible". But careful: p = 0 always gives Ap = 0, and it is useless. P is only defined up to scale anyway, so we fix the scale and ask for the best unit vector (H&Z Appendix 5 Least-squares Minimization, §A5.3, alg. A5.4, pp. 592–593):

• $A=UD{V}^{\top }$ is the singular value decomposition (H&Z Appendix 4 Matrix Properties and Decompositions, A4.4, pp. 585–587): D holds the singular values σ₁ ≥ … ≥ σ₁₂ ≥ 0, and the columns of V are unit directions.
• Why the last column: write any unit p as a mix of V's columns with weights y₁ … y₁₂, where y₁² + … + y₁₂² = 1. Then $\Vert A𝐩{\Vert }^2={\sigma }_1^2{y}_1^2+\dots +{\sigma }_{12}^2{y}_{12}^2$, smallest when all the weight sits on the smallest singular value: p is the last column and $\Vert A𝐩\Vert ={\sigma }_{12}$.
• On exact data that smallest singular value is 0 and the reprojection error is 8.8e−14 px for six points and 7.8e−14 px for all 32: the straight-line estimate is exact when the data are.

This way of solving is the DLT, the direct linear transformation: direct because there is no starting guess and no iteration, one decomposition and done.

Why it exists. With noise, Ap = 0 has no exact answer, and p = 0 is always a useless one. Fixing ‖p‖ = 1 uses up P's free scale and turns "as close to zero as possible" into a question with one answer, from one decomposition.

What it assumes. - A has rank 11 on exact data: exactly one direction of p costs nothing (H&Z Ch. 7 Computation of the Camera Matrix P, p. 179). - ‖Ap‖ is a fair thing to make small. It is the algebraic error, which the book calls "not geometrically or statistically meaningful" (H&Z Ch. 4 Estimation – 2D Projective Transformations, §4.2.1, p. 94): it is not a distance in pixels.

Where it breaks. - Rank below 11: several singular values sit at 0, and the last column of V is just one of many equally good answers. The flat target of section 6 has four. - The smallest ‖Ap‖ is not the smallest pixel error. The book polishes the DLT's P on the pixel distance itself (H&Z Ch. 7 Computation of the Camera Matrix P, alg. 7.1, p. 181), which is section 4. On its own target the polish changed the residual only from 0.365 to 0.364 px (H&Z Ch. 7 Computation of the Camera Matrix P, table 7.1, p. 182).

The same SVD does one more job that read 9 needs, so I will say it once here. Set the smallest singular value in D to zero and multiply back, $U{D}^{\prime }{V}^{\top }$: that is the nearest matrix of rank one lower to A, nearest counting every entry equally (H&Z Ch. 4 Estimation – 2D Projective Transformations, p. 108). When a matrix ought to have rank 2 and noise gave it rank 3, this is how you push it back.

Try this: from "no noise", nudge σ to 0.05 px and see the grey camera peel away from the rust one.

3. Why Normalise

corner noise σ (px)

0.5

corners used

6

angle between the faces (°)

90

normalisedraw pixels

Camera: focal length and principal point

focal length f (px)

500

principal point cₓ (px)

320

principal point c_y (px)

240

Camera 2: move it and turn it

camera 2 sideways x (m)

1.0

camera 2 height (m, up is −)

0.0

camera 2 forward z (m)

0.0

camera 2 turn left/right (°)

10

camera 2 tilt up/down (°)

0.0

the worked rigall 32 cornersraw pixelsfold it flatone bad cornerno noiseReset

This is the same figure with the toggle on "raw pixels". Same corners, same noise, same equations, and the six-point camera gets worse: f drops to 428.8 px and the centre lands at (0.733, 0.176, 0.499) m.

The reason sits in A itself. Pixels are in the hundreds and metres are near 1, so one row of A holds entries like 1 and 4 next to products of pixels and coordinates that pass a thousand. The SVD treats a change of 100 in a big entry and in a small one alike, although for the points they mean very different things (H&Z Ch. 4 Estimation – 2D Projective Transformations, p. 108). The fix is to shift and scale both sides first (H&Z Ch. 4 Estimation – 2D Projective Transformations, §4.4.4, pp. 107–109; H&Z Ch. 4 Estimation – 2D Projective Transformations, alg. 4.2, p. 109; H&Z Ch. 7 Computation of the Camera Matrix P, for P, p. 180):

• T (3 × 3) moves the pixels' centroid to the origin and scales them so that their RMS distance from it is √2.
• U (4 × 4) does the same to the world points with RMS distance √3. (This U is a new one, not the SVD's.)
• P_n is the camera found between the normalised points; the last line undoes both maps.

For our six corners, T scales by 0.01874 per px and shifts by (−5.01724, −6.20574); U scales by 2.7713 per metre about the centroid (−0.0417, 0.6667, 3.7083) m. What that does to A is best read through its conditioning, the ratio σ₁/σ₁₁ of the largest singular value to the eleventh (the last one that should be non-zero):

• six points: 463 745.5 raw against 12.2 normalised;
• 32 points: 231 767.1 raw against 7.21 normalised.

One draw proves nothing, so here are 200: over 200 draws with 0.5 px noise on all 32 corners, the median focal error |f − 500| is 7.7 px normalised against 15.5 px raw, and the median centre error is 0.070 m against 0.132 m. Normalising wins on both counts for the price of two small matrices, and the book is blunt about it: "Data normalization is an essential step in the DLT algorithm. It must not be considered optional" (H&Z Ch. 4 Estimation – 2D Projective Transformations, p. 108).

Why it exists. Without it the DLT's answer depends on where the origin sits and what the units are; with it the answer is, in practice, the same whatever coordinates the corners came in (H&Z Ch. 4 Estimation – 2D Projective Transformations, §4.4.4, p. 107). It also keeps the conditioning low, and a high conditioning blows up the noise (H&Z Ch. 4 Estimation – 2D Projective Transformations, p. 108).

What it assumes. - The world points are compact around their centroid, with little spread in depth from the camera; that is when the √3 scaling suits them (H&Z Ch. 7 Computation of the Camera Matrix P, p. 180). - There is noise to care about: on exact data the answer is the same with or without normalising (H&Z Ch. 4 Estimation – 2D Projective Transformations, p. 108).

Where it breaks. - Points over a great range of depths, near points together with very far ones or vanishing points, as in an oblique view of terrain: shifting them to their centroid is "not possible or reasonable", and the book's alternative "has not been thoroughly tested" (H&Z Ch. 7 Computation of the Camera Matrix P, p. 180).

Try this: with "raw pixels" on, compare the conditioning tile with the normalised number in its note.

4. The Residual Trap

The figure's readout gives the residual, and it is tempting to read it as a grade: small residual, good camera. It is not.

corner noise σ (px)

0.5

Said in plain words before any formula: with few points, the estimate has enough freedom to bend toward the noise. Eleven numbers fitted to twelve measurements can follow almost every wobble of the dots, so the rings sit on the dots, the residual is tiny, and the camera is wrong. Only extra points, which the camera cannot all follow at once, reveal how noisy the pixels really were.

The residual itself is the reprojection error: the pixel distance between each measured corner and where P puts it back (H&Z Ch. 7 Computation of the Camera Matrix P, eq. 7.4, p. 181), and what the book expects it to be for the best possible fit of 11 numbers to n points with pixel noise σ (H&Z Ch. 7 Computation of the Camera Matrix P, p. 188; H&Z Ch. 5 Algorithm Evaluation and Error Analysis, per coordinate, result 5.2, p. 136):

• $d({𝐱}_i,P{𝐗}_i)$ is the distance in px between the measured pixel and the reprojected one.
• The 2n counts coordinates, not points: this is the book's per-coordinate residual.
• σ is the noise on each pixel coordinate, in px; 11 is the number of unknowns in P.

At σ = 0.5 px the expected residual is 0.144, 0.368, 0.455 and 0.486 px at n = 6, 12, 32 and 100. It only creeps up toward σ = 0.5 px as n grows; at six points it is 0.144 px. The measured residual in the figure above follows that curve. Now the first figure's one draw:

• six points: residual 0.246 px per coordinate (0.348 px per point), while α = 445.8 / 451.2 px against the true 500 px and the centre lands at (0.774, 0.148, 0.35) m, 0.442 m from the truth;
• 32 points: residual 0.530 px per coordinate (0.749 px per point), f = 502.6 px, centre 0.051 m off.

So the residual grew from 0.246 to 0.530 px while the camera moved from 0.442 m off to 0.051 m off. A residual is only worth reading next to its n.

Why it exists. The residual is the only grade you get without knowing the true camera. The expected residual says what the best possible fit should show at this n and σ, so a measured residual can be read against it.

What it assumes. - The noise is Gaussian and independent, with the same σ on every pixel coordinate, and all of it is in the pixels: the world points are exact (H&Z Ch. 5 Algorithm Evaluation and Error Analysis, result 5.2, p. 136; H&Z Ch. 7 Computation of the Camera Matrix P, p. 188). - The fit is the best one, the maximum-likelihood fit: the formula is for that estimate, and the DLT only comes close to it. - The camera model is right.

Where it breaks. - A wrong model shows up as noise. The book's target gave 0.365 px over 197 points, which means σ = 0.37 px for corners it expected to be "far better than 1/10 of a pixel" (H&Z Ch. 7 Computation of the Camera Matrix P, p. 182); the reason was the radial distortion it ignored (H&Z Ch. 7 Computation of the Camera Matrix P, p. 188). That is Lens Distortion — Bending Straight Lines Back. - A bad match: "one bad corner" lifts the six-corner residual to 1.996 px per coordinate against the expected 0.144 px. The residual does flag it, but only if you compare it with the expected value. - Too few points: at 5½ points 1 − 11/2n is 0, and the fit is exact whatever the noise (H&Z Ch. 7 Computation of the Camera Matrix P, p. 179). The book's rule of thumb is five times more measurements than unknowns: at least 28 points (H&Z Ch. 7 Computation of the Camera Matrix P, p. 182).

NOTE: two conventions for the same number. The book divides by the number of coordinates, 2n. OpenCV's calibrateCamera returns the RMS per point: its reference calls it "the overall RMS re-projection error" [OPEN24], and its 4.x source divides the sum of squared point distances by the number of points. That is √2 larger, so the book's 0.455 px at 32 points is 0.643 px in OpenCV's units. Know which one you are comparing.

The straight-line estimate is rarely the last word. The book polishes it by minimising the reprojection error of eq. 4 directly over P, starting from the DLT (H&Z Ch. 7 Computation of the Camera Matrix P, its Gold Standard algorithm, alg. 7.1, p. 181), with Levenberg–Marquardt, an iterative least-squares method that takes small careful steps when it is far from the answer and fast ones when it is close.

Try this: in the figure above, drag σ and watch the gap at six points stay wide.

5. Taking P Apart

We have P. We want K, R and the centre. Step through the figure: three turns, then a sign flip.

◀1 / 5 · M = K R▶ Next▶

Camera: focal length and principal point

focal length f (px)

500

principal point cₓ (px)

320

principal point c_y (px)

240

Camera 2: move it and turn it

camera 2 sideways x (m)

1.0

camera 2 height (m, up is −)

0.0

camera 2 forward z (m)

0.0

camera 2 turn left/right (°)

10

camera 2 tilt up/down (°)

0.0

the worked rigcamera 2 tilted downReset

The left 3 × 3 block of P = K[R | −RC̃] is M = KR: an upper-triangular matrix times a rotation. Finding that pair from M is an RQ decomposition, and the book does it with three Givens turns (H&Z Appendix 4 Matrix Properties and Decompositions, A4.1.1, p. 579), each a rotation about one axis chosen to zero one entry below the diagonal: m₃₂, then m₃₁, then m₂₁. The centre is the point P sends to zero, as in Where Is the Camera? — R, C and t (H&Z Ch. 6 Camera Models, §6.2.4, pp. 163–164):

• ${𝐩}_4$ is P's last column; $Q_x,Q_y,Q_z$ are the turns about x, y and z.
• The pair is not unique: $KR=(KD)(DR)$ for any D = diag(±1, ±1, ±1). The book removes the choice by "requiring that K have positive diagonal entries" (H&Z Ch. 6 Camera Models, p. 163). That is the sign pass.

On the true P₂ the turns leave the raw diagonal (−500, −500, 1), so D = diag(−1, −1, 1), and the sign pass gives

exactly. On the six-point DLT's P from exact pixels the raw diagonal is (−80.7821, −80.7821, 0.1616): nothing like 500, because P's scale is arbitrary. The last step divides K by its corner entry, and the same 500, 500, (320, 240) come out.

Why it exists. P alone tells where points land; the focal length, the turn and the position are what you report and reuse. RQ splits M into K and R, and the point P sends to zero gives C̃ (H&Z Ch. 6 Camera Models, §6.2.4, p. 163).

What it assumes. - M is invertible: a finite camera. If M is singular the centre is at infinity, C̃ = −M⁻¹p₄ has nothing to invert, and the centre is the null vector (d, 0) of P instead (H&Z Ch. 6 Camera Models, pp. 158–159).

Where it breaks. - A camera far away with a long lens is close to that case: tracking back while zooming in ends in an affine camera, whose centre is at infinity (H&Z Ch. 6 Camera Models, §6.3.1, pp. 166–167). - The noise lands in K. RQ returns whatever upper-triangular K fits, so the six noisy corners give α = 445.8 / 451.2 px and a skew of 0.4. If you know the skew is 0 and the pixels are square, impose that while estimating (H&Z Ch. 7 Computation of the Camera Matrix P, §7.3, pp. 184–188); in the book's example it moved the other numbers and raised the residual from 0.365 to 0.601 px (H&Z Ch. 7 Computation of the Camera Matrix P, tables 7.1 and 7.2, pp. 182 and 188). - det M < 0 or det R = −1: see the note below.

NOTE: as in Where Is the Camera? — R, C and t, if det M < 0, negate P first; the sign pass then leaves det R = +1. If you skip it and det R comes out −1, you have a reflection, not a rotation: negate P and decompose again.

Try this: step to "fix the signs" and check that only two columns of K and two rows of R changed.

6. A Flat Target Cannot Do It

corner noise σ (px)

0.5

corners used

32

angle between the faces (°)

0

normalisedraw pixels

Camera: focal length and principal point

focal length f (px)

500

principal point cₓ (px)

320

principal point c_y (px)

240

Camera 2: move it and turn it

camera 2 sideways x (m)

1.0

camera 2 height (m, up is −)

0.0

camera 2 forward z (m)

0.0

camera 2 turn left/right (°)

10

camera 2 tilt up/down (°)

0.0

the worked rigall 32 cornersraw pixelsfold it flatone bad cornerno noiseReset

Here the floor has swung down about its hinge (the line y = 1 m on the wall's plane) into that plane: all 32 corners now sit on z = 4 m, and every tile says the same thing, no single camera fits.

Why? If every point satisfies one plane equation π·X = 0, then any P whose three rows are multiples of π sends every point to zero, and Ap = 0 for all of them, whatever the pixels. That is three directions of p that cost nothing, on top of the true camera: many cameras fit equally well (H&Z Ch. 7 Computation of the Camera Matrix P, pp. 179–180). Check it on the wall alone, 16 coplanar corners: the last four singular values of A are all 0, a 4-dimensional null space.

But a plane on its own is not useless. It still gives a homography, by the same recipe with a 3 × 3 matrix: two equations per point, 9 numbers in h, 8 degrees of freedom, so four points (H&Z Ch. 4 Estimation – 2D Projective Transformations, alg. 4.1, p. 91; H&Z Ch. 4 Estimation – 2D Projective Transformations, normalised, alg. 4.2, p. 109):

• ${𝐬}_i$ is the corner's position on the plane, in metres along the plane's two directions.
• h lists H's nine entries; it is the last column of V again, after the same T normalisation on both sides.

Run on camera 1's picture of the 16 wall corners, the plane DLT returns

exactly (scaled so that the corner entry is 4), the same matrix Points at Infinity — Horizons, Vanishing Points and the Floor wrote down by hand. One flat board gives a homography, not a camera. What calibrateCamera Actually Does takes several photos of a flat board, one homography each, and gets K back from them, which is why a checkerboard is enough for calibration.

Why it exists. A flat target cannot give P, but it still fixes the map from its plane to the picture. That map is what Points at Infinity — Horizons, Vanishing Points and the Floor used for the floor, and what What calibrateCamera Actually Does builds K from.

What it assumes. - The points really are on one plane: H maps that plane and nothing else (H&Z Ch. 8 More Single View Geometry, §8.1.1, p. 196). - The camera centre is not on the plane; the book assumes it (H&Z Ch. 8 More Single View Geometry, p. 196), and otherwise the whole plane is seen edge-on as one line.

Where it breaks. - Three of four points on one line: a rank-1 matrix also solves the equations, and either there is no proper H or a whole family of them; more points on that same line do not help (H&Z Ch. 4 Estimation – 2D Projective Transformations, §4.1.3, pp. 91–92). Spread the corners over the board. - A bad match pulls H just as it pulls P in section 1; this is what the RANSAC option of findHomography is for (H&Z Ch. 4 Estimation – 2D Projective Transformations, §4.7, pp. 116–122).

Try this: set the fold to 30° and then 10°; the camera still comes out, but watch the conditioning tile climb and the found camera drift away while the residual hardly moves.

7. In OpenCV

Table. 2 This read in OpenCV 4.x [OPEN24].

8. What Came Next

Every residual in this read assumed a lens that keeps straight lines straight. Real lenses do not, and their bend is largest exactly where calibration corners like to sit: near the edges of the picture. What does a lens do to a pixel, and how do we bend it back? That is Lens Distortion — Bending Straight Lines Back.

9. References

1. [HART04] R. Hartley, A. Zisserman, Multiple View Geometry in Computer Vision, 2nd ed., Cambridge University Press, 2004. Chapters used in this read: Ch. 4 Estimation – 2D Projective Transformations; Ch. 5 Algorithm Evaluation and Error Analysis; Ch. 6 Camera Models; Ch. 7 Computation of the Camera Matrix P; Ch. 8 More Single View Geometry; Appendix 4 Matrix Properties and Decompositions; Appendix 5 Least-squares Minimization.
2. [OPEN24] OpenCV, Camera Calibration and 3D Reconstruction (calib3d), OpenCV 4.x documentation.