Where Is the Camera? — R, C and t

The camera of the last read sat at the world's origin and looked straight down z. Real cameras do not: they are bolted to poles, cars and tripods, turned this way and that. So how do we tell K where the camera is, and which way it is looking?

The answer is one extra step at the front of the chain: before dividing by depth, move the world point into the camera's own axes. Everything in this read lives in that one box.

Brush up on The Pinhole Camera — A Pixel Is a Ray first, and bring two things from it:

• x = K X for a point that is already in the camera's own axes, with K = [[500, 0, 320], [0, 500, 240], [0, 0, 1]].
• Divide by the last number. A camera hands you a 3-vector; the pixel is its first two entries divided by the third.

For the whole series, the world frame is camera 1's frame: x right, y down, z forward, camera 1 at the origin. The camera of this read is camera 2: its centre is at (1, 0, 0), one metre to the right of camera 1, and it is turned 10° toward camera 1. The point we follow is the same X = (0.3, −0.2, 4) as before.

1. Where It Stands and Where It Looks

Ask someone where the front door is. "Three metres north of the gate" is the door in the world's terms: anyone standing anywhere can use it. "Two steps ahead and a bit to my left" is the door in your terms: it only makes sense from where you stand and the way you face. A camera needs the second kind of answer, because it can only divide by "ahead".

Keep that pair in mind; it comes back in the next section.

Camera: focal length and principal point

focal length f (px)

500

principal point cₓ (px)

320

principal point c_y (px)

240

Camera 1: move it and turn it

camera 1 sideways x (m)

0.0

camera 1 height (m, up is −)

0.0

camera 1 forward z (m)

0.0

camera 1 turn left/right (°)

0.0

camera 1 tilt up/down (°)

0.0

Camera 2: move it and turn it

camera 2 sideways x (m)

1.0

camera 2 height (m, up is −)

0.0

camera 2 forward z (m)

0.0

camera 2 turn left/right (°)

10

camera 2 tilt up/down (°)

0.0

the worked rigcamera 1's poseturn without movingface awayReset

What the figure shows:

• Two things place a camera. Its centre C̃, a point in the world (the rust dot C₂ at (1, 0, 0)), and its rotation R, which says which way its axes point (the short rust arrows x₂, y₂, z₂).
• Each row of R is one camera axis, written in world coordinates. Row 1 is camera 2's right, row 2 its down, row 3 its forward. Read the matrix tile row by row and you are reading the arrows.
• The arc at C₂ is the 10° between the world's z direction (the faint dashed line from C₂) and camera 2's forward axis. Turn camera 2 with the slider and the matrix changes; drag C₂ and the matrix stays as it is.

Let's write the "your terms" answer. Take the arrow from the camera's centre to the point, X − C̃, and measure it along each of the camera's three axes. Measuring along an axis is a dot product with that axis, and three dot products with the three rows of R are one matrix multiply (Hartley & Zisserman, Ch. 6 Camera Models, §6.1, p. 155; eq. 6.6 on p. 156 is the same with a 1 added):

• $𝐗$: the point in world coordinates, in metres.
• $\widetilde{𝐂}$: the camera's centre in world coordinates, in metres. The tilde says "a plain 3-vector, no 1 added".
• $R$: a 3 × 3 rotation, rows = the camera's right, down and forward axes in world coordinates. A rotation keeps lengths and has det R = 1.
• ${𝐗}_{\mathrm{c}\mathrm{a}\mathrm{m}}$: the same point in the camera's own axes, in metres: how far right, how far down, how far ahead.

The numbers for camera 2:

• Its forward axis, row 3, is (−0.17365, 0, 0.98481): mostly along the world's z, with a small step toward −x, which is toward camera 1. That small step is the 10° turn.
• Its right axis, row 1, is (0.98481, 0, 0.17365), and its down axis, row 2, is the world's own y, (0, 1, 0): a turn about the vertical axis leaves "down" alone.
• det R₂ = 1, and C̃₂ = (1, 0, 0).
• R₂ ((0.3, −0.2, 4) − (1, 0, 0)) = (0.00523, −0.2, 4.06078). Camera 2 sees X almost dead ahead (0.00523 m to the right), 0.2 m up, and 4.06078 m in front.

That first number is the 10° turn at work: camera 1 had X 0.3 m to its right, and camera 2, standing a metre further right and turned in, has it almost on its axis.

Why it exists. The divide of the last read only works on "ahead", in the camera's own axes. This is how a point in the world's terms becomes a point in the camera's terms, so nothing in this read, or in any read with two cameras, works without it.

What it assumes. - R is a true rotation: rows of length 1, at right angles, det R = 1. Then it keeps lengths, and X − C̃ is measured, not stretched. - Both frames have the same handedness. Here the world and the camera are both x right, y down, z forward. - X and C̃ are in the same units. OpenCV's tvec comes back in whatever units your points were in (Table 2) [OPEN24].

Where it breaks. - Frames of different handedness. The book's camera has y up; measure the image's y downwards with that camera and the 3 × 4 matrix still exists, but R then describes the camera looking along −Z, and every point in front gets a negative depth (H&Z Ch. 6 Camera Models, pp. 164–165). The NOTE in Section 5 has the fix.

Try this: press "camera 1's pose". C₂ jumps to the origin, R becomes the identity, t becomes (0, 0, 0), and the arrows x₂, y₂, z₂ line up with x, y, z.

2. t Is Not Where the Camera Is

Here is the same figure again, with camera 2 turned 40° without moving it:

Camera: focal length and principal point

focal length f (px)

500

principal point cₓ (px)

320

principal point c_y (px)

240

Camera 1: move it and turn it

camera 1 sideways x (m)

0

camera 1 height (m, up is −)

0

camera 1 forward z (m)

0

camera 1 turn left/right (°)

0

camera 1 tilt up/down (°)

0

Camera 2: move it and turn it

camera 2 sideways x (m)

1

camera 2 height (m, up is −)

0

camera 2 forward z (m)

0

camera 2 turn left/right (°)

40

camera 2 tilt up/down (°)

0

the worked rigcamera 1's poseturn without movingface awayReset

• C₂ still stands at (1, 0, 0), and C̃ in the t tile's note has not changed.
• The t tile has changed anyway: it was (−0.98481, 0, 0.17365) at 10°, and now it is a different vector of the same length, 1 m.

That is the whole point of this section. Multiply out eq. 1 and the centre hides inside a new vector (H&Z Ch. 6 Camera Models, the text around eq. 6.8, p. 156):

• $𝐭$: a 3-vector in metres, written in the camera's axes.
• Put the world origin, X = (0, 0, 0), into the first line: X_cam = t. So t is the world origin as the camera sees it, the "two steps ahead and a bit to my left" answer for the point (0, 0, 0). It is not where the camera is; C̃ is.

For camera 2:

• t₂ = −R₂ (1, 0, 0) = (−0.98481, 0, 0.17365).
• In the door analogy: camera 1's centre is 0.98481 m to camera 2's left (x is negative) and 0.17365 m ahead of it (z is positive). The world's version of the same fact is C̃₂ = (1, 0, 0): "one metre to the right of the origin".
• |t₂| = |C̃₂| = 1. A rotation keeps lengths, so t always has the length of C̃; only its direction depends on which way the camera faces. That is why the turn-without-moving preset swings t round while C̃ stays put.

In the figure, t is the dashed rust arrow from C₂ back to the origin. It is the same arrow in the world either way; what R changes is the three numbers that describe it in camera 2's axes.

Why it exists. Code does not want C̃; it wants one shift to add after the rotation. t is that shift, and it is what OpenCV's tvec holds.

What it assumes. - t belongs to this R. It is written in the camera's axes, so it means nothing apart from the R it was made with.

Where it breaks. - Reading t as the camera's position. For camera 2 that puts it at (−0.98481, 0, 0.17365), 1.99 m from where it stands. The second NOTE below has the other half of this trap: tools that store camera to world.

NOTE: OpenCV (outside the book) stores R as an rvec: the rotation axis times the angle in radians. Ours is (0, 0.174533, 0), which is 10° about y, and Rodrigues turns it into the 3 × 3 R₂ above. Its tvec is our t, world to camera [OPEN24].

NOTE: many SLAM and graphics tools store the inverse, camera to world, which is Rᵀ and C̃. Ours is Rᵀ = [[0.98481, 0, −0.17365], [0, 1, 0], [0.17365, 0, 0.98481]] and C̃ = (1, 0, 0). Both are called "the pose". Check which one a library hands you before you use it: the wrong one puts the camera at t and turns it the wrong way.

3. One Matrix From World to Pixel

Now chain it all: world axes → camera axes → divide by depth → K. Step through it on camera 2:

step◀1 / 4 · the world point▶ Next▶X (sideways, height) at z = 4

(0.3, -0.2)

Camera: focal length and principal point

focal length f (px)

500

principal point cₓ (px)

320

principal point c_y (px)

240

Camera 1: move it and turn it

camera 1 sideways x (m)

0.0

camera 1 height (m, up is −)

0.0

camera 1 forward z (m)

0.0

camera 1 turn left/right (°)

0.0

camera 1 tilt up/down (°)

0.0

Camera 2: move it and turn it

camera 2 sideways x (m)

1.0

camera 2 height (m, up is −)

0.0

camera 2 forward z (m)

0.0

camera 2 turn left/right (°)

10

camera 2 tilt up/down (°)

0.0

the worked rigReset

• Step 1 is X in world metres, measured along x, y and z from camera 1's centre.
• Step 2 measures the same point from C₂ along camera 2's axes: forward 4.06078 m, down −0.2 m, right 0.00523 m. Nothing has been divided yet; this is eq. 2.
• Step 3 slides X down its ray onto the plane 1 m in front of C₂, where it sits at (x/z, y/z) = (0.001287, −0.049252). That is the division by depth, and it is the only step that is not a matrix multiply.
• Step 4 is K from the last read, turning metres on that plane into pixels.

Steps 2 and 4 are matrix multiplies and step 3 is "divide by the last number". K's last row is (0, 0, 1), so K leaves that last number alone, and the division can wait until the very end. So the whole camera is one 3 × 4 matrix, applied to the world point with a 1 added (H&Z Ch. 6 Camera Models, eqs. 6.7, 6.8 and 6.11, pp. 156–157):

• $[R\mid 𝐭]$: R with t glued on as a fourth column, 3 × 4. This is what libraries call the extrinsic matrix: just [R | t], nothing more.
• $KR[I\mid -\widetilde{𝐂}]$: the same matrix, written with the centre instead of t. The first spelling is what code multiplies; the second is what the camera is.
• $P$: the camera matrix, 3 × 4. P times (X, 1) gives a 3-vector; divide by its last number for the pixel.

Count its numbers. K has 5 (two focal lengths, the skew and the principal point), R has 3 (three angles) and C̃ has 3 (three coordinates): 5 + 3 + 3 = 11. P has 12 entries, but any multiple of P is the same camera, which takes 1 away: 12 − 1 = 11 again (H&Z Ch. 6 Camera Models, p. 157).

For camera 2, multiplying out K[R₂ | t₂] gives

• P₂ = [[436.836, 0, 401.963, −436.836], [−41.676, 500, 236.354, 41.676], [−0.174, 0, 0.985, 0.174]];
• the chain on X: (0.3, −0.2, 4) → R₂X + t₂ = (0.00523, −0.2, 4.06078) → divide by depth → (0.001287, −0.049252) → K → x₂ = (320.644, 215.374).

Check the last step by hand: u = 500 × 0.001287 + 320 ≈ 320.644 and v = 500 × (−0.049252) + 240 = 215.374. Camera 1 saw the same point at (357.5, 215); camera 2, turned in toward it, sees it almost on its own principal point column.

Why it exists. It packs the pose and K into one 3 × 4 matrix, so a world point becomes a pixel with one multiply and one divide. Every later read works with P: fitting it to points (read 4), and two of them at once (reads 7 to 11).

What it assumes. - Everything before the divide is linear: no lens distortion (H&Z Ch. 7 Computation of the Camera Matrix P, the opening of the chapter, p. 178). - A finite camera: M = K R is invertible, which it is whenever K and R are (H&Z Ch. 6 Camera Models, p. 157). - One K for every picture. Zooming by a factor k multiplies K by diag(k, k, 1) (H&Z Ch. 6 Camera Models, §6.3.1, p. 167), so a zoomed shot needs a new P even if the camera did not move.

Where it breaks. - A real lens. Straight lines in the world come out curved, mostly near the edges of the picture (H&Z Ch. 7 Computation of the Camera Matrix P, fig. 7.4, p. 190), and no 3 × 4 matrix can bend a line. Read 5 adds the bend as its own step. - A point behind the camera. P still hands back a pixel; Section 4 shows the trap and the test. - Walking back while zooming in, as in the dolly zoom of read 1. In the limit P's last row becomes (0, 0, 0, d₀): M is singular and the centre has gone to infinity, an affine camera (H&Z Ch. 6 Camera Models, §6.3, pp. 166–167). The C̃ = −M⁻¹p₄ of Section 5 then has no answer.

Try this: press Next three times and watch the tiles. Only the third step divides; the second and the fourth are plain matrix multiplies.

4. In Front or Behind?

Here is the first figure once more, with camera 2 turned 180° to face away:

Camera: focal length and principal point

focal length f (px)

500

principal point cₓ (px)

320

principal point c_y (px)

240

Camera 1: move it and turn it

camera 1 sideways x (m)

0

camera 1 height (m, up is −)

0

camera 1 forward z (m)

0

camera 1 turn left/right (°)

0

camera 1 tilt up/down (°)

0

Camera 2: move it and turn it

camera 2 sideways x (m)

1

camera 2 height (m, up is −)

0

camera 2 forward z (m)

0

camera 2 turn left/right (°)

180

camera 2 tilt up/down (°)

0

the worked rigcamera 1's poseturn without movingface awayReset

• X is now behind camera 2. Its ray is red and dashed, and the inset says "(behind the camera: X)".
• The depth tile has gone negative, −4 m, with "no image" beside it.

The depth was in P all along. The third entry of X_cam is the distance ahead along the camera's forward axis, and the forward axis is row 3 of R (H&Z Ch. 6 Camera Models, result 6.1 and eq. 6.15, pp. 162–163):

• ${𝐫}_3$: row 3 of R, the camera's forward axis in world coordinates (a unit vector).
• $\mathrm{d}\mathrm{e}\mathrm{p}\mathrm{t}\mathrm{h}(𝐗)$: in metres, along the forward axis, the third entry of X_cam. For a P built as K[R | t] it is also the last number of P(X, 1), the one we divide by: K's last row is (0, 0, 1), so that number is the third entry of R X + t, which is r₃ · X − r₃ · C̃: the same thing.

For camera 2:

• depth(X) = (−0.17365, 0, 0.98481) · ((0.3, −0.2, 4) − (1, 0, 0)) = 4.06078 m. Positive: X is in front of camera 2.
• Now take the point (0.3, −0.2, −4), four metres behind camera 1. P₁ gives it a last number of −4, and dividing still hands back a "pixel": (282.5, 265). It is inside the frame and looks perfectly reasonable.

Why it exists. Eq. 3 gives every point a pixel, in front or behind. The depth says which, so no projection can be trusted without it, and read 9 picks camera 2's pose with it.

What it assumes. - P was built as K[R | t] with R a rotation, so r₃ has length 1 and det M > 0. For a P from anywhere else the book divides by the length of M's third row and multiplies by the sign of det M (H&Z Ch. 6 Camera Models, eq. 6.15, p. 162). - X is a finite point, with a 1 added. A point at infinity has no depth.

Where it breaks. - A P that was scaled by a negative number. −P is the same camera, but the last number of P(X, 1) flips sign for every point; the NOTE in Section 5 has the check. - Depth exactly zero. The point is on the principal plane and has no pixel at all (H&Z Ch. 6 Camera Models, p. 160). Try this: in the "face away" figure above, open camera 2's panel and set "camera 2 turn left/right" to 90°, then step it up to 100°. The depth of X falls from 0.7 m to 0.06564 m at 99°; at 100° it reads −0.00523 m and the figure says "no image". In between, X crossed camera 2's principal plane.

That is the trap. P never refuses: any point whose depth is not exactly zero gets a pixel, in front or behind. The sign of the depth is the only tell, so check it before you trust a projection. The figures in this series never draw a pixel for a point with negative depth; they show the red ray and say "no image" instead. Read 9 leans on exactly this test to choose camera 2's pose out of four candidates.

5. Reading P Back

So far we built P from a camera we placed. Calibration and the two-view reads hand it back the other way: as twelve plain numbers. Here are camera 2's, with the first three columns marked M and the last p₄:

Table. 1 P₂ = K[R₂ | t₂] for the series' camera 2, split into M = K R₂ and the last column p₄ = K t₂.

Where is the camera? Think of what P does to the camera's own centre. Every point on a line through the centre lands on the same pixel; that is what "a pixel is a ray" means. The centre itself lies on every such line, so it cannot land on any one pixel. P sends it to (0, 0, 0), which is no pixel at all (H&Z Ch. 6 Camera Models, §6.2.1 and table 6.1, pp. 158–159):

• $M$: the left 3 × 3 block of P. From eq. 3 it equals K R.
• ${𝐩}_4$: the last column of P, which equals K t = −K R C̃ = −M C̃. Solve that for C̃ and you get the second line.
• $𝐂$: the centre with a 1 added, the one point P sends to zero.

For camera 2:

• C̃₂ = −M₂⁻¹ p₄ = (1, 0, 0). The camera is back where we put it, read from the twelve numbers alone.
• det M₂ = 250 000 > 0, and M₂'s third row is r₃, a unit vector, so the last number of P₂(X, 1) is the true depth, sign and all (H&Z Ch. 6 Camera Models, result 6.1, p. 162).
• The principal point is M times its own third row, M m³ (m³ is that row stood up as a column), which divides to (320, 240), the K we started from.

Why it exists. Calibration and the two-view reads hand back P as twelve numbers, not as K, R and C̃. This finds where that camera stands without taking anything else apart.

What it assumes. - P has rank 3, so exactly one point, up to scale, goes to zero (H&Z Ch. 6 Camera Models, §6.2.1, p. 158). The first line needs nothing more: the null vector is the centre even when the frames are only projective (H&Z Ch. 6 Camera Models, §6.2.5, p. 165). - M is invertible, a finite camera, for the second line.

Where it breaks. - A camera at infinity. M is singular and M⁻¹ does not exist (H&Z Ch. 6 Camera Models, §6.3, p. 166). The first line still works: take the null vector of P, which the book finds with an SVD (H&Z Ch. 6 Camera Models, §6.2.4, p. 163), and it comes out with a last number of 0, a direction instead of a place (H&Z Ch. 6 Camera Models, table 6.1, p. 158).

NOTE: P is homogeneous, so −P is the same camera, but −P flips the sign of every last number. When a P comes from somewhere else, check det M first and negate P if det M < 0. If the depths of points you know are in front are still negative, your image and camera frames disagree in handedness. The book's fix is to negate y (H&Z Ch. 6 *Camera Models*, pp. 164–165); ours is to pick x right, y down, z forward from the start.

One more number, as a teaser. Camera 1 sends C₂ to P₁(C₂, 1) = (500, 0, 0). Its last number is 0: no division possible. In Section 4's words, C₂ has depth zero for camera 1, so it lies on camera 1's principal plane, the plane through C₁ parallel to its image. Read 7 will show why that one zero makes every epipolar line in camera 1's picture a row.

6. In OpenCV

Table. 2 This read in OpenCV 4.x [OPEN24], with the series' camera 2.

7. What Came Next

P₂ has four columns, and we only read the last one. The first three are pictures too: the pictures of points no camera can ever reach, the far ends of the world's three axes. Stand between two railway tracks and they meet at the horizon, a point you can see but never walk to. Where do parallel rails meet, and why is it a column of P? That is Points at Infinity — Horizons, Vanishing Points and the Floor.

8. References

1. [HART04] R. Hartley, A. Zisserman, Multiple View Geometry in Computer Vision, 2nd ed., Cambridge University Press, 2004. Chapters used in this read: Ch. 6 Camera Models; Ch. 7 Computation of the Camera Matrix P.
2. [OPEN24] OpenCV, Camera Calibration and 3D Reconstruction (calib3d), OpenCV 4.x documentation.