Stereo Depth — Rectification and Z = fB/d

Read 9 found camera 2's pose, and read 8 crossed two rays with a singular vector. Do that for every pixel of a 640 × 480 photo and it is slow and fragile. Line the two pictures up first and the whole thing is one subtraction and one division.

The read is three steps:

• turn both pictures until every match sits on the same row in both;
• subtract: how far the match slid along its row is the disparity d;
• divide: depth is Z = fB/d, and its error grows with the square of the distance.

Look out of a train window. The poles by the track rush past, while the hills on the horizon crawl. How fast something slides across your view is how near it is. Hartley and Zisserman use the same picture for the same equation: points closer to the camera "appear to move faster than those further away – a common experience when looking out of a train window" (Hartley & Zisserman, Ch. 9 *Epipolar Geometry and the Fundamental Matrix*, eq. 9.6, p. 249). A stereo rig is that train ride frozen into two photos taken one baseline apart.

Brush up on these first ;)

• Points at Infinity — Horizons, Vanishing Points and the Floor: a camera that only turns moves its pixels by KRK⁻¹, and needs no new photo to do it.
• From Matches to Motion — Four Cameras, One Right Answer: R and t of camera 2, plus one known length, the baseline B in metres.
• Triangulation — Two Rays That Almost Meet: the ray angle at X, the guide to how well two rays pin a point down.

The rig is the one from every read of the series. Both cameras share

camera 1 sits at the origin facing straight ahead, camera 2 sits at C̃₂ = (1, 0, 0) m turned 10° toward camera 1, and X = (0.3, −0.2, 4) m lands at x₁ = (357.5, 215) px and x₂ = (320.644, 215.374) px.

1. Turn Both Pictures Until the Rows Agree

how far the pictures are turned

0

▶

Camera: focal length and principal point

focal length f (px)

500

principal point cₓ (px)

320

principal point c_y (px)

240

Camera 1: move it and turn it

camera 1 sideways x (m)

0.0

camera 1 height (m, up is −)

0.0

camera 1 forward z (m)

0.0

camera 1 turn left/right (°)

0.0

camera 1 tilt up/down (°)

0.0

Camera 2: move it and turn it

camera 2 sideways x (m)

1.0

camera 2 height (m, up is −)

0.0

camera 2 forward z (m)

0.0

camera 2 turn left/right (°)

10

camera 2 tilt up/down (°)

0.0

lined upthe worked rigReset

Stand still and turn your head. Nothing new appears from behind anything; the whole view only slides and stretches. A camera that turns about its own centre is the same, so we do not need a new photo to turn it: every pixel just moves to a new place, and read 3 showed where, by KRK⁻¹.

What the figure shows:

• camera 2 (rust) starts turned 10° in, and each point sits a little off its row in camera 2's picture: the small rust numbers are the row mismatches v₁ − v₂ in px, measured against the grey row each point has in camera 1's picture;
• press play: camera 2 turns back, its picture is remapped, and every match slides onto its row; the mismatches all go to 0;
• camera 1 (blue) never moves. In our rig it is already lined up: its epipole is e = (500, 0, 0), a point at infinity along x (Epipolar Geometry — A Point Becomes a Line), so its epipolar lines are rows already.

So the plan for rectification is: turn each camera about its centre until both face the same way, with their x axes along the baseline. Then the baseline is parallel to both image planes, both epipoles go to infinity along x, and every epipolar line is a row. Hartley and Zisserman state the goal as "a pair of identical cameras placed side by side with their principal axes parallel" (H&Z Ch. 11 Computation of the Fundamental Matrix F, §11.12, p. 303).

The book reaches it from F alone, without calibration (H&Z Ch. 11 Computation of the Fundamental Matrix F, §11.12, pp. 302–307). We know K and R₂, so we can take the calibrated route: undo camera 2's turn with the turning-camera map of Example 9.7 (H&Z Ch. 9 Epipolar Geometry and the Fundamental Matrix, p. 250). Let's write it:

• H₁, H₂ are the 3 × 3 maps applied to the pixels of picture 1 and picture 2. Camera 1 needs no turn, so H₁ = I.
• R₂ is camera 2's rotation (10° about y). As in every read, R maps world to camera: x₂ = K R₂ (X − C̃₂), and the turned camera is x₂ʳ = K (X − C̃₂) = K R₂ᵀ K⁻¹ x₂.
• Read right to left: K⁻¹ takes a pixel to its ray, R₂ᵀ turns the ray back by 10°, and K takes it back to a pixel.
• x₂ is a pixel of the original picture 2 (homogeneous), x₂ʳ its place in the turned picture. The r is for rectified; a prime in this series means picture 2, which x₂ already is.

With our numbers:

• x₂ = (320.644, 215.374) goes to x₂ʳ = (232.5, 215). Its row is now 215, the same as x₁'s, so v₁ − v₂ = 0.
• The four other points in the figure start with row mismatches of −2.25, 2.71, −1.61 and −8.91 px. After the turn all four are 0.
• The turned camera 2 is P₂ʳ = K[I | −C̃₂]. Its last column is (−500, 0, 0): the baseline, in pixels, along x only.

Why it exists. Matching and depth both need every epipolar line to be a row. This is the map that puts picture 2's pixels there without taking a new photo, and the rest of the read runs on the turned pictures.

What it assumes. - camera 2 only turns about its own centre. Then every pixel moves by K R K⁻¹ whatever its depth, with no parallax (H&Z Ch. 8 More Single View Geometry, §8.4.2 and Fig. 8.6, p. 204). - K and R₂ are known (read 6 and read 9), and both cameras share K. With two different K the map is K′ R K⁻¹ (H&Z Ch. 9 Epipolar Geometry and the Fundamental Matrix, Example 9.7, p. 250). - the baseline already lies along camera 1's x axis, so H₁ = I. Rows agree when what is left between the two turned cameras is a slide along x (H&Z Ch. 9 Epipolar Geometry and the Fundamental Matrix, Example 9.6, pp. 248–249). - the pixels are pinhole pixels: the lens is undone first (Lens Distortion — Bending Straight Lines Back). In a raw photo the epipolar lines are curves, and no 3 × 3 map makes a curve a row.

Where it breaks. - Camera 2 sits forward as well as sideways. Camera 1 has to turn too, and its picture swings away from where it was. When the cameras move straight ahead, the epipole sits inside the picture (H&Z Ch. 9 Epipolar Geometry and the Fundamental Matrix, Fig. 9.8, p. 248), and the pixels near it are thrown out toward infinity, as the map G of §11.12.1 shows (H&Z Ch. 11 Computation of the Fundamental Matrix F, p. 305). Try this: open the camera panel, set camera 2 forward z to 1 m and press play: camera 1 now turns 45°, and X's pixel in camera 1 ends at (901.08, 201.78) px, off the 640 px picture. - R₂ is a little wrong. A turn about y that is 0.2° short leaves the rows almost right, but it moves the match sideways, and that goes straight into d. Try this: drag "how far the pictures are turned" to 0.98: X's match sits at (234.3, 215.02) px, so d reads 123.2 px instead of 125 and Z = fB/d (section 3) gives 4.06 m. A turn about x would show up as rows that miss; for a correct rectification the row difference should be 0 (H&Z Ch. 11 Computation of the Fundamental Matrix F, Fig. 11.12, p. 308).

NOTE: OpenCV's stereoRectify does not turn only one camera. It returns a turn for each, R1 and R2, and in the 4.x source each camera gets half of the relative turn before both are turned together onto the baseline. The rows come out lined up all the same. Its two turned cameras differ from ours by one common turn, so its d for a given point, and the axis its Z is measured along, differ slightly from ours; Z = fB/d holds just the same and gives back the same point (outside the book).

Try this: press play and watch the largest row mismatch fall from −8.91 px to 0 px while the H tile grows into the matrix above.

2. Disparity

depth Z (m)

4

▶matching error Δd (px)

0.5

Camera: focal length and principal point

focal length f (px)

500

principal point cₓ (px)

320

principal point c_y (px)

240

Camera 1: move it and turn it

camera 1 sideways x (m)

0.0

camera 1 height (m, up is −)

0.0

camera 1 forward z (m)

0.0

camera 1 turn left/right (°)

0.0

camera 1 tilt up/down (°)

0.0

Camera 2: move it and turn it

camera 2 sideways x (m)

1.0

camera 2 height (m, up is −)

0.0

camera 2 forward z (m)

0.0

camera 2 turn left/right (°)

0.0

camera 2 tilt up/down (°)

0.0

twice as farhalf the baselinea sloppy matchcamera 2 turned inthe worked rigReset

Now the two cameras face the same way, B apart, like two eyes. Close one eye, then the other: a pen held near your face jumps a lot, the wall behind it barely moves. That jump is what we measure.

What the figure shows:

• the rig after the turn: camera 2 sits at (B, 0, 0) and faces straight ahead like camera 1;
• in camera 2's picture the match x₂ sits on the same row as x₁, but a little to the left. The blue tick marks where x₁ sat in picture 1, and the rust dimension is the slide;
• under the picture, a plot of depth against that slide, and a strip that enlarges the thick rust band on camera 1's ray (section 4).

The slide is the disparity:

• u₁ is the column of the match in picture 1, u₂ its column in the turned picture 2, both in px.
• After rectification the rows agree, so the whole match is this one number. The search for x₂ is now along one row, "a 1-dimensional search" (H&Z Ch. 11 Computation of the Fundamental Matrix F, p. 307).
• With camera 2 to the right of camera 1, the match always lands further left in picture 2, so d > 0 for any point in front.

For our point: d = 357.5 − 232.5 = 125 px.

Where it breaks. - A point that one camera sees and the other does not, because something nearer hides it, has no u₂ at all, so no d. Matchers find those pixels by matching both ways and throwing out the ones that disagree ([SCHA02] §3.4; read 11).

Try this: drag Z in the figure: x₁ never moves (the point slides along camera 1's ray), while x₂ runs right toward x₁ as the point goes away.

3. Depth Is fB Over d

depth Z (m)

8

▶matching error Δd (px)

0.5

Camera: focal length and principal point

focal length f (px)

500

principal point cₓ (px)

320

principal point c_y (px)

240

Camera 1: move it and turn it

camera 1 sideways x (m)

0.0

camera 1 height (m, up is −)

0.0

camera 1 forward z (m)

0.0

camera 1 turn left/right (°)

0.0

camera 1 tilt up/down (°)

0.0

Camera 2: move it and turn it

camera 2 sideways x (m)

1.0

camera 2 height (m, up is −)

0.0

camera 2 forward z (m)

0.0

camera 2 turn left/right (°)

0.0

camera 2 tilt up/down (°)

0.0

twice as farhalf the baselinea sloppy matchcamera 2 turned inthe worked rigReset

The same figure, with the point twice as far away: Z = 8 m, and d has halved to 62.5 px. The curve under the picture is a hyperbola: every time Z doubles, d halves.

Why? It is similar triangles once more, as in The Pinhole Camera — A Pixel Is a Ray. In the rectified rig camera 1 is at the origin and camera 2 at (B, 0, 0), both with R = I. The pinhole projection, once per camera:

• X and Z are the point's sideways position and depth in camera 1's frame, in m. Depth is the distance along the axis, not along the ray.
• f = 500 px is the focal length and c_x = 320 px the principal point, the same for both turned cameras.
• B = 1 m is the baseline, the distance between C₁ and C₂.
• c_x cancels and so does X: d depends only on how far away the point is. The book gets the same thing from eq. 9.6 for a camera that only slides sideways (H&Z Ch. 9 Epipolar Geometry and the Fundamental Matrix, pp. 248–249); the same relation is in the stereo chapter of [SZEL22].

Turn it around and we have depth:

• the baseline B is to the depth Z as the disparity d is to the focal length f, the two triangles with their apex at X.

For our point: Z = 500 × 1 / 125 = 4 m, exactly the depth of the X we started with.

Other depths on the same ray, with B = 1 m:

Table. 1 d = fB/Z with f = 500 px and B = 1 m.

A point 20 m away moves only 25 px between two cameras 1 m apart. Everything beyond it is squeezed into the last 25 px.

Why it exists. It is the read in one line. Once the rows agree, a pixel's depth is one subtraction and one division, instead of read 8's singular vector per pixel.

What it assumes. - the two cameras are rectified: same orientation, the baseline along x, the same f and the same c_x. That is the pure sideways slide of eq. 9.6 (H&Z Ch. 9 Epipolar Geometry and the Fundamental Matrix, Example 9.6, pp. 248–249). - pinhole pixels: the lens is undone before d is measured. - B is known in metres. Read 9 gives the pose only up to scale, so B is the one length measured by hand, and Z comes out in the units of B. - the point is in front of both cameras, so d > 0.

Where it breaks. - f or B is off. Every depth is scaled by the same factor, and nothing in the pictures shows it. Measure B, and take f from a calibration (read 6). - d goes to 0. By eq. 9.6 a pixel moves by Kt/Z, so a point at infinity does not move at all between two parallel cameras (H&Z Ch. 9 Epipolar Geometry and the Fundamental Matrix, p. 249), and fB/0 has no answer. Table 1 shows it coming: 25 px at 20 m, and one pixel of d there is 0.8 m (Table 2).

Try this: drag Z back from 8 m to 4 m and watch d double from 62.5 px to 125 px.

4. How Fast the Error Grows

depth Z (m)

4

▶matching error Δd (px)

5

Camera: focal length and principal point

focal length f (px)

500

principal point cₓ (px)

320

principal point c_y (px)

240

Camera 1: move it and turn it

camera 1 sideways x (m)

0.0

camera 1 height (m, up is −)

0.0

camera 1 forward z (m)

0.0

camera 1 turn left/right (°)

0.0

camera 1 tilt up/down (°)

0.0

Camera 2: move it and turn it

camera 2 sideways x (m)

1.0

camera 2 height (m, up is −)

0.0

camera 2 forward z (m)

0.0

camera 2 turn left/right (°)

0.0

camera 2 tilt up/down (°)

0.0

twice as farhalf the baselinea sloppy matchcamera 2 turned inthe worked rigReset

This copy opens with a sloppy match: the matching error Δd is 5 px instead of 0.5 px, and the thick rust band along camera 1's ray, every depth the point could have with that error, stretches.

What it shows:

• d shrinks like 1/Z, so the same pixel of error is a bigger share of a small d: a fixed Δd costs more depth the farther you look;
• the band is lopsided. Its far end is farther from X than its near end; the strip under the picture enlarges it and writes both sides;
• play Z: d collapses while the band runs away. Shrink B as well and d can fall to Δd; from there on the far end is at infinity.

How fast does it grow? Take the derivative of eq. 6 (outside the book, one line of calculus):

• Δd is the matching error in px, ΔZ the depth error it causes in m.
• The minus sign only says that a bigger d means a nearer point; the size of the error is what matters.
• Z² is the whole story: double the distance and the error is four times larger. Double f or B and it halves.

At 4 m: 4² / (500 × 1) = 0.032 m per px, so a 0.5 px match costs ΔZ ≈ 0.016 m.

The linear rule hides the lopsidedness. The exact range comes straight from eq. 6, with d moved by Δd either way:

• the near bound uses d + Δd, the far bound d − Δd; the far bound needs d > Δd, otherwise it is at infinity.
• Since 1/d bends upward, the same Δd buys more depth on the far side than on the near side.

For Δd = 0.5 px: 3.98406 ≤ Z ≤ 4.01606 m. For Δd = 5 px the linear rule says 0.16 m, and the exact bounds are 3.8462–4.1667 m; the far side is visibly the longer one.

Per pixel of disparity, on the same ray:

Table. 2 ΔZ ≈ Z²Δd/(fB) with f = 500 px, B = 1 m and Δd = 1 px.

Ten times farther, from 2 m to 20 m, is a hundred times worse, from 0.008 m to 0.8 m per px.

Why it exists. Z = fB/d gives a depth; this says how far to trust it. It is also the rule to size a rig by: pick the farthest depth and the error you can live with, and it gives fB.

What it assumes. - f and B are exact; only the match is off. - Δd is small next to d, so the straight line of the derivative can stand in for the curve fB/d. - the same Δd at every pixel and every depth.

Where it breaks. - Δd is not small next to d. Far away, or with a short baseline, the linear rule gives a band of the same size on both sides, and the true band is nothing like that. Try this: in camera 2's panel set sideways x to 0.25 m, then Δd to 5 px and Z to 20 m. d is 6.25 px, the linear rule says ±16 m, and the exact band runs from 11.1111 m to 100 m: 8.9 m on the near side, 80 m on the far side. - f or B is off as well. From Z = fB/d, an error in B moves Z by Z ΔB/B (outside the book). It grows only like Z, not Z², but it never averages out. - Δd is not the same everywhere. Blank walls, depth edges and pixels hidden from one camera are the typical problem areas for matchers ([SCHA02] §5.1), so the per-pixel numbers of Tables 2 and 3 hold only where the match is good. Read 11 cleans those pixels out.

This is read 8 again, from the other side. There, 0.5 px of noise along camera 2's epipolar line cost 16.55 mm of depth (16.62 mm in all) on the turned-in rig. For a parallel rig with the same f, B and Z, eq. 7 predicts 16.00 mm. The two agree to within a millimetre, although eq. 7 is for two cameras facing the same way and read 8's camera 2 is turned 10°.

NOTE: Z is depth along camera 1's axis, not the distance from the camera to the point. For a point near the edge of the picture the two differ; the error rule is about Z.

Try this: play Z and read the band's two ends off the last tile.

5. Wider Baseline: Better and Worse

depth Z (m)

4

▶matching error Δd (px)

0.5

Camera: focal length and principal point

focal length f (px)

500

principal point cₓ (px)

320

principal point c_y (px)

240

Camera 1: move it and turn it

camera 1 sideways x (m)

0

camera 1 height (m, up is −)

0

camera 1 forward z (m)

0

camera 1 turn left/right (°)

0

camera 1 tilt up/down (°)

0

Camera 2: move it and turn it

camera 2 sideways x (m)

0.5

camera 2 height (m, up is −)

0

camera 2 forward z (m)

0

camera 2 turn left/right (°)

0

camera 2 tilt up/down (°)

0

twice as farhalf the baselinea sloppy matchcamera 2 turned inthe worked rigReset

This copy opens with half the baseline: B is 0.5 m instead of 1 m, d has halved to 62.5 px and the error per pixel has doubled to 0.064 m.

The rule is in eq. 7: B sits in the denominator next to f. Doubling B doubles d and halves ΔZ.

Table. 3 The same point at 4 m seen by three rigs, f = 500 px.

So why not build every rig 2 m wide? Because a wider rig sees each object from two more different angles, and matching pays for that:

• more is hidden from one camera. Things behind a near object are seen by one camera and not the other, and those pixels have no match at all;
• the search is longer. A near point's d grows with B; the matcher must slide its window over a longer stretch of the row, and has more chances to stop at the wrong place;
• patches look less alike. A surface seen from two directions far apart is stretched and lit differently in the two pictures, and matching compares patches (read 11).

Read 8 said the same thing from the other side. The ray angle at X is 14.20° for our 1 m baseline, 7.14° with camera 2 at 0.5 m and 27.28° at 2 m: a bigger angle pins the point down better, and makes the two views less alike.

One contrast before we move on. In Points at Infinity — Horizons, Vanishing Points and the Floor, one camera got depth from the ground plane: a pixel went back to the floor through a homography, which only works for things standing on that floor. With two cameras, depth needs no plane at all, only B.

Try this: raise B from 0.5 m to 2 m and watch d double twice, to 250 px, while the error per pixel halves twice, to 0.016 m.

6. In OpenCV

Table. 4 This read in OpenCV 4.x [OPEN24].

• stereoCalibrate's R and T take a point from camera 1's frame to camera 2's, the same direction as our R₂ and t₂.
• stereoRectify returns R1 and R2, the turns for the two cameras, and P1, P2, the turned cameras. For side-by-side cameras P2's last column is T_x·f, with T_x the shift between the cameras: here T_x = −1 (m) and T_x·f = −500.
• With CALIB_ZERO_DISPARITY both turned pictures share the principal point, so d is u₁ − u₂ and nothing else. Without it the function may shift the pictures to keep more valid pixels, and the difference of the two c_x has to be taken out of d first.

7. What Came Next

Z = fB/d needs d, and we read d off one known match: x₁ and x₂ were handed to us. A depth map needs d for every pixel of picture 1, which means finding every match, along its row, in a picture where half the surfaces are blank walls and the other half repeat. How? That is From Two Photos to a Point Cloud.

8. References

1. [HART04] R. Hartley, A. Zisserman, Multiple View Geometry in Computer Vision, 2nd ed., Cambridge University Press, 2004. Chapters used in this read: Ch. 8 More Single View Geometry; Ch. 9 Epipolar Geometry and the Fundamental Matrix; Ch. 11 Computation of the Fundamental Matrix F.
2. [OPEN24] OpenCV 4.x, Camera Calibration and 3D Reconstruction (calib3d): stereoCalibrate, stereoRectify.
3. [SZEL22] R. Szeliski, Computer Vision: Algorithms and Applications, 2nd ed., 2022: the stereo chapter (outside the book).
4. [SCHA02] D. Scharstein, R. Szeliski, "A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms", IJCV 47, 2002: §3.4 and §5.1 (outside the book).