Points at Infinity — Horizons, Vanishing Points and the Floor

Stand between two rails and they meet at the horizon. On the ground they never meet; in the photo they do, at a pixel you can point to. Which pixel? And can the camera tell us where on the ground any pixel is?

Both answers come out of the chain we already have. This read sends some world points to infinity, and the chain gets simpler instead of breaking: for a direction, t drops out and only K R is left.

Brush up ;)

• Where Is the Camera? — R, C and t: bring P₂ = K[R₂ | t₂] for camera 2, with its numbers, and the habit of reading P as four columns.
• The Pinhole Camera — A Pixel Is a Ray: bring "add a 1, divide by the last number". This whole read is about what happens when that last number is 0.

Same rig as always: K has f = 500 px and principal point (320, 240), camera 1 sits at the origin looking down z, camera 2 sits at C₂ = (1, 0, 0) turned 10° toward camera 1, y points down, and the floor is y = 1 m.

1. Lines Are Three Numbers Too

A pixel is already three numbers (u, v, 1), and any multiple of them is the same pixel. A line in the picture gets the same treatment. The line a·u + b·v + c = 0 is the triple l = (a, b, c), and doubling all three draws the same line. Once points and lines are both triples, the two questions we keep asking of a picture become one-liners.

make them parallelmeet off the pictureReset

• The blue line l₁ goes through A and B, the rust line l₂ through C and D.
• The three tiles are the two lines as triples and their meet as a triple. The meet's last number, w, is not 1: divide by it and you get the pixel where the lines cross.
• Drag any of the four points and every triple updates.

Let's write the three facts down (Hartley & Zisserman, Ch. 2 Projective Geometry and Transformations of 2D, results 2.1, 2.2 and 2.4, pp. 27–28):

• x = (u, v, w) is a point in the picture; it stands for the pixel (u/w, v/w). p and q are two more points.
• l = (a, b, c) is a line; xᵀl = a·u + b·v + c·w, which is the line's equation with the point plugged in.
• × is the ordinary cross product of two 3-vectors. The cross product is perpendicular to both inputs, so l₁ × l₂ has a zero dot product with both lines: it lies on both, which is exactly "where they meet".

The book's own check (H&Z Ch. 2 Projective Geometry and Transformations of 2D, ex. 2.3, p. 27): the line x = 1 is (−1, 0, 1), the line y = 1 is (0, −1, 1), and

• (−1, 0, 1) × (0, −1, 1) = (1, 1, 1), the point (1, 1). As it should be.

Now the figure's defaults, A = (80, 300), B = (560, 120), C = (100, 80) and D = (540, 360):

• l₁ = A × B = (180, 480, −158400) and l₂ = C × D = (−280, 440, −7200);
• l₁ × l₂ = (66240000, 45648000, 213600). Divide by 213600 and the lines cross at (310.11, 213.71).

No slopes, no special case for vertical lines, no solving two equations by hand. Two cross products and one division.

Why it exists. The rails question is "where do two picture lines cross?", and I want one answer for every pair of lines, vertical and parallel ones included. With points and lines both as triples, that answer is one cross product, and the rest of this read leans on it.

What it assumes. - The lines are straight in the picture. A real lens bends them, and then no single triple describes the line (H&Z Ch. 7 Computation of the Camera Matrix P, §7.4, p. 189). - The two lines are different lines: the book's rule is that "two distinct lines meet in a single point" (H&Z Ch. 2 Projective Geometry and Transformations of 2D, p. 28).

Where it breaks. - The same line twice: l × l = (0, 0, 0), which is no point at all. Check that the meet is not all zeros before you divide. - Nearly parallel lines: the meet runs far out and jumps with every pixel. From the defaults, moving D down 1 px moves the meet from (310.11, 213.71) to (309.64, 213.88). From "make them parallel", the same 1 px puts it at (54100, −19957.5), and 2 px at (27100, −9832.5). With noisy segments the book does not trust one pair: it fits all the lines through one common point (H&Z Ch. 8 More Single View Geometry, §8.6.1, pp. 215–216). - Bent lines: undo the lens first, which is Lens Distortion — Bending Straight Lines Back. - Try this: press "make them parallel", then drag D down a little and watch the meet come back from infinity, far off the picture.

Try this: press "meet off the picture". The meet leaves the frame and an arrow at the edge points to it, but the triple is as ordinary as before.

2. Where Parallel Lines Meet

Here is the same figure again, with C and D moved so that l₂ runs parallel to l₁. The meet has run off to infinity, and its last number is exactly 0.

make them parallelmeet off the pictureReset

That is the whole idea of this read, so let's see why. Two parallel lines share their a and b and differ only in c (H&Z Ch. 2 Projective Geometry and Transformations of 2D, §2.2.2, pp. 28–29):

• The meet is a perfectly good triple; it just has no pixel, because we cannot divide by its last number. It is a point at infinity (the book's ideal point).
• (b, −a) is the direction the two lines run in. So a point at infinity is a direction, and the scale in front, (c′ − c), does not matter.
• Every point at infinity (u, v, 0) satisfies (0, 0, 1)·x = 0, so they all lie on one line, the line at infinity l∞ = (0, 0, 1).

The book's number (H&Z Ch. 2 Projective Geometry and Transformations of 2D, ex. 2.5, p. 28): the lines x = 1 and x = 2 are (−1, 0, 1) and (−1, 0, 2), and

• (−1, 0, 1) × (−1, 0, 2) = (0, 1, 0): the point at infinity along y, which is where two vertical lines "meet".

Why it exists. Section 1's rule would have an exception without it: parallel lines. Giving them a meet, a triple with last number 0, keeps the rule exception-free, and the line l∞ through all such meets is what becomes the horizon in section 3.

What it assumes. - The lines are parallel in the picture, which is not the same as parallel in the world; see the NOTE below.

Where it breaks. - Dividing by w = 0: the triple has no pixel, and code that divides anyway returns infinities. Keep the triple, or its direction (b, −a), and test w before dividing; section 5's note says the same for the floor. - l∞ = (0, 0, 1) is the line at infinity only in the picture's own coordinates. A homography sends it to a finite line unless the map is affine (H&Z Ch. 2 Projective Geometry and Transformations of 2D, result 2.17, p. 48). For the floor that finite line is the horizon of section 3.1.

The picture behind this (H&Z Ch. 2 Projective Geometry and Transformations of 2D, fig. 2.1, p. 29) is read 1's: a triple is a ray from the origin, and the plane w = 1 catches it at the pixel. A point at infinity is a ray lying flat, parallel to that plane, so it never lands.

NOTE: parallel lines in the picture meet at w = 0. Parallel lines in the world usually do not stay parallel in the picture: the rails meet at a finite pixel, which is section 3.

3. Vanishing Points Are the Columns of P

Driving at night, the moon follows the car. The trees rush past because the car's translation moves them across your view; the moon is so far away that translation does nothing to it, and only turning your head moves it. Points at infinity in the world behave like that moon, and it shows up in the algebra as a vanishing t.

zoom the picture out

7

Camera: focal length and principal point

focal length f (px)

500

principal point cₓ (px)

320

principal point c_y (px)

240

Camera 2: move it and turn it

camera 2 sideways x (m)

1.0

camera 2 height (m, up is −)

0.0

camera 2 forward z (m)

0.0

camera 2 turn left/right (°)

10

camera 2 tilt up/down (°)

0.0

the worked rigcamera 1's viewpitch down 20°true sizeReset

• The rails run in the world's z direction and the faint floor lines in the world's x direction. In camera 2's zoomed-out picture each family meets at one rust point: v_z for the rails, v_x for the floor lines.
• The upright posts stay parallel in camera 2's picture: their v_y is at infinity.
• The first tile is P₂ itself, and its first three columns are the three vanishing points.

A world direction d is a world point with last number 0, (d, 0). Feed it to P = K[R | t] and the fourth column, the one carrying t, gets multiplied by that 0 (H&Z Ch. 8 More Single View Geometry, §8.6.1, result 8.20, pp. 213–215):

• v is the vanishing point of every world line with direction d, as a triple; divide by its last number to get the pixel.
• d (bold) is a 3-D direction, not a point and not a disparity. Only its direction counts; its length is a scale on v.
• R is the camera's rotation and K its intrinsics, both from earlier reads. t is gone: moving the camera without turning it never moves a vanishing point.

Why it exists. I want the pixel where the rails meet from the camera alone, without knowing where the rails are. v = KRd says that pixel depends only on the direction and the camera's turn, not on the rails' position or on t (H&Z Ch. 8 More Single View Geometry, result 8.20, pp. 214–215).

What it assumes. - d is written in the same world frame that R turns from; in this series, camera 1's. - The pinhole chain: straight world lines stay straight in the picture (H&Z Ch. 7 Computation of the Camera Matrix P, §7.4, p. 189).

Where it breaks. - d parallel to the picture (the last number of Rd is 0): v is at infinity and those lines stay parallel in the picture, like the posts' v_y = (0, 500, 0) in camera 2. The reverse does not hold: lines that meet on the camera's principal plane also come out parallel (H&Z Ch. 8 More Single View Geometry, p. 215). Keep v as a triple; section 3.1 shows v_y coming in from infinity once camera 2 pitches down. - d nearly parallel to the picture: v lands far outside, like v_x at (−2515.64, 240). Measured from two picture lines, such a point is the nearly-parallel case of section 1, and a pixel of error moves it a long way.

Plug in the world's three axes, (1, 0, 0), (0, 1, 0) and (0, 0, 1), and KRd picks out the first, second and third column of KR. Those are the first three columns of P (H&Z Ch. 6 Camera Models, §6.2.1, table 6.1, pp. 158–159). For camera 2:

• p₁ = v_x: divide by −0.174 and it is the pixel (−2515.64, 240), far off the left of a 640-wide picture.
• p₂ = v_y = (0, 500, 0): last number 0, at infinity, because camera 2 is level.
• p₃ = v_z: divide by 0.985 and it is (408.16, 240). That is where the rails meet.
• p₄ is P applied to the world origin (0, 0, 0, 1): the image of the world origin, which is camera 1's centre. It is −1 times p₁, so it lands on the same pixel as v_x: camera 1 sits exactly along the world x axis from camera 2. Epipolar Geometry — A Point Becomes a Line will give that pixel a name of its own.

Camera 1 makes a good sanity check. Its P₁ = K[I | 0], so its columns are the columns of K: v_x = (500, 0, 0) and v_y = (0, 500, 0) are both at infinity, and v_z = (320, 240) is the principal point. Looking straight down z, the rails meet at the centre of the picture.

Try this: press "camera 1's view". Camera 2 keeps its place but takes camera 1's orientation, and its vanishing points become camera 1's, (320, 240) for v_z and the other two at infinity. Same R, same vanishing points, whatever t.

3.1 The Horizon

The horizon is not a thing in the scene; it is a line in the picture. Every direction along the floor, a mix of x and z, vanishes somewhere, and all those points lie on one picture line: the line through v_x and v_z. With section 1's tool that is one cross product, v_x × v_z, and it works even when one of the two is at infinity.

• For camera 2 both v_x and v_z sit on row 240, so the horizon is the row v = 240, the same as camera 1's.
• Press "pitch down 20°". Now all three vanishing points are finite: v_x = (−2697.63, 58.01), v_y = (320, 1613.74), v_z = (413.82, 58.01). The horizon rises to row 58.01, and v_y comes in from infinity far below the picture, where the posts now meet.

Try this: pitch camera 2 down slowly. The horizon climbs the picture and v_y climbs up from the bottom of the zoomed-out view.

3.2 A Camera That Only Turns

Same figure, third tile. If a camera turns about its own centre but does not move, every pixel moves by one 3×3 matrix, whatever the depth of the point behind it (H&Z Ch. 8 More Single View Geometry, §8.4.2, pp. 204–205). With x = K[I | 0]X before the turn and x′ = K[R | 0]X after:

• X = (X, Y, Z, 1) is a point in camera 1's frame, in metres. K⁻¹x gives back (X, Y, Z) up to a scale, its ray, and K[R | 0]X is KR(X, Y, Z): hence K R K⁻¹. The depth is only that scale, so it drops out.
• H∞ is the book's name, the infinite homography (H&Z Ch. 8 More Single View Geometry, p. 205): it is what turning does to the points at infinity.
• Camera 1's pixel x₁ = (357.5, 215) goes to (447.348, 214.274) for any point along its ray. A turn has no parallax.
• (320, 240) goes to (408.16, 240), which is camera 2's v_z. Camera 2 also moved by 1 m, but vanishing points ignore t, so H∞ still maps camera 1's vanishing points onto camera 2's.

Why it exists. A camera that only turns moves every pixel by one matrix, whatever the depth; that is what a panorama is stitched with (H&Z Ch. 8 More Single View Geometry, §8.4.3, p. 205). Between two cameras that also moved, H∞ still carries the vanishing points across, which is how the read checks camera 2's v_z from camera 1's.

What it assumes. - The camera centre does not move, or the point is at infinity. In general x′ = H∞x + Kt/Z, with Z the depth (H&Z Ch. 13 Scene Planes and Homographies, eq. 13.10, p. 339); H∞ alone is the t = 0 or Z = ∞ case. - K is the same before and after: no zoom between the two pictures (H&Z Ch. 8 More Single View Geometry, §8.4.2, p. 204).

Where it breaks. - A near point when the camera moved: the Kt/Z term is parallax (H&Z Ch. 8 More Single View Geometry, fig. 8.6, p. 204). Camera 2 moved 1 m, so H∞ puts x₁ at (447.348, 214.274) while camera 2 really sees X at (320.644, 215.374), 126.71 px away. Along the same ray at 40 m the gap is 13.18 px, at 400 m 1.32 px. - So a panorama needs the camera turned about its own centre, and two cameras that moved need the full two-view geometry of Epipolar Geometry — A Point Becomes a Line.

This matrix comes back twice. Epipolar Geometry — A Point Becomes a Line finds it inside the two-view geometry, and Stereo Depth — Rectification and Z = fB/d builds the same kind of matrix, with the turn undone, to turn the two pictures until their rows agree.

4. The Floor in One Matrix

Lay a sheet of graph paper on the floor and photograph it. Every square comes out a different four-sided shape, but straight lines stay straight, and one rule stretches the whole sheet at once. That rule is a 3×3 matrix, and running it backwards takes any pixel back to its spot on the paper.

Camera: focal length and principal point

focal length f (px)

500

principal point cₓ (px)

320

principal point c_y (px)

240

Camera 1: move it and turn it

camera 1 sideways x (m)

0.0

camera 1 height (m, up is −)

0.0

camera 1 forward z (m)

0.0

camera 1 turn left/right (°)

0.0

camera 1 tilt up/down (°)

0.0

the worked rigabove the horizoncamera 2 m upbottom edgeReset

• Camera 1 (blue) stands 1 m above the floor. The blue ray goes from C₁ through the pixel x₁ to the floor point.
• The default pixel is (357.5, 365), which lands right under the series' point X, shown as a ghost with a dashed drop line.
• Drag the pixel in camera 1's picture and the floor point moves; the dims give its x and z in metres.

A floor point is (x, 1, z): the middle number is fixed by the floor, so only two numbers are free. Write it as x·e_x + z·e_z + o, with o = (0, 1, 0) the floor point under the camera, and push it through P₁ = K[I | 0]:

• e_x = (1, 0, 0) and e_z = (0, 0, 1) are the floor's two axes, and o its origin under the camera, all in metres in camera 1's frame. H's columns are those three, seen through K.
• This is the book's plane homography H = K[r₁ r₂ t] for the plane Z = 0 (H&Z Ch. 8 More Single View Geometry, eq. 8.1, ex. 8.1, p. 196). r₁ and r₂ are the plane's two axes as the camera sees them, and t is the plane's origin; ours are e_x, e_z and o because camera 1 has R = I and t = 0.
• H is a homography: an invertible 3×3 matrix acting on triples. It maps the floor to the picture, and H⁻¹ maps the picture back to the floor.

The numbers:

• the floor point (0.3, 1, 4) goes to the pixel (357.5, 365);
• backwards, H⁻¹ takes (357.5, 365) to the floor point (0.3, 4): x = 0.3 m across and z = 4 m ahead, the spot right under X.
• The pixel (357.5, 200) gives H⁻¹x = (0.075, 1, −0.08). Its last number is negative: the pixel is above the horizon (row 240), its ray climbs, and it never meets the floor.

Why it exists. This is the pixel → floor map that everything in this section runs on: one 3×3 matrix and its inverse instead of a ray-and-plane intersection per pixel.

What it assumes. - The point is on the floor. H maps the plane y = 1 m and nothing else (H&Z Ch. 8 More Single View Geometry, §8.1.1, p. 196). - The floor is flat and the camera's height and turn above it are right: the third column, o, carries the height.

Where it breaks. - A point above the floor: a point 0.5 m above the floor at (0.3, 4) m shows at (357.5, 302.5), and H⁻¹ sends it to (0.6, 8) m, twice as far. Off-plane points do not follow the plane's homography (H&Z Ch. 13 Scene Planes and Homographies, fig. 13.8, p. 336), so use the pixel where the object touches the floor, such as the bottom edge of its box. - Near the horizon one pixel is many metres: row 365 gives z = 4 m and row 366 gives 3.97 m, but row 245 gives 100 m and row 246 gives 83.33 m. - A wrong height: z = 500 × h / (v − 240) grows in proportion to the height h, so a height 10 % off puts every floor point 10 % too far or too near. "camera 2 m up" sends (357.5, 365) to (0.6, 8). - Try this: drag the pixel up toward row 240 and watch z run away.

The floor is not special. The wall z = 4 has its own matrix the same way, with e_x, e_y and the wall point (0, 0, 4):

• H_wall,1 = [[500, 0, 1280], [0, 500, 960], [0, 0, 4]], and it sends the wall point (0.3, −0.2), which is X, to (357.5, 215).

NOTE: our H_floor,1 has H₃₃ = 0, because the floor's origin o sits on camera 1's principal plane (it has depth 0). OpenCV's findHomography scales its answer so that H₃₃ = 1 whenever H₃₃ is not zero, and with H₃₃ = 0 that scaling has nothing to divide by. Measure floor z from 3 m, say, and the third column becomes K times a point at depth 3: H₃₃ is no longer 0 and the problem goes away.

Try this: press "above the horizon", then "camera 2 m up". The higher camera sends the same pixel farther out, in proportion to its height.

4.1 Four Points Fix It

How many numbers does a homography really have? Nine entries, but any multiple of H is the same map, exactly like a pixel's triple (H&Z Ch. 2 Projective Geometry and Transformations of 2D, §2.3, pp. 32–36):

• Each point pair, a floor point and its pixel, gives two equations, one for u and one for v.
• So four pairs, no three of them on one line, fix H. Four tape marks on a floor are enough to map every floor pixel.

Where it breaks. - With exactly four pairs nothing is left over to check them: a misplaced tape mark gives an exact H for the wrong floor (H&Z Ch. 4 Estimation – 2D Projective Transformations, the opening of the chapter, p. 88). Use more pairs and least squares, and RANSAC to throw out bad ones (H&Z Ch. 4 Estimation – 2D Projective Transformations, §4.7, pp. 116–122).

Compare that with the full camera: P has 11 numbers to find and a homography has 8, which is why a flat target is so much easier to pin down. Finding the Camera From Known Points — P by Least Squares shows how to estimate both from points.

4.2 What Survives a Homography

Straight lines survive; almost nothing else does. Table 2.1 of the book (H&Z Ch. 2 Projective Geometry and Transformations of 2D, p. 44) sorts the maps of a plane by how much they keep:

Table. 1 What each kind of plane map keeps; each row also keeps everything kept by the rows below it. After table 2.1 (H&Z Ch. 2 *Projective Geometry and Transformations of 2D*, p. 44) of [HART04].

A photo of a floor is projective, which is why right angles on the floor are not right angles in the picture, and why parallel rails meet.

4.3 Where I Met This

• Elderlab, York University: "Designed unsupervised lane-assignment algorithm via backprojection on ground plane by simulating the oriented bounding boxes around vehicles in real-time."
• Cloudastructure: "Built a camera mapping framework leveraging SfM, photogrammetry, MCMO tracking, and ground-plane alignment".

Both rest on the pixel → floor map of this section.

5. In OpenCV

Table. 2 This read in OpenCV 4.x [OPEN24]: calib3d for findHomography, imgproc and core for the rest.

• findHomography then polishes H by Levenberg–Marquardt (least squares on the pixel error); the documentation says for every method, but the 4.x source skips it for RHO and for exactly four pairs. RANSAC, LMEDS and RHO try many random four-pair subsets so a few bad matches cannot spoil H.
• warpPerspective expects the map from the source picture to the output and inverts it itself; pass the WARP_INVERSE_MAP flag if you already hold the output → source map.

NOTE: perspectiveTransform divides by the last number whenever it is not 0, and does not look at its sign. Give it a pixel above the horizon and H⁻¹ and it returns a "floor point" on the far side of the camera. Check that w > 0 before you trust a floor point.

6. What Came Next

All three reads so far were handed P. Real cameras do not come with one. Hand me a set of points whose place I know, and their pixels in a camera I have never seen, and I'll hand you back its P. Turning those points and their pixels into P, and P back into K, R and C, is Finding the Camera From Known Points — P by Least Squares. Stay tuned.

7. References

1. [HART04] R. Hartley, A. Zisserman, Multiple View Geometry in Computer Vision, 2nd ed., Cambridge University Press, 2004. Chapters used in this read: Ch. 2 Projective Geometry and Transformations of 2D; Ch. 4 Estimation – 2D Projective Transformations; Ch. 6 Camera Models; Ch. 7 Computation of the Camera Matrix P; Ch. 8 More Single View Geometry; Ch. 13 Scene Planes and Homographies.
2. [OPEN24] OpenCV, Camera Calibration and 3D Reconstruction (calib3d), OpenCV 4.x documentation.