## Reconstruction of 3D points from images in a calibrated binocular system

In a binocular system one can write

$\begin{bmatrix} u \\ v \\ w \end{bmatrix} \propto P \begin{bmatrix} X \\ Y \\ Z \\ W \end{bmatrix}$

and

$\begin{bmatrix} u' \\ v' \\ w' \end{bmatrix} \propto P' \begin{bmatrix} X \\ Y \\ Z \\ W \end{bmatrix}$

where

$\begin{bmatrix} X \\ Y \\ Z \\ W \end{bmatrix}$ is an object point in homogeneous coordinates in mm (*),

$\begin{bmatrix} u \\ v \\ w \end{bmatrix}$ and $\begin{bmatrix} u' \\ v' \\ w' \end{bmatrix}$ are its image points in homogeneous coordinates in pixels,

$P=K \begin{bmatrix} R & \mathbf{t} \end{bmatrix}$ and $P'=K' \begin{bmatrix} R' & \mathbf{t}' \end{bmatrix}$ are the projection matrices of the two cameras,

$K$ and $K'$ are the calibration matrices of the two cameras,

$\begin{bmatrix} R & \mathbf{t} \end{bmatrix}$ and $\begin{bmatrix} R' & \mathbf{t}' \end{bmatrix}$ are the rototranslations from a global reference system to the standard reference systems of the two cameras.

If the system is calibrated, $P$ and $P'$ are known up to scale; if the points are not at infinity, their euclidean coordinates $\frac{X}{W}, \frac{Y}{W}, \frac{Z}{W}$ can be obtained by DLT , in complete analogy with what described here for homographies:

$\dfrac{X}{W} (P_{31} \dfrac{u}{w} - P_{11}) + \dfrac{Y}{W}(P_{32} \dfrac{u}{w}-P_{12}) + \dfrac{Z}{W}(P_{33} \dfrac{u}{w} - P_{13})=P_{14}-P_{34} \dfrac{u}{w}$

$\dfrac{X}{W} (P_{31} \dfrac{v}{w} - P_{21}) + \dfrac{Y}{W}(P_{32} \dfrac{v}{w}-P_{22}) + \dfrac{Z}{W}(P_{33} \dfrac{v}{w} - P_{23})=P_{24}-P_{34} \dfrac{v}{w}$

$\dfrac{X}{W} (P'_{31} \dfrac{u'}{w'} - P'_{11}) + \dfrac{Y}{W}(P'_{32} \dfrac{u'}{w'}-P'_{12}) + \dfrac{Z}{W}(P'_{33} \dfrac{u'}{w'} - P'_{13})=P'_{14}-P'_{34} \dfrac{u'}{w'}$

$\dfrac{X}{W} (P'_{31} \dfrac{v'}{w'} - P'_{21}) + \dfrac{Y}{W}(P'_{32} \dfrac{v'}{w'}-P'_{22}) + \dfrac{Z}{W}(P'_{33} \dfrac{v'}{w'} - P'_{23})=P'_{24}-P'_{34} \dfrac{v'}{w'}$

This is a inhomogeneous overdetermined linear system of four equations in the three unknowns $\dfrac{X}{W}$, $\dfrac{Y}{W}$ and $\dfrac{Z}{W}$, and can be solved eg by singular value decomposition, but its closure has no apparent geometric meaning. As ususal, a method minimizing some sort of geometric error would be preferred.

The traditional way to reconstruct a 3D point with geometric optimization, as described eg by Hartley & Zisserman, Multiple View Geometry in Computer Vision, §12.3, is to find the minimum displacements of the reprojected image points on the image planes, so that the corrected image points do satisfy the epipolar constraint. This is a nonlinear problem whose solution is not straightforward.

As an alternative, one can use here the same concept underlying forward bundle adjustment: that is, minimize a distance in object space instead of image space, as follows:

The optical centre $\mathbf{C}= \begin{bmatrix} \tilde{\mathbf{C}} \\ 1 \end{bmatrix}$ is a null vector of the projection matrix: $P \mathbf{C} = \begin{bmatrix} M \mathbf{P}_{.4} \end{bmatrix} \begin{bmatrix} \tilde{\mathbf{C}} \\ 1 \end{bmatrix}= \mathbf{0}$, whence $\tilde{\mathbf{C}} = -M^{-1} \mathbf{P}_{.4}$; the direction of the optical ray $\tilde{\mathbf{D}}$ is the projection of the image point onto the plane at infinity: $P \mathbf{D} = \begin{bmatrix} M \mathbf{P}_{.4} \end{bmatrix} \begin{bmatrix} \tilde{\mathbf{D}} \\ 0 \end{bmatrix} = \mathbf{x}$, whence $\tilde{\mathbf{D}}=M^{-1} \mathbf{x}$.

The parametric equations of the optical rays are

$\tilde{\mathbf{L}}(\tau)=\tilde{\mathbf{C}} + \tau \tilde{\mathbf{D}}$

and

$\tilde{\mathbf{L}}'(\tau')=\tilde{\mathbf{C}}'+ \tau' \tilde{\mathbf{D}}'$

One wants to find a point $\tilde{\mathbf{X}}=\begin{bmatrix} X \\ Y \\ Z \end{bmatrix}$ and values of $\tau, \tau'$ such that the sum of squared distances $S=(\tilde{\mathbf{L}}(\tau) - \tilde{\mathbf{X}})^2+(\tilde{\mathbf{L}}'(\tau')-\tilde{\mathbf{X}})^2$ be minimum. By equating to zero the derivatives of $S$ with respect to $X, Y, Z, \tau, \tau'$ one gets a linear system of five equations in five unknowns, whose solution is straightforward.

This method is readily applied to the case of trinocular systems too, giving a linear system of six equations in six unknowns.

———-

(*) $\dfrac{X}{W}$, $\dfrac{Y}{W}$, $\dfrac{Z}{W}$ are mm.