4. Camera calibration solution

1. Homography matrix

Before learning the camera calibration solution, first understand a concept: homography matrix, which is defined as a projection mapping from one plane to another plane, so the mapping between the object plane in the world coordinate system and the imaging plane can be understood is a homography relationship. We know that camera calibration generally uses a calibration board for calibration, so the plane in the world coordinate system is the plane of the calibration board.

Assuming a point on the calibration plate plane $Q=[X,Y,Z,1]^{T}$ , the point mapped to the imaging plane is $q=[x,y,1]^{T}$ , and the homography relationship is satisfied between the two: $q=s\cdot H\cdot Q$ , where H is the homography matrix, s is the scale factor, and the homography matrix has 8 parameters, namely 8 parameters need to be solved, so 8 equations are needed, and 8 equations can be obtained through 4 points, so at least 4 points are needed to solve the homography relationship.

opencv provides an interface: fingHomography is used to solve the homography matrix.

2. Basic equations

2.1 Basic variables

Define the point in the pixel coordinate system $m=[u,v]^{T}$ , which corresponds to the three-dimensional space point in the world coordinate system $M= [X,Y,Z]^{T}$ . For the convenience of calculation, add a 1 to get the augmented vector $\widetilde{m}=[u,v,1]^{T}$ . $\widetilde{M}=[X,Y,Z,1]^{T}$ According to the small hole imaging principle mentioned in the previous chapter, the relationship between the three-dimensional point and the projected point can be Described as: $s\widetilde{m}=A[R,t]\widetilde{M}$ , where s is the scale factor, and [R,t] is the external parameter rotation and translation matrix. A is the internal reference:

$A = \begin{bmatrix} f_{x} &\gamma &u_{0} \\ 0& f_{y} &v_{0} \\ 0&0 &1 \end{bmatrix}$

$\gamma$ It is the inclination angle of the two coordinate systems, which is not mentioned above, because the two coordinate axes are vertical by default, but they may not be vertical in fact.

2.2 Camera Calibration

We can assume that the z coordinate of the plane world coordinate system of the calibration board is 0, and the i-th column of R is $r_{i}$ , then we can get:

$s\begin{bmatrix} u\\v \\ 1 \end{bmatrix}=A\begin{bmatrix} r_{1} & r_{2} &r_{3} &t \end{bmatrix}\begin{bmatrix} X\\ Y\\ 0\\ 1 \end{bmatrix}=A\begin{bmatrix} r_{1} &r_{2} &t \end{bmatrix}\begin{bmatrix} X\\ Y\\ 1 \end{bmatrix}$

The homography relationship between the calibration plane and the imaging plane: $s\widetilde{m}=H\widetilde{M}$ , with $H =A\begin{bmatrix} r_{1} & r_{2} &t \end{bmatrix}$ .\

As mentioned before, we can solve the homography matrix H through multiple points on the two planes

suppose $H=\begin{bmatrix} h_{1} & h_{2} & h_{3} \end{bmatrix}$ , $\Rightarrow$ $\begin{bmatrix} h_{1} &h_{2} &h_{3} \end{bmatrix} = \lambda A\begin{bmatrix} r_{1}& r_{2} &t \end{bmatrix}$

According to the above formula can be obtained:

$\left\{\begin{matrix} h_{1}=\lambda \cdot A\cdot r_{1}\\ h_{2}=\lambda \cdot A\cdot r_{2} \\ h_{3}=\lambda \cdot A\cdot t \end{matrix}\right.$

Thus:

$\left\{\begin{matrix} r_{1}=\frac{1}{\lambda }\cdot A^{-1}\cdot h_{1} \\ r_{2}=\frac{1}{\lambda }\cdot A^{-1}\cdot h_{2} \\ t=\frac{1}{\lambda }\cdot A^{-1}\cdot h_{3} \end{matrix}\right.$

Camera imaging has two constraints:

1. The rotation vectors $r_{1},r_{2}$ represent the rotation around the x and y axes respectively, so these two vectors are orthogonal, so:

$r_{1}^{T}r_{2}=0$ , so that: $h_{1}^{T}A^{-T}A^{-1}h_{2}=0$

2. The rotation vector is a unit vector, so there are:

$r_{1}^{T}r_{1}=r_{2}^{T}r_{2}$ and $\left | \left |r_{1} \right | \right |=\left | \left |r_{2} \right | \right |$

And then get:

$h_{1}^{T}A^{-T}A^{-1}h_{1}=h_{2}^{T}A^{-T}A^{-1}h_{2}$

3. Closed solution

The two constraints of camera imaging are mentioned above.

Next define intermediate variables $B= A^{-T}AA^{-1}=\begin{bmatrix} B_{11} &B_{12} &B_{13} \\ B_{21}& B_{22} &B_{23} \\ B_{31}& B_{32} &B_{33} \end{bmatrix}=\begin{bmatrix} \frac{1}{f_{x}^{2}} & -\frac{\gamma }{f_{x}^{2}f_{y}} & \frac{v_{0}\gamma -u_{0}f_{y}}{f_{x}^{2}f_{y}}\\ -\frac{\gamma }{f_{x}^{2}f_{y}} &\frac{\gamma ^{2}}{f_{x}^{2}f_{y}^2} +\frac{1}{f_{y}^2} & -\frac{\gamma (v_{0}\gamma -u_{0}f_{y}))}{f_{x}^{2}f_{y}^2}-\frac{v_{0}}{f_{y}^2}\\ \frac{v_{0}\gamma -u_{0}f_{y}}{f_{x}^{2}f_{y}}& -\frac{\gamma (v_{0}\gamma -u_{0}f_{y}))}{f_{x}^{2}f_{y}^2}-\frac{v_{0}}{f_{y}^2} & -\frac{(v_{0}\gamma -u_{0}f_{y}))}{f_{x}^{2}f_{y}^2}+\frac{v_{0}^2}{f_{y}^2}+1 \end{bmatrix}$

B is a symmetric matrix, so only 6 parameters need to be solved, define $b=\begin{bmatrix} B_{11} & B_{12} & B_{22} & B_{13} & B_{23} &B_{33} \end{bmatrix}$

Decompose the column vectors of the homography matrix: $h_{i}=\begin{bmatrix} h_{i1} &h_{i2} & h_{i3} \end{bmatrix}$

Substitute into the above formula to get: $h_{i}^{T}Bh_{j}=v_{ij}b$

$v_{ij}=\begin{bmatrix} h_{i1}h_{j1}& h_{i1}h_{j2}+h_{i2}h_{j1} &h_{i2}h_{j2} &h_{i3}h_{j1}+h_{i1}h_{j3} &h_{i3}h_{j2}+h_{i2}h_{j3} & h_{i3}h_{j3} \end{bmatrix}$

Substitute into the above formula to get:

$\begin{bmatrix} v_{12}^{T}\\ (v_{11}-v_{22})^{T}) \end{bmatrix}b=0$

By taking N pictures of the calibration board, the above N equations can be put together, and we can get $Vb=0$

V is a 2n*6 matrix, b has 6 parameters, and b can be calculated by solving the equation. Since b has 6 parameters, at least 3 homography matrices are required, that is to say, at least 3 different angles need to be taken A picture of the calibration board.

After calculating b, you can solve the camera internal parameters:

$v_{0}=(B_{12}B_{13}-B_{11}B_{23})/(B_{11}B_{22}-B_{12}^{2})$

$\lambda =B_{33}-[B_{13}^2+v_{0}(B_{12}B_{13}-B_{11}B_{23})]/B_{11}$

$f_{x}=\sqrt{\lambda /B_{11}}$

$f_{y}=\sqrt{\lambda B_{11}/(B_{11}B_{22}-B_{12}^2)}$

$\gamma =-B_{12}f_{x^2}f_{y}/\lambda$

$u_{0}=\gamma v_{0}/f_{y}-B_{13}f_{x}^2/\lambda$

Calculate the rotation-translation matrix: $\begin{bmatrix} h_{1} &h_{2} &h_{3} \end{bmatrix}=\lambda A\begin{bmatrix} r_{1} &r_{2} &t \end{bmatrix}$ , so that:

$r_{1}=\lambda ^{-1}\cdot A^{-1}\cdot h_{1}$

$r_{2}=\lambda ^{-1}\cdot A^{-1}\cdot h_{2}$

$r_{3}=r_{1}\times r_{2}$

$t=\lambda ^{-1}\cdot A^{-1}\cdot h_{3}$

4. Solving radial distortion parameters

Earlier we talked about the radial distortion model:

$\left\{\begin{matrix} x_{distorted} = x(1+k_{1}r^{2}+k_{2}r^{4}+k_{3}r^{6}))\\ y_{distorted} = y(1+k_{1}r^{2}+k_{2}r^{4}+k_{3}r^{6})) \end{matrix}\right.$

Among them $r^{2}=x^{2}+y^{2}$ , radial distortion generally only considers the first two items:

Therefore:

$\left\{\begin{matrix} \widehat{x}=x+x[k_{1}(x^2+y^2)+k_{2}(x^2+y^2)^2)]\\ \widehat{y}=y+y[k_{1}(x^2+y^2)+k_{2}(x^2+y^2)^2)] \end{matrix}\right.$

Wherein $(x,y)$ is the ideal coordinate of the point in the image coordinate system, that is, the coordinate without any distortion, and is the $(\widehat{x},\widehat{y})$ coordinate of the real point in the image coordinate system, that is, the coordinate after distortion.

In pixel coordinate system:

$\left\{\begin{matrix} \widehat{u}=u+(u-u_{0})[k_{1}(x^2+y^2)+k_{2}(x^2+y^2)^2)]\\ \widehat{v}=v+(v-v_{0})[k_{1}(x^2+y^2)+k_{2}(x^2+y^2)^2)] \end{matrix}\right.$

Wherein $(u,v)$ is the ideal coordinate of the point in the pixel coordinate system, that is, the coordinate without any distortion, and is the $(\widehat{u},\widehat{v})$ coordinate of the real point in the pixel coordinate system, that is, the coordinate after distortion.

The above equations are expressed in matrix form:

$\begin{bmatrix} (u-u_{0}(x^2+y^2) &(u-u_{0}(x^2+y^2)^2 \\ (v-v_{0}(x^2+y^2) & (v-v_{0}(x^2+y^2)^2 \end{bmatrix}\begin{bmatrix} k_{1}\\k_{2} \end{bmatrix}=\begin{bmatrix} \widehat{u}-u\\ \widehat{v}-v \end{bmatrix}$

Among them, ( $u,v$ ) can be obtained by solving the internal and external parameters of the camera, and $(\widehat{u},\widehat{v})$ the corner points can be obtained through image recognition, and the same is true for (x, y).

Assuming that n pictures of the calibration board are taken, and each picture has m corner points, $2mn$ an equation can be obtained and written in matrix form:

$DK=d$ ,in $K=\begin{bmatrix} k_{1} &k_{2} \end{bmatrix}^{T}$

Solve the equations according to the least square method (minimize the root mean square error to find the best match of the data, which is used to solve the situation where the number of equations is more than the unknown), and the distortion coefficient can be solved according to the above equation.