[Stereo Vision (3)] Principles of Zhang Zhengyou Calibration Method

This is a personal study note. I have borrowed a lot of good pictures and articles (references are at the end of the article), mainly to sort out relevant knowledge in order to form my own system. The text expression is pieced together, and the symbolic formula is manually entered. Please point out if there are any mistakes.

1. Camera Calibration

The so-called calibration (calibration) is the process of fitting a parameter model by a large number of observations, and the parameter model fitted here is known, so it is necessary to explore a scheme that can easily obtain a large number of observations as much as possible. It is more helpful to solve the specific single parameter value if it also satisfies some other geometric constraints.
Camera calibration, the purpose is to determine the internal parameter matrix $K$ , external parameter matrix$R,T$ and distortion coefficients$k_1,k_2,k_3,p_1,p_2$。

Zhang's calibration method provides a convenient solution to obtain a large number of observations, and at the same time, the observations also meet a type of obvious geometric constraints (ie, plane constraints), which can directly solve the internal and external parameters. Its operation method is very simple. You only need to shoot the plane with the calibration plate pattern to complete the camera calibration, which greatly reduces the difficulty of calibration. If you do not pursue high precision, print a checkerboard calibration plate pattern and paste it on an approximately flat hard surface. Calibration can be completed on cardboard, which speeds up the introduction and popularization of stereo vision, and has far-reaching influence. It is an absolute classic in the field of camera calibration.
Leaving aside the distortion coefficient for now, the known image coordinates $p$ and world coordinates$P_w$Establish a projection relationship between internal and external parameters (projection matrix):

$$p\_=K\left[\begin{array}{cc}{R}_{3\times 3}& T_{3\times 1}\end{array}\right]\left[\begin{array}{c}{P}_w\\ 1\end{array}\right]=M\left[\begin{array}{c}{P}_w\\ 1\end{array}\right]$$

for $r_i$Denotes the rotation matrix $R$ 's$column\; i$ , and use$t$ instead of$T_{3\times 1}$, assuming that the model plane is in the world coordinate system $W=On\; the\; 0$ plane (unified symbols, the world coordinate system hereis consistentthe previous expression$P_w(U,V,W)$ ), then

$$p\_=K\left[\begin{array}{cccc}{r}_1& r_2& r_3& t\end{array}\right]\left[\begin{array}{c}U\\ V\\ 0\\ 1\end{array}\right]=K\left[\begin{array}{ccc}{r}_1& r_2& t\end{array}\right]\left[\begin{array}{c}U\\ V\\ 1\end{array}\right]$$

As mentioned earlier , points on the world coordinate plane are related to points on their image via a homography matrix:

$$p\_=HP$$
equation,$p=[u,v,1{]}^T$，$P=[U,V,1]$，$H=R[r_1,,r_2,t]$ is$3\times 3$ matrix. still use$h_i$to represent HHiiin$H$$column\; i$ . have

$$[h_1,h_2,h_3]=\Lambda K[r_1,r_2,t]$$
where$\Lambda$ is any scalar ($The\; existence\; of\; \Lambda$ is due to the scale invariance of homogeneous coordinates, it can also be considered equal to 1, let it be 1 here, and ignore it later). $r_1$and $r_2$is the rotation matrix $The\; column\; components\; of\; R$ , which are a pair of orthonormal basis. satisfy$r_1^T{r}_2=0,r_1^T{r}_1=r_2^T{r}_2=1$ . Contact the above formula:
$$\begin{gathered} r_1=K^{-1}{h}_1 \\ {r}_2=K^{-1}{h}_2 \end{gathered}$$
So:
$$h_1^T{K}^{-T}{K}^{-1}{h}_2=0$$$$h_1^T{K}^{-T}{K}^{-1}{h}_1=h_2^T{K}^{-T}{K}^{-1}{h}_2$$

It can be seen that the homography matrix $H$ and internal reference matrix$The\; elements\; of\; K$ satisfy two linear equation constraints. The homography has 8 degrees of freedom and with 6 extrinsics (3 for rotation and 3 for translation), we can only obtain 2 constraints on the intrinsics.

2. Parameter solution

According to the structure of the paper, it starts from the analytical solution, then proceeds to the nonlinear optimization technique based on the maximum likelihood criterion, and finally considers the camera distortion.

1) Closed solution

Similarly, replace the middle part of the above equation with a simple matrix (a common operation), let
$$B=K^{-T}{K}^{-1}=\left[\begin{array}{ccc}{B}_{11}& B_{12}& B_{13}\\ B_{12}& B_{22}& B_{23}\\ B_{13}& B_{23}& B_{33}\end{array}\right]$$
具体地：
$$B=\left[\begin{array}{ccc}\frac{1}{f_x^2}& -\frac{s}{f_x^2{f}_y}& \frac{v_{0}s-u_0{f}_y}{f_x^2{f}_y}\\ -\frac{s}{f_x^2{f}_y}& \frac{s^2}{f_x^2{f}_y^2}+\frac{1}{f_y^2}& -\frac{s(v_{0}s-u_0{f}_y}{f_x^2{f}_y^2)}-\frac{v_0}{f_y^2}\\ \frac{v_{0}s-u_0{f}_y}{f_x^2{f}_y}& -\frac{s(v_{0}s-u_0{f}_y)}{f_x^2{f}_y^2}-\frac{v_0}{f_y^2}& \frac{(v_{0}s-u_0{f}_y{)}^2}{f_x^2{f}_y^2}+\frac{v_0^2}{f_y^2}+1\end{array}\right]$$

HH has been set above$H$ 's$i$ column vectors are$h_i=[h_i,h_{i2},h_{and}{]}^T$ 。则有
$$\begin{gathered} h_i^{T}B{h}_j=\left[\begin{array}{ccc}{h}_i& h_{i2}& h_{and}\end{array}\right]\left[\begin{array}{ccc}{B}_{11}& B_{12}& B_{13}\\ B_{12}& B_{22}& B_{23}\\ B_{13}& B_{23}& B_{33}\end{array}\right]\left[\begin{array}{c}{h}_j\\ h_j\\ h_{j3}\end{array}\right] \\ =\left[\begin{array}{ccc}{h}_i{B}_{11}+h_{i2}{B}_{12}+h_{and}{B}_{13}& h_i{B}_{12}+h_{i2}{B}_{22}+h_{and}{B}_{23}& h_i{B}_{13}+h_{i2}{B}_{23}+h_{and}{B}_{33}\end{array}\right]\left[\begin{array}{c}{h}_j\\ h_j\\ h_{j3}\end{array}\right] \end{gathered}$$ $$=h_i{h}_j{B}_{11}+h_{i2}{h}_j{B}_{12}+h_{and}{h}_j{B}_{13}+h_i{h}_j{B}_{12}+h_{i2}{h}_j{B}_{22}+h_{and}{h}_j{B}_{23}+h_i{h}_{j3}{B}_{13}+h_{i2}{h}_{j3}{B}_{23}+h_{and}{h}_{j3}{B}_{33}$$ $$=\left[\begin{array}{cccccc}{h}_i{h}_j& h_i{h}_j+h_{i2}{h}_j& h_{i2}{h}_j& h_{and}{h}_j+h_i{h}_{j3}& h_{and}{h}_j+h_{i2}{h}_{j3}& h_{and}{h}_{j3}\end{array}\right]\left[\begin{array}{c}{B}_{11}\\ B_{12}\\ B_{22}\\ B_{13}\\ B_{23}\\ B_{33}\end{array}\right]$$

make

$$b=[B_{11},B_{12},B_{22},B_{13},B_{23},B_{33}{]}^T$$$$v_{ij}=[h_i{h}_j,h_i{h}_j+h_{i2}{h}_j,h_{i2}{h}_j,h_{and}{h}_j+h_i{h}_{j3},h_{and}{h}_j+h_{i2}{h}_{j3},h_{and}{h}_{j3}{]}^T$$
则有：
$$h_i^{T}B{h}_j=v_{ij}^{T}b$$

(Note: The original paper directly gave the above formula, and I was stunned to find out that this formula is not derived, but a coefficient that is calculated first and then extracted, so here I choose to calculate it again, which is helpful for understanding. )

$$h_i^{T}B{h}_j=v_{ij}^{T}b$$
and then contact the above homography matrix$H$ and internal reference matrix$The\; elements\; of\; K$ satisfy two linear equation constraints, and there are

$$\left[\begin{array}{c}{v}_{11}^T\\ (v_{11}-v_{22}{)}^T\end{array}\right]b=0$$
After the camera shoots the calibration plate pattern in one pose, after extracting the pixel coordinates of the corner points, the corresponding relationship between the world coordinate system and the pixel coordinate system of all corner points can be obtained, and then through the least squares solution of the linear equationsHHunder the current pose$H$ , can get the above formula. But it only has two lines, used to solve the 6-dimensional$The\; b$ vector requires at least 3 homography matrices, that is, at least 3 pictures are required to complete the camera calibration. The total equation can be expressed as:
$$Vb=0$$

• If the number of pictures is $n\ge 3$ , usually can get$A\; unique\; solution\; to\; b$ (due to scale equivalence, the resulting$Any\; multiple\; of\; b$ is still the correct solution).
• If the number of pictures is $n=2$ , you can impose unbiased constraints$s=0$ , put$[0,1,0,0,0,0]b$ is added to the above equation as an additional equation.
• If the number of pictures is $n=2$ , it can be assumed that$u_{、}{v}_0,s$ are known (for example, they are all 0), and$f_x、f_y$Solve it.

Solve for $After\; b$ , all camera intrinsic parameters can be calculated as follows (since bybbThe matrix BBcomposed of$b$$B$ does not strictly satisfy$B=K^{-T}{K}^{-1}$ , but there is an arbitrary scaling factor$\lambda$ (again) satisfies$B=\lambda K^{-T}{K}^{-1}$）：

$$v_0=\frac{B_{12}{B}_{13}-B_{11}{B}_{23}}{B_{11}{B}_{22}-B_{22}^2}$$
$$l=B_{33}-\frac{B_{13}^2+v_0(B_{12}{B}_{13}-B_{11}{B}_{23})}{B_{11}}$$
$$f_x=\sqrt{\frac{}{B_{11}}}$$
$$f_y=\sqrt{\frac{\lambda B_{11}}{B_{11}{B}_{22}-B_{22}^2}}$$

$$s=-\frac{B_{12}{f}_x^2{f}_y}{l}$$
$$u_0=\frac{s{v}_0}{f_x}-\frac{B_{13}{f}_x^2}{l}$$

When the internal parameter matrix $After\; K$ is solved, the external parameter matrix R, TR, Tof each pose$R,T$ can be further obtained:
$$\begin{gathered} r_1=\lambda K^{-1}{h}_1 \\ {r}_2=\lambda K^{-1}{h}_2 \\ {r}_3=r_1\times r_2 \\ t=\lambda K^{-1}{h}_3 \end{gathered}$$
where, $l=\frac{1}{||K^{-1}{h}_1||}=\frac{1}{||K^{-1}{h}_2||}$. Due to the presence of noise in the data, the calculated matrix $R$ usually does not satisfy the nature of the rotation matrix, and the optimal rotation matrix can be obtained by singular value decomposition.

2) Maximum likelihood solution

The above solution is obtained by minimizing an algebraic distance that has no physical meaning. Due to the existence of noise, the solution will not be very accurate. We can obtain a more accurate solution by means of maximum likelihood estimation. nn
for a given calibration plate plane$n$ images with$m$ points. It is assumed that the noise of image points is independent and identically distributed. The maximum likelihood estimate can be obtained by minimizing the following function:

$$\sum _{i=1}^n\sum _{j=1}^m||p_{ij}-\hat{p}(K,R_i,t_i,P_j)|{|}^2$$

其中 $\hat{p}(K,R_i,t_i,P_j)$ is the spatial point$P_j$in image iiProjection point on$i\; .$Rotation matrix$R$ can be composed of three vectors$r$ means that$r$ is parallel to the axis of rotation and its magnitude (mode length) is equal to the angle of rotation. $R$ and$r$ through the Rodrigues (Rodrigues) formula linked. The minimization of the above formula is a nonlinear minimization problem, which can be solved by the Levenberg-Marquardt algorithm. This type of problem requires a more accurate initial value, and the closed solution mentioned above can be used as the initial value.

3) Consider camera distortion

As mentioned above , three radial distortion parameters $k_1、k_2、k_3$and two tangential distortion parameters $p_1、p_2$, in Zhang's calibration, only two radial distortion parameters $k_1、k_2$. More terms will be considered in practical application, the principle is the same.
Accurate undistorted coordinates cannot be calculated when the internal and external parameters are unknown (the observations are always in error), but the distortion parameters are not considered when estimating the internal and external parameters (biting the tail). It doesn't matter, the concept theory will take action: use the internal and external parameters obtained by the closed solution as the initial value, find the approximate ideal coordinates, and then establish a linear equation system according to the distortion correction formula to solve the approximate k 1 , k 2 k_1, $k_1、k_2$As an initial value for the following maximum likelihood estimation:
$$\sum _{i=1}^n\sum _{j=1}^m||p_{ij}-\hat{p}(K,k_1,k_2,R_i,t_i,P_j)|{|}^2$$
Solve all internal and external parameters and distortion coefficients by nonlinear solution method.
In fact, because the value of the distortion coefficient is very small, it is also possible to directly set all the initial values ​​of the distortion coefficient to 0, which does not need to solve the linear equation system.

3. Experimental process

Summarize Zhang's calibration process:

1. Print the calibration pattern and paste it on a flat surface, called the calibration plate.
2. Take multiple images of the calibration board in different poses by moving the camera or moving the calibration board (number of images >=3).
3. Detect feature points (corner points or circle center points) on all images.
4. Solve for all intrinsic and extrinsic parameters using the closed solution method.
5. Calculate accurate internal and external parameters and distortion coefficients through nonlinear optimization (the initial value of the distortion coefficient can be solved by the distortion correction linear equations or directly assigned to 0).

Calibration board used in the paper

Other Types of Calibration Plates

It’s superficial on paper, and I’ll consider getting started and practicing it later.

---

Reference:
[1] Zhang Z . A Flexible New Technique for Camera Calibration[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2000, 22(11):1330-1334. [2]
Stereo Vision Getting Started Guide (3): Zhang-style calibration method for camera calibration
[3] Calibration Patterns Explained