Understand the principle of camera calibration in simple terms

Important note: This article is compiled from online information. It only records the blogger's process of learning relevant knowledge points and has not been deleted.

1. Reference materials

WeChat public account: Computer Vision Life Column: #cameracalibration

Camera Calibration
Camera Calibration and 3D Reconstruction
Zhang Zhengyou Calibration Method - Complete Study Notes - From Principle to Practical Practice
The principles of camera calibration and the relationship between the four coordinate systems

2. Introduction to Camera Calibration

1. Perspective projection

Using the central projection method, the object is projected onto the projection surface, thereby obtaining a single-sided projection image that is closer to the visual effect, which is an imaging method where we see the scenery near and far .

2. The concept of camera calibration

Simply put, camera calibration is the process of converting from the world coordinate system to the pixel coordinate system. This process includes two stages: establishing a camera imaging geometric model and correcting lens distortion.

Establish a camera imaging geometric model : Establish a geometric model in the process of mapping an object from the three-dimensional world to the camera imaging plane. The most critical part is to get the internal and external parameters of the camera .

Correction of lens distortion : Due to the lens manufacturing process, various forms of distortion occur. In order to remove distortion, the distortion coefficient is calculated and used to correct the aberration so that the image after imaging is consistent with the real-world scene.

3. Four coordinate systems

World coordinate system : A user-defined coordinate system of the three-dimensional world, introduced to describe the position of the target object in the real world. The unit is meters and the coordinate system is $(X_w,Y_w,Z_w)$。

Camera coordinate system : The coordinate system established on the camera is defined to describe the position of the object from the perspective of the camera, and serves as an intermediate link between the world coordinate system and the image/pixel coordinate system. The unit is meters. Generally, the optical axis of the camera is the Z axis, and the coordinate system is $(X_c,Y_c,Z_c)$。

Image physical coordinate system (image coordinate system) : Introduced to describe the projection and transmission relationship of an object from the camera coordinate system to the image coordinate system during the imaging process, so as to facilitate further obtaining the coordinates in the pixel coordinate system. The unit is millimeters. The coordinate origin is the intersection point of the camera optical axis and the image physical coordinate system. The coordinate system is $(x,y)$。

Image pixel coordinate system (pixel coordinate system) : Introduced to describe the coordinates of the image point of the object on the digital image (photo). It is the coordinate system in which the information we actually read from the camera is located. The unit is pixels, the coordinate origin is at the upper left corner, and the coordinate system is $(u,v)$。

The relationship between the four coordinate systems is as shown in the figure below:

Among them, the Z axis of the camera coordinate system coincides with the optical axis, is perpendicular to the plane of the image coordinate system and passes through the origin of the image coordinate system. The distance between the camera coordinate system and the image coordinate system is the focal length f (that is, the origin of the image coordinate system and focus overlap). The pixel coordinate system plane uv coincides with the image coordinate system plane xy, but the origin of the pixel coordinate system is located in the upper left corner of the figure (the reason for this definition is to start reading and writing from the first address of stored information).

4. Conversion of the four major coordinate systems

4.1 World coordinate system vs camera coordinate system

The world coordinate system is converted to the camera coordinate system. The conversion between the two coordinate systems is a rigid body movement . The object only changes the spatial position (translation) and orientation (rotation) in the two coordinate systems without changing its shape. The conversion relationship between them can be completed by the translation matrix T and the rotation matrix R. These two matrices reflect the conversion relationship between the world coordinate system and the camera coordinate system, and are collectively called the external parameter matrix $L_w$. The external parameter matrix is ​​obtained, and a point in the world coordinate system is known. Through the conversion relationship, the position of this point in the camera coordinate system can be obtained.

The transformation matrix is ​​represented by a homogeneous coordinate system composed of a rotation matrix and a translation vector:
$$\left[\begin{array}{c}{x}_C\\ y_C\\ z_C\\ 1\end{array}\right]=\left[\begin{array}{cc}R& t\\ 0_3^T& 1\end{array}\right]\left[\begin{array}{c}{x}_W\\ y_W\\ z_W\\ 1\end{array}\right]=\left[\begin{array}{cccc}{r}_1& r_2& r_3& t\end{array}\right]\left[\begin{array}{c}{x}_W\\ y_W\\ 0\\ 1\end{array}\right]=\left[\begin{array}{ccc}{r}_1& r_2& t\end{array}\right]\left[\begin{array}{c}{x}_W\\ y_W\\ 1\end{array}\right]$$
Among them, R is the rotation matrix, and t is the translation vector. Assume that in the world coordinate system, the plane where the object point is located passes through the origin of the world coordinate system, and $Z_w=The\; 0$ axis is vertical, that is, the chessboard plane is with$X_w-Y_w$The purpose of plane coincidence is to facilitate subsequent calculations. Among them, the change matrix is:
$$\left[\begin{array}{cc}R& t\\ 0_3^T& 1\end{array}\right]$$
The following figure shows the process of converting the world coordinate system to the camera coordinate system using R and t.

4.2 Camera coordinate system vs image physical coordinate system (no distortion)

The conversion of the camera coordinate system to the image physical coordinate system is to convert the three-dimensional coordinate system into a two-dimensional coordinate system, which is a projection perspective process. As shown in the figure below, the pinhole imaging process: the pinhole surface (camera coordinate system) is between the image plane (image physical coordinate system) and the object point plane (chessboard plane), and the formed image is an inverted real image .

but,In order to make it easier to describe mathematically, the positions of the camera coordinate system and the image physical coordinate system are reversed (no actual physical meaning, just to facilitate calculation), becomes as shown in the figure below:

Assuming that there is a point M in the camera coordinate system, according to the principle of similar triangles, the coordinates of the imaging point P in the image physical coordinate system (without distortion) are: xp =
$${\mathrm{x}}_p=f\frac{x_M}{z_M},y_p=f\frac{y_M}{z_M}$$
Let us divide and conquer the equation:
$$z_M\left[\begin{array}{c}{x}_p\\ y_p\\ 1\end{array}\right]=\left[\begin{array}{cccc}f& 0& 0& 0\\ 0& f& 0& 0\\ 0& 0& 1& 0\end{array}\right]\left[\begin{array}{c}{x}_{\mathrm{M}}\\ y_M\\ z_M\\ 1\end{array}\right]=\left[\begin{array}{ccc}f& 0& 0\\ 0& f& 0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{cccc}1& 0& 0& 0\\ 0& 1& 0& 0\\ 0& 0& 1& 0\end{array}\right]\left[\begin{array}{c}{x}_M\\ y_M\\ z_M\\ 1\end{array}\right]$$

4.3 Camera coordinate system vs image physical coordinate system (with distortion)

Please refer to the chapter: Camera Distortion Model

4.4 Image physical coordinate system vs image pixel coordinate system

Take a vivid example to illustrate the difference between the image physical coordinate system and the image pixel coordinate system. The physical coordinate system is a continuous concept, and it is measured in millimeters, just like the specific coordinate value of a certain audience in a cinema is (3.4, 5.9); while the pixel coordinate system is a discrete concept, and it is measured in pixels. , before you are integer-valued coordinates, just like the position of a certain audience in a movie theater is (third row, sixth column). In addition, it should be noted that the origin positions of the two coordinate systems are not the same,The origin of the physical coordinate system is the intersection of the camera optical axis and the image physical coordinate system, usually called the main point ; while the pixel coordinate system takes the upper left corner of the pixel image as the origin.。

Since the defined origin of the image pixel coordinate system does not coincide with the origin of the image physical coordinate system, it is assumed that the coordinates of the origin of the image physical coordinate system in the image pixel coordinate system are (u 0, v 0 $(u_0,v_0)$ , the dimensions of each pixel in the x-axis and y-axis directions of the image physical coordinate system are:$d_x、d_y$, and the coordinates of the image point in the image physical coordinate system are $(x_c,y_c)$ , the coordinates of the ideal point in the image coordinate system are:
$$u=\frac{x_c}{d_x}+u_0,\mathbf{v}=\frac{y_c}{d_y}+v_0$$
The expression of the homogeneous coordinate system is:
$$\left[\begin{array}{c}u\\ n\\ 1\end{array}\right]=\left[\begin{array}{ccc}1/d_x& 0& u_0\\ 0& 1/d_y& v_0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{c}{x}_c\\ y_c\\ 1\end{array}\right]$$

5. Camera distortion model

Lens distortion is mainly divided into: radial distortion, tangential distortion, and thin lens distortion, etc., but none of them have a significant impact on radial distortion and tangential distortion, so generally only radial distortion k and tangential distortion p are considered. A total of 5 distortion parameters (k1, k2, k3, p1, and p2) are required to describe lens distortion, among which radial distortion has 3 parameters (k1, k2, k3), and tangential distortion has 2 parameters (p1, p2) .

5.1 Radial distortion

Radial distortion is caused by the manufacturing process of the lens shape. The further you move toward the edge of the lens, the more severe the radial distortion becomes. The figure below shows two types of radial distortion: barrel distortion (k<0) and pincushion distortion (k>0).

• When k>0, the larger r (the farther the point is from the center), the larger the distortion, and the smaller r, the smaller the distortion, which is a pincushion shape.
• When k<0, the larger r (the farther the point is from the center), the smaller the distortion, and the smaller r, the larger the distortion, which is barrel-shaped.

In actual situations, the first few terms of the Taylor series expansion at r=0 are often used to approximately describe radial distortion. The coordinate relationship before and after correction of radial distortion is:
$$\{\begin{array}{l}{\mathbf{x}}_{distorted}=x_p(1+k_1{r}^2+k_2{r}^4+k_3{r}^6)\\ y_{distorted}=y_p(1+k_1{r}^2+k_2{r}^4+k_3{r}^6)\end{array}$$
It can be seen that for radial distortion, there are three radial distortion parameters (k1, k2, K3) that need to be solved.

Research shows that generally choosing 2 or 3 k values ​​for radial distortion parameters can achieve better results. If k is obtained no matter how much, the impact will not be great and can be ignored, and it may also lead to poor results.

5.2 Tangential distortion

Tangential distortion is caused by the installation position error of the lens and CMOS or CCD. Therefore, if there is tangential distortion, a rectangle is likely to become a trapezoid when projected onto the imaging plane. Tangential distortion requires two additional distortion parameters (p1 and p2) to describe. The coordinate relationship before and after correction is: {
$$\{\begin{array}{l}{\mathbf{x}}_{distorted}=x_p+[2p_1{x}_p{y}_p+p_2(r^2+2x{{\_}_p}^2)]\\ y_{distorted}=y_p+[p_1(r^2+2{y_p}^2)+2p_2{x}_p{y}_p]\end{array}$$

6. Camera calibration parameters

internal reference

• Focal length: $f_x、f_y$；

• Principal point (optical center) pixel coordinates: $c_x、c_y$；

• Distortion coefficient: $k_1,k_2,k_3,p_{1,}{p}_2$；

$$camera\text{matri}x=\left[\begin{array}{ccc}{f}_x& 0& c_x\\ 0& f_y& c_y\\ 0& 0& 1\end{array}\right]$$

$$Distortioncoefficients=(k_1{k}_2{p}_1{p}_2{k}_3)$$

external reference

• rotation matrix R;
• Translation matrix T.

7. Homography matrix

Detailed explanation of the principle of Zhang Zhengyou’s calibration algorithm

7.1 Introduction

o'-uv is the image pixel coordinate system, o-XYZ is the world coordinate system. Assume that the coordinates of a point in the image are (u, v) and the corresponding three-dimensional point position is (x, y, z). Then the conversion relationship between them is as follows: [ u ν 1 ] = s [ fx γ u
$$\left[\begin{array}{c}u\\ n\\ 1\end{array}\right]=s\left[\begin{array}{ccc}{f}_x& c& u_0\\ 0& f_y& n_0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{ccc}{r}_1& r_2& t\end{array}\right]\left[\begin{array}{c}{x}_W\\ y_W\\ 1\end{array}\right]$$
Among them, s is the scale factor, which means that the rays going out from the optical center of the camera will fall on the same point on the imaging plane. u, v represent the coordinates in the image pixel coordinate system, $f_x、f_y、u_0、v_0,\gamma$ represents 5 camera internal parameters, R and t represent camera external parameters,$X_w、Y_w、Z_w$Represents the coordinates in the world coordinate system. The chessboard plane is located in the world coordinate system $Z_w=0$ plane.

7.2 The concept of homography

Homography transformation describes the positional mapping relationship between the object in the world coordinate system and the image pixel coordinate system. This transformation matrix is ​​called the homography matrix. In the above formula, the homography matrix is ​​defined as:
$$H=s\left[\begin{array}{ccc}{f}_x& c& u_0\\ 0& f_y& n_0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{ccc}{r}_1& r_2& t\end{array}\right]=sM\left[\begin{array}{ccc}{r}_1& r_2& t\end{array}\right]$$
Among them, M is the camera internal reference matrix:
$$M=\left[\begin{array}{ccc}{f}_x& c& u_0\\ 0& f_y& n_0\\ 0& 0& 1\end{array}\right]$$
The homography matrix contains both intrinsic and extrinsic parameters of the camera.

7.3 Applications of homography

7.3.1 Image Correction

Using the homography matrix for image rectification requires at least four corresponding points.

7.3.2 Perspective change

The homography matrix can easily convert the ordinary view into a bird's-eye view, as shown in the figure below, the left is the ordinary view and the right is the bird's-eye view:

7.3.3 Image stitching

Since the homography matrix can perform perspective conversion, image stitching can be realized by converting images taken from different angles to the same perspective. As shown in the figure below, both image1 and image2 can be transformed to the same plane through the homography matrix H.

7.3.4 Augmented Reality AR

Plane two-dimensional markers are often used for AR displays. Based on the images from different perspectives of the marker, the position and posture of the virtual object can be easily obtained and realized, as shown in the following figure:

7.4 The principle of homography matrix

Assume that the homogeneous coordinates of the corresponding points in the two images are $(x^{\prime },y^{\prime },1)$ , the homography matrix H is defined as:
$$H=\left[\begin{array}{ccc}{h}_{{}_{11}}& h_{{}_{12}}& h_{{}_{13}}\\ h_{{}_{21}}& h_{{}_{22}}& h_{{}_{23}}\\ h_{{}_{31}}& h_{{}_{32}}& h_{{}_{33}}\end{array}\right]$$
则有：
$$\left[\begin{array}{c}{x}^{\prime }\\ y^{\prime }\\ 1\end{array}\right]\sim \left[\begin{array}{ccc}{h}_{11}& h_{12}& h_{13}\\ h_{21}& h_{22}& h_{23}\\ h_{31}& h_{32}& h_{33}\end{array}\right]\left[\begin{array}{c}x\\ y\\ 1\end{array}\right]$$
There are 3 equations after matrix expansion. Putting the third equation into the first two equations, we can get:
$$x^{\prime }=\frac{h_{11}x+h_{12}y+h_{13}}{h_{31}x+h_{32}y+h_{33}}$$

$$y^{\prime }=\frac{h_{21}x+h_{22}y+h_{23}}{h_{31}x+h_{32}y+h_{33}}$$

That is, a point pair, corresponding to two equations.

Add constraints to H so that the H matrix modulus becomes 1, that is, $||H||=1$，如下：
$$x^{\prime }=\frac{h_{11}x+h_{12}y+h_{13}}{h_{31}x+h_{32}y+h_{33}}(1)$$

$$y^{\prime }=\frac{h_{21}x+h_{22}y+h_{23}}{h_{31}x+h_{32}y+h_{33}}(2)$$

$$h_{11}^2+h_{12}^2+h_{13}^2+h_{21}^2+h_{22}^2+h_{23}^2+h_{31}^2+h_{32}^2+h_{33}^2=1$$

Multiply the above formula (1) and (2) by the denominator to expand:
$$\left(h_{31}x+h_{32}y+h_{33}\right)x^{\prime }=h_{11}x+h_{12}y+h_{13}$$

$$\left(h_{31}x+h_{32}y+h_{33}\right)y^{\prime }=h_{21}x+h_{22}y+h_{23}$$

After finishing, we get:
$$h_{11}x+h_{12}y+h_{13}-h_{31}x{x}^{\prime }-h_{32}y{x}^{\prime }-h_{33}{x}^{\prime }=0$$

$$h_{21}x+h_{22}y+h_{23}-h_{31}x{y}^{\prime }-h_{32}y{y}^{\prime }-h_{33}{y}^{\prime }=0$$

Suppose we get N point pairs (feature point matching pairs) corresponding to the two pictures, we can get the following linear equations:

The graph is:
$$\begin{array}{cccc}2Nx9& 9x1& & 2Nx1\\ \mathbf{A}& \mathbf{h}& =& 0\end{array}$$
In real application scenarios, the calculated point pairs will contain noise. For example, the position of a point deviates by a few pixels, and even feature point pairs may be mismatched. If only four point pairs are used to calculate the homography matrix, a large error will occur. Therefore, in order to make the calculation more accurate, far larger than 4 point pairs are generally used to calculate the homography matrix. In addition, the above-mentioned linear equations adopt direct linear solution method, which is usually difficult to obtain the optimal solution. Therefore, in actual use, other optimization methods are generally used, such as singular value decomposition, Levenberg-Marquarat (LM) algorithm, etc. for solution.

7.5 Calculation process of homography matrix

The homography matrix H is calculated based on the printed chessboard calibration diagram and the taken photos. The general process is as follows:

1. Print a checkerboard calibration drawing and stick it on the surface of the flat object;

2. Taking a set of checkerboard pictures in different directions can be achieved by moving the camera or by moving the calibration image;

3. For each chessboard picture taken, the corner points of all chessboards in the picture are detected.

4. Because the world coordinates of all corner points in the chessboard calibration drawing are known, the image pixel coordinates corresponding to these corner points are also known. If there are N>=4 matching point pairs, the homography matrix H can be obtained based on optimization methods such as LM. The more matching point pairs, the more robust the calculation results will be. Calculating the homography matrix generally does not require writing a function yourself. There are ready-made functions that can be called in OpenCV. The corresponding C++ function is:

   Mat findHomography(InputArray srcPoints, InputArray dstPoints, int method=0, double ransacReprojThreshold=3, OutputArray mask=noArray() )
   

   From the function point of view, as long as the matching point pair is input and the specific calculation method is specified, the homography matrix can be obtained.

3. Zhang Zhengyou’s camera calibration method

1 Introduction

Dr. Zhang Zhengyou published a paper "Flexible Camera Calibration By Viewing a Plane From Unknown Orientations" at the top international conference ICCV in 1999, proposing a practical method for camera calibration using a planar checkerboard. This method is between the photographic calibration method and the self-calibration method. It not only overcomes the shortcomings of the high-precision three-dimensional calibration objects required by the photographic calibration method, but also solves the problem of poor robustness of the self-calibration method. The calibration process only requires a printed checkerboard and several sets of images taken from different directions. Anyone can make the calibration pattern by themselves. It is not only practical, flexible and convenient, but also has high precision and good robustness. Therefore, it was quickly widely adopted around the world, greatly promoting the process of 3D computer vision from the laboratory to the real world.

Zhang Zhengyou calibration method uses a calibration plate composed of two-dimensional squares for calibration, collects pictures of the calibration plate in different poses, extracts the pixel coordinates of the corner points in the pictures, calculates the initial values ​​of the internal and external parameters of the camera through the homography matrix, and uses nonlinear least square The distortion coefficients are estimated multiplicatively, and finally the parameters are optimized using the maximum likelihood estimation method. This method is simple to operate and has high precision, which is suitable for most occasions.

2. Introduction to Zhang Zhengyou

Dr. Zhang Zhengyou is a world-renowned expert in computer vision and multimedia technology, ACM Fellow, IEEE Fellow. He is currently a senior researcher in the Visual Computing Group of Microsoft Research. He has made seminal contributions in stereo vision, 3D reconstruction, motion analysis, image registration, camera calibration, etc.

3. Checkerboard calibration board

3.1 Introduction to checkerboard calibration board

The checkerboard calibration board is a calibration board composed of black and white square intervals, used as a calibration object for camera calibration (objects mapped from the real world to digital images). The reason for using a checkerboard as a calibration object is that the planar checkerboard pattern is easier to handle than complex 3D objects. At the same time, two-dimensional objects lack some information compared to three-dimensional objects. Images are captured by changing the orientation of the chessboard multiple times to obtain richer coordinate information. As shown in the figure below, it is an image of the same checkerboard calibration board taken by the camera in different directions.

3.2 Obtain the checkerboard calibration board

calib.io

3.2.1 Square calibration plate

Download: Grid Calibration Plate

The size of the calibration plate: w=9, h=6.

3.2.2 Circular calibration plate

Download: Circular Calibration Plate

The size of the calibration plate: w=11, h=4.

The size of the calibration plate: w=17, h=6.

3.2.3 ChAruco calibration plate

• 7X5 ChAruco board；
• square size: 30 mm ；
• marker size: 15 mm；
• aruco dict: DICT_5X5_100；
• page width: 210 mm, page height: 297 mm。
  

3.2.4 Generate checkerboard calibration board

Create calibration pattern

Use python to generate a custom calibration board, download the python script file: gen_pattern.py

3.3 Corner points

The corner point is the intersection point of the black and white checkerboard, as shown in the picture below, the magenta circle in the middle is a corner point.

As shown in the figure below, there are 8 corner points in the X direction and 3 corner points in the Y direction.

3.4 Internal corner points

The corner point refers to the fixed-point part of the square where black and white meet; the inner corner point is the inner corner point that does not touch the edge of the calibration plate.

patternSize:(w,h), the number of interior corners in each row and column on the chessboard. w=the number of black and white blocks in one row of the chessboard -1, h=the number of black and white blocks in one column of the chessboard -1, for example: a 9x4 chessboard, then (w,h)=(8,3)

3.5 Checkerboard calibration board coordinate system

The checkerboard calibration board coordinate system adopts the plane calibration mode by default, that is, the Z axis is zero.

Define the checkerboard calibration plate to be located in the world coordinate system $Z_w=On\; the\; plane\; of\; 0$ , the origin of the world coordinate system is located at a fixed corner of the checkerboard calibration plate, such as the yellow point in the picture below. The origin of the image pixel coordinate system is located in the upper left corner of the image.

When writing coordinates, make sure that the Z axis is 0, and write in the order of X changing first, then Y changing, from small to large. If the grid size of the chessboard calibration board is 5cm, the coordinates in the world coordinate system can be written as: (0,0,0), (5,0,0), (10,0,0)…(0,5, 0), (5,5,0), (10,5,0),…

4. Calculation process of Zhang Zhengyou calibration method

Using a checkerboard calibration board for calibration, you need to find the inner corners of the checkerboard. According to the coordinates of the inner corners of the checkerboard in the image pixel coordinate system and the coordinates of each point in the world coordinate system, shoot multiple positions of the checkerboard and multiple points. Solve the internal and external parameters of the camera.

Using OpenCV to perform Zhang Zhengyou calibration method, the calculation process is as follows:

1. Prepare calibration pictures, theoretically at least 4 pictures, usually about 20 pictures collected from multiple angles;
2. Extract the corner points of the calibration plate and calculate the coordinates of the corner points on the calibration plate. Generally, the calibration plate is regarded as the XY plane, and Z is 0;
3. Camera calibration, internal and external parameters and distortion parameters are calculated through Zhang Zhengyou calibration method;
4. The calibration results are evaluated, generally through reprojection error;
5. Check the calibration effect and use the calibration results to correct the checkerboard diagram.

5. Core functions

cornerSubPix()

Feature2D learning in OpenCV - sub-pixel corner detection (cornerSubPix)

OpenCV3 sub-pixel corner detection: cornerSubPix()

Function: Sub-pixel level corner point detection, accurate internal corner point position to sub-pixel level accuracy.

void cornerSubPix(InputArray image, 
                  InputOutputArray corners, 
                  Size winSize, 
                  Size zeroZone, 
                  TermCriteria criteria);

Parameter explanation

• imageIndicates the input of the original chessboard calibration board image (image matrix), which is the input parameter. The image matrix must be an 8-bit grayscale image or a color image. Before the image is passed into the function, it usually undergoes grayscale processing and filtering operations;

• cornersRepresents the interior corner point coordinates, which is both an input parameter and an output parameter. Use findChessboardCorners()the output inner corner points as the input corners of this function, and output the corners after sub-pixel level inner corner point detection;

• winSizeRepresents half the side length of the search window. For example, if winSize= Size(5, 5), then the size of a $(5\ast 2+1)\ast (5\ast 2+1)=11\ast A\; search\; window\; of\; 11$ will be used;

• zeroZoneRepresents half the side length of the dead region in the middle of the search area, sometimes used to avoid the singularity of the autocorrelation matrix. If the value is set to (-1,-1), it means there is no such area;

• criteriaIndicates the termination condition of the iterative process of corner point sub-pixel level precision. Optional values ​​are:

  • cv::TermCriteria::MAX_ITERIndicates that it will stop when the number of iterations reaches the maximum number;
  • cv::TermCriteria::EPSIndicates that the iteration stops when the corner point position change has reached the minimum value.

  Both are cv::TermCriteria()specified using constructors and can be combined.

  //指定亚像素计算迭代条件
  cv::TermCriteria criteria = cv::TermCriteria(
      cv::TermCriteria::MAX_ITER + cv::TermCriteria::EPS,  // 终止条件
      40,  // 最大次数
      0.01); // 最小值
  

findChessboardCorners()

Detailed explanation of findChessboardCorners() corner detection

Function: Corner detection, obtain the position of the corner points in the checkerboard calibration board .

Return value: All corner points are detected and arranged in a certain order, and a non-zero number is returned, otherwise 0 is returned.

// 函数原型
findChessboardCorners(InputArray image,
                      Size patternSize,
                      OutputArray corners,
                      int flags);

Parameter explanation

• imageIndicates the input of the original chessboard calibration board image (image matrix), which is the input parameter. The image matrix must be an 8-bit grayscale image or a color image. Before the image is passed into the function, it usually undergoes grayscale processing and filtering operations;

• patternSizeThe size representing the interior corner point is an input parameter.

  Data type: Size patternSize(w,h), w and h respectively represent the number of interior angles in each row and column on the chessboard.

  w=the total number of black and white blocks in one row of the checkerboard calibration board-1, h=the total number of black and white blocks in one column of the checkerboard calibration board-1。

  For example: 10x6 checkerboard calibration board, then (w, h) = (9, 5).

  Note : Determine the direction of the calibration plate according to the size of the inner corner point. Specifically, if w and h are different, this function can identify the direction of the calibration plate; if w and h are the same, the starting position of the corner point drawn by this function will change each time, which is not conducive to calibration.。

• cornersOutput array representing the detected interior corner points, output parameters.

  Data type: vector<vector<point2f>>.

• flagsThe flag is an input parameter with a default value.

  • CV_CALIB_CB_ADAPTIVE_THRESH: The default mode of the function, using the adaptive threshold method to binarize the image instead of using a fixed threshold;
  • CV_CALIB_CB_NORMALIZE_IMAGE: Before using the fixed threshold or adaptive threshold method to binarize the image, call the EqualizeHist() function to normalize the image and use the histogram to equalize the image;
  • CV_CALIB_CB_FILTER_QUADS: After the binarization is completed, the function starts to locate the quadrilateral in the image (it should not be called a square here because there is distortion). Use additional criteria (such as contour area, perimeter, square shape) to filter out false quadrilaterals extracted during the contour retrieval stage, thereby making corner detection more accurate and rigorous;
  • CALIB_CB_FAST_CHECK: Quick detection option, a quick inspection mechanism is run on the image to find the corners of the checkerboard. For the case where the detection of corner points is likely to be unsuccessful, this flag can improve the efficiency of the function.

  Notice: Flag bits can be used in combination. CALIB_CB_FAST_CHECKGenerally used for rapid detection, it is very likely that the detection will be unsuccessful, especially when the light on the board is uneven. Overall, CV_CALIB_CB_ADAPTIVE_THRESHthe most likely way to detect checkerboards. It is recommended to use the combination CV_CALIB_CB_ADAPTIVE_THRESH|CV_CALIB_CB_NORMALIZE_IMAGE. If the corner points cannot be detected in the default method or this combination method, you basically need to re-acquire the image.

The function of this function is to determine whether the image contains a complete checkerboard. If it can be completely detected, record the corner position sequence (from left to right, top to bottom) and return a non-zero number, otherwise return 0 . The parameter requirements here patternSizeare very strict. The function must detect the same size before returning non-0, otherwise it returns 0.

The coordinates of the corner points detected by this function are inaccurate. To obtain the precise coordinates of the corner points, you can use cornerSubPix()the function to extract sub-pixels of the corner points. For usage methods, please refer to the blog: Corner Point Detection and Optimization Function Usage

drawChessboardCorners()

Function: Draw the detected corner points in the original image.

calibrateCamera()

calibrateCamera of OpenCV function usage

Function: Camera calibration.

double calibrateCamera(InputArrayOfArrays objectPoints, 
                       InputArrayOfArrays imagePoints,
                       Size imageSize, 
                       InputOutputArray cameraMatrix, 
                       InputOutputArray distCoeffs, 
                       OutputArrayOfArrays rvecs, 
                       OutputArrayOfArrays tvecs, 
                       int flags=0);

Parameter explanation

• objectPointsRepresents points in the world coordinate system, that is, 3D points.

  The data type is: std::vector<std::vector<cv::Point3f>> objectPoints, the first layer of vector represents each perspective (each picture), and the second layer of vector represents each point.

• imagePoints Represents the points in the corresponding image pixel coordinate system, that is, 2D points. This value can findChessboardCorners()be obtained from the image with the function.

  The data type is: std::vector<std::vector<cv::Point2f>> imagePoints, the first layer of vector represents each angle of view, and the second layer of vector represents each interior corner point.

• imageSizeIndicates the size of the image, which is needed to calculate the internal parameters and distortion matrix of the camera;

• cameraMatrixRepresents the camera internal parameter matrix, the size of which is 3x3. This matrix is ​​the result of camera calibration output.

  The data type is: cv::Mat cameraMatrix;

• distCoeffsRepresents the distortion matrix, the specific size depends on the parameters flags. This matrix is ​​the result of camera calibration output.

  Data type: cv::Mat distCoeffs;

• rvecsrepresents the rotation vector. Each vector<Point3f>will get an rvecs, each vec is 3x1, which can be Rodrigues()converted to a 3x3 rotation matrix using the function.

  Data type: vector<cv::Mat> rvecs;

• tvecsRepresents the translation vector, same rvecsas , each vec is 3x1.

  Data type: vector<cv::Mat> tvecs;

  The parameters rvecs and tvecs are camera external parameters. For each view, its world coordinate system can be converted into the camera coordinate system.。

• flagsIndicates the parameters used during calibration, including deciding whether to use the initial value, the number of parameters of the distortion matrix, etc. There are the following parameters:

  • CV_CALIB_USE_INTRINSIC_GUESS: When using this parameter, there should be estimated values ​​of fx, fy, cx, cy in the cameraMatrix matrix. Otherwise, the center point of the image (cx, cy) will be initialized, and fx, fy will be estimated using least squares;
  • CV_CALIB_FIX_PRINCIPAL_POINT: The optical axis point is fixed when optimizing. When the CV_CALIB_USE_INTRINSIC_GUESS parameter is set, the optical axis point will remain at the center or an input value;
  • CV_CALIB_FIX_ASPECT_RATIO: Fix the ratio of fx/fy, and only use fy as a variable variable for optimization calculation. When CV_CALIB_USE_INTRINSIC_GUESS is not set, fx and fy will be ignored. Only the fx/fy ratio is used in the calculation;
  • CV_CALIB_ZERO_TANGENT_DIST: Set the tangential distortion parameters (p1, p2) to zero;
  • CV_CALIB_FIX_K1,…,CV_CALIB_FIX_K6: The corresponding radial distortion remains unchanged during optimization;
  • CV_CALIB_RATIONAL_MODEL: Calculate the three distortion parameters of k4, k5 and k6. If not set, only the other 5 distortion parameters are calculated.

6. Sample code

Use OpenCV to implement Zhang Zhengyou’s calibration process.

6.1 C++ implementation

Zhang Zhengyou Visual Calibration Algorithm Study Notes

Example 1

Calibrate multiple views.

#include <iostream>
#include <opencv2/opencv.hpp>
#include  <boost/filesystem.hpp>

std::vector<std::string> get_all_image_file(std::string image_folder_path){
    boost::filesystem::path dirpath = image_folder_path;
    boost::filesystem::directory_iterator end;
    std::vector<std::string> files;
    for (boost::filesystem::directory_iterator iter(dirpath); iter != end; iter++)
    {
        boost::filesystem::path p = *iter;
        files.push_back(dirpath.string()+ "/"+ p.leaf().string());
    }
     std::sort(files.begin(),files.end());
    return files;
}

std::vector<cv::Mat> get_all_iamge(std::string image_folder_path)
{
    std::vector<cv::Mat> images;
    std::vector<std::string> image_files_path = get_all_image_file(image_folder_path);
    for(int i=0; i<  image_files_path.size() ;i++){
        cv::Mat image;
        image = cv::imread(image_files_path[i]);
        images.push_back(image);
    }
    return images;
}

int find_chessboard(cv::Mat image, std::vector<cv::Point2f> &image_points, cv::Size board_size)
{
    if (0 == findChessboardCorners(image, board_size, image_points))
    {
        std::cout<<"can not find chessboard corners!\n";
        return 0;
    }
    else
    {
        cv::Mat view_gray;
        cv::cvtColor(image,view_gray,cv::COLOR_RGB2GRAY);
        //对粗提取的角点进行亚像素精确化
        cv::find4QuadCornerSubpix(view_gray,image_points,cv::Size(11,11)); 
        //int nChessBoardFlags = cv::CALIB_CB_EXHAUSTIVE | cv::CALIB_CB_ACCURACY;
        //bool bFindResult = findChessboardCornersSB(view_gray,board_size,image_points,nChessBoardFlags );   
        //Opencv4 识别棋盘格方法，比opencv3有较大提升
    }
    return 1;
}

int init_chessboard_3dpoints(cv::Size board_size, std::vector<cv::Point3f> &points, float point_size)
{
    cv::Size2f square_size = cv::Size2f(point_size,point_size);
    for (int i=0;i<board_size.height;i++){
        for (int j=0;j<board_size.width;j++){
            cv::Point3f realPoint;
            realPoint.x = j*square_size.width;
            realPoint.y = i*square_size.height;
            realPoint.z = 0;
            points.push_back(realPoint);
        }
    }
    return 0;
}


void calib_monocular(std::vector<cv::Mat> images){
    cv::Size image_size;
    cv::Size board_size(11,4);
    std::vector<cv::Mat> images_tvecs_mat;
    std::vector<cv::Mat> images_rvecs_mat;
    image_size.width = images[0].cols;
    image_size.height = images[0].rows;
    std::vector<std::vector<cv::Point2f> > images_points;
  	// 识别所有图片的棋盘格
    for(int i=0;i<images.size();i++){
        std::vector<cv::Point2f> image_points;
        if(find_chessboard(images[i],image_points,board_size)>0){
             images_points.push_back(image_points);
        }
    }

    std::vector<cv::Point3f> image_points_in3d;
	// 计算棋盘格角点在棋盘格坐标系中的位置
    init_chessboard_3dpoints(board_size,image_points_in3d,0.045);  // 0.045为棋盘格一个格子的大小
    std::vector<std::vector<cv::Point3f> > images_points_in3d;
	// 生成所有识别出的标定板对应在各自棋盘格坐标系中的位置
    for(int i=0;i<images_points.size();i++){
        images_points_in3d.push_back(image_points_in3d);
    }
    cv::Mat intrinsic,distortion;
	// 使用张正友标定法计算内参畸变以及外参
    cv::calibrateCamera(images_points_in3d,
                        images_points,
                        image_size,
                        intrinsic,
                        distortion,
                        images_rvecs_mat,
                        images_tvecs_mat);
}

int main(int argc, char *argv[])
{
    std::string image_file_path = argv[1];
    std::vector<cv::Mat> images = get_all_iamge(image_file_path);
    calib_monocular(images);
    return 0;
}

Example 2

Calibrate a single view.

int cols = 10;
int rows = 7;
float distance = 30;    //间距30mm

cv::Size patternSize(cols,rows);
std::vector<cv::Point2f> corners;
std::vector<std::vector<cv::Point2f>> cornersVect;
std::vector<cv::Point3f> worldPoints;
std::vector<std::vector<cv::Point3f>> worldPointsVect;

for (int i=0;i<cols;i++)
{
    for (int j=0;j<rows;j++)
    {
        worldPoints.push_back(cv::Point3f(i*distance,j*distance,0));
    }
}

bool find=cv::findChessboardCorners(image,patternSize,corners);
cv::drawChessboardCorners(image,patternSize,corners,find);
cv::Mat cameraMatirx,distCoeffs;
std::vector<cv::Mat> rvecs,tvecs,rvecs2,tvecs2;
if (find)
{    
    cornersVect.push_back(corners);
    worldPointsVect.push_back(worldPoints);
    cv::calibrateCamera(worldPointsVect,
                        cornersVect,image.size(),
                        cameraMatirx,
                        distCoeffs,
                        rvecs,tvecs);
}

6.2 Python implementation

[Study notes] Basic use of python-Opencv cv2.findChessboardCorners()

Example 1

Calibrate a single view.

import numpy as np
import cv2
import matplotlib.pyplot as plt
import matplotlib.image as mpimg

fname='calibration_test.png'

image=cv2.imread(fname)
# plt.imshow(image)
# plt.show()

gray=cv2.cvtColor(image,cv2.COLOR_RGB2GRAY)
plt.imshow(gray,cmap='gray')
plt.show()

# Find the chessboard corners
nx=8
ny=6
ret, corners = cv2.findChessboardCorners(gray, (nx, ny), None)

print('ret:',ret)
# print(len(corners))

# If found, draw corners
if ret == True:
    # Draw and display the corners
    cv2.drawChessboardCorners(image, (nx, ny), corners, ret)
    plt.imshow(image)

patternSize(w,h)As shown in the figure below, for a 9x7 checkerboard calibration board, a row of checkerboards is 9, and a column of checkerboards is 7, so w=8 and h=6 in the inner corner points .

Example 2

Calibrate a single view.

import numpy as np
import cv2 as cv
import glob


# termination criteria
# 
criteria = (cv.TERM_CRITERIA_MAX_ITER + cv.TERM_CRITERIA_EPS, 30, 0.001)

# prepare object points, like (0,0,0), (1,0,0), (2,0,0) ....,(6,5,0)
"""
objp = 
[[0 0 0], 
[0 0 0],
...,
[0 0 0]], (42, 3)
"""
objp = np.zeros((6*7,3), np.float32)

"""
np.mgrid[0:7,0:6] = array([
[[0, 0, 0, 0, 0, 0],
[1, 1, 1, 1, 1, 1],
[2, 2, 2, 2, 2, 2],
[3, 3, 3, 3, 3, 3],
[4, 4, 4, 4, 4, 4],
[5, 5, 5, 5, 5, 5],
[6, 6, 6, 6, 6, 6]],

[[0, 1, 2, 3, 4, 5],
[0, 1, 2, 3, 4, 5],
[0, 1, 2, 3, 4, 5],
[0, 1, 2, 3, 4, 5],
[0, 1, 2, 3, 4, 5],
[0, 1, 2, 3, 4, 5],
[0, 1, 2, 3, 4, 5]]]), (2, 7, 6)

objp = array([
[0 0 0], 
[1 0 0], 
[2 0 0],
...,
[6 5 0]], (42, 3)
"""
objp[:,:2] = np.mgrid[0:7,0:6].T.reshape(-1,2)

# Arrays to store object points and image points from all the images.
objpoints = [] # 3d point in real world space
imgpoints = [] # 2d points in image plane.

images = glob.glob('*.jpg')
for fname in images:
	img = cv.imread(fname)
    gray = cv.cvtColor(img, cv.COLOR_BGR2GRAY)
    
    # Find the chess board corners
    ret, corners = cv.findChessboardCorners(gray, (7,6), None)
    
    # If found, add object points, image points (after refining them)
    if ret == True:
        objpoints.append(objp)
        corners2 = cv.cornerSubPix(gray,corners, (11,11), (-1,-1), criteria)
        imgpoints.append(corners2)
        
        # Draw and display the corners
        cv.drawChessboardCorners(img, (7,6), corners2, ret)

        cv.imshow('img', img)
        cv.waitKey(500)
cv.destroyAllWindows()

# Calibration
"""
camera matrix: mtx
distortion coefficients: dist
rotation vectors: rvecs
translation vectors: tvecs
"""
ret, mtx, dist, rvecs, tvecs = cv.calibrateCamera(objpoints, imgpoints, gray.shape[::-1], None, None)

if ret == True:
    # Re-projection Error
    mean_error = 0
    for i in range(len(objpoints)):
         imgpoints2, _ = cv.projectPoints(objpoints[i], rvecs[i], tvecs[i], mtx, dist)
         error = cv.norm(imgpoints[i], imgpoints2, cv.NORM_L2)/len(imgpoints2)
         mean_error += error
    print( "total error: {}".format(mean_error/len(objpoints)) )

patternSize(w,h)As shown in the figure below, for an 8x7 checkerboard calibration board, a row of checkerboards is 9, and a column of checkerboards is 7, so w=7, h=6 in the inner corner points .

4. Relevant experience

Summary of opencv calibration implementation (dots, checkerboards and asymmetric dots)