Machine Vision - Camera Calibration

1 Why does machine vision need camera calibration

Machine vision is the use of camera imaging to achieve the measurement, positioning, reconstruction and other processes of three-dimensional scenes. It is a process of using two-dimensional images to perform three-dimensional inversion. The world we live in is three-dimensional, while images or photos are two-dimensional. The camera can be regarded as a function, the input is a three-dimensional scene, and the output is a two-dimensional image. Normally, this process from three-dimensional to two-dimensional is irreversible.
If we can find a suitable mathematical model to approximate the above three-dimensional to two-dimensional process, and then find the inverse function of this mathematical model, we can realize the inverse process from two-dimensional to three-dimensional, that is, use a simple mathematical model to Express the complex imaging process and find out the inverse process of imaging.

In the previous articles " Machine Vision Model - Projection Matrix " and " Machine Vision Model - Distortion Model ", we discussed two important mathematical models of machine vision - projection imaging model and distortion model . We know that this Several models contain several unknown parameters. So how are these unknown parameters obtained? This is the role of camera calibration .

2 The role of camera calibration

As mentioned earlier, the imaging matrix and distortion model include the following unknowns:

1. Internal reference matrix: 4 parameters, $f/dx$、$f/dy$、$u_0$、$v_0$；
2. External parameter matrix: 6 parameters, 3 parameters for the pose, translation and rotation of the rigid body of the camera.
3. Distortion parameters: 5 parameters, radial distortion 3 $k_1$、$k_2$、$k_3$, tangential distortion 2 $p_1$、$p_2$；

Camera calibration: It is the process of determining the internal and external parameters and distortion parameters of the camera.
Camera calibration can be said to be the basis of computer vision/machine vision. The task is to solve 9 (internal parameters + distortion) or 15 (internal parameters + distortion + external ref) Unknown parameter. Most of the camera calibration is to calibrate the internal parameters and distortion parameters of the camera, because the external parameters belong to the camera pose, and the camera pose is constantly changing during the work process, but some applications also need to calibrate the external parameters at the same time, that is, the camera pose.

3 Common methods of camera calibration

According to the mathematical model of the calibration process, camera calibration can be divided into:
(1) Linear calibration method: fast calculation speed, relatively low calibration accuracy
(2) Non-linear optimization calibration method: high calibration accuracy, complex model, and large amount of calculation;
( 3) Two-step method: the combination of the above two, which combines the advantages and disadvantages of the two, such as Tsai's two-step method, Zhang's calibration method, etc.
The table below gives a comparison of several commonly used calibration methods.

Calibration method	advantage	shortcoming	common method
Camera self-calibration method	Strong flexibility and online calibration	Low accuracy and poor robustness	Hierarchical step-by-step calibration, based on Kruppa equation
Active Vision Camera Calibration Method	No need for calibrator, simple algorithm, high robustness	High cost and expensive equipment	The active system controls the camera to make specific movements
calibrator calibration method	Can be used with any camera model, high accuracy	Need calibrator, complex algorithm	Tsai's two-step method, Zhang's calibration method

3.1 Camera self-calibration method

The camera self-calibration method does not need to know the three-dimensional coordinates of the image point. It determines the parameters of the camera calibration by calculating the relative relationship of a certain point in the scene graph at different shooting angles. This characteristic of the self-calibration method enables it to complete the calibration of some unknown camera parameters.
The camera self-calibration method can obtain the internal and external parameters of the calibration system only by using the correspondence or constraint relationship between image points without calibration objects, which makes camera calibration in some arbitrary camera movements or complex unknown scenes a reality. This benefited from the self-calibration concept first proposed by Luong, Faugeras, Maybank, etc. in the early 1990s.
At present, the main methods of self-calibration are: self-calibration method based on Kruppa equation, layered step-by-step calibration method. These camera self-calibration methods solve the parameters through the constraint relationship of the camera's internal parameters, completely ignoring the external environment of the camera system, so this calibration method is more flexible than other methods, but the stability is not high.

1) Hierarchical step-by-step calibration method

The layered step-by-step calibration method is to firstly reconstruct the image sequence projectively, then perform affine calibration and European calibration on the basis of the reconstruction, and obtain the internal and external parameters of the camera through a nonlinear optimization algorithm. Since the initial parameters are fuzzy values, the convergence of the optimization algorithm is uncertain.

2) Self-calibration method based on Kruppa

The Kruppa-based self-calibration method is to establish a constraint equation about the camera internal parameter matrix through a quadratic curve, and use at least 3 pairs of images to calibrate the camera. The length of the image sequence will affect the stability of the calibration algorithm, and the infinite plane in the projective space cannot be guaranteed.

3.2 Calibration method based on active vision

The camera calibration method based on active vision is to control the camera to do specific motion through the active system, take multiple sets of images, and solve the internal and external parameters of the camera according to the image information and known displacement changes. There are two typical methods: one is to make the camera translate stably in three-dimensional space; the other is to make the camera rotate with fixed parameters. The calibration method based on active vision can simplify the calculation process and obtain linear results. The disadvantage of this method is that it needs to be equipped with a precise control platform, the cost of the system is high, and it is not applicable to some occasions where the motion parameters are unknown and uncontrollable.

3.3 Calibration method based on calibrator

Different from the active vision calibration method, the calibration object is indispensable in the camera calibration based on the calibration object. It not only needs to determine the size and shape, but also determine the coordinates of special points on the surface of the object. The principle is to use mathematical methods to find out the corresponding relationship between the spatial coordinates of a certain point and the image coordinates, and then obtain the camera parameters.
Typical ones are: direct linear transformation method (DLT), Tsai two-step method, Zhang Zhengyou plane calibration method, etc. Among them, the calibration objects are further divided into three-dimensional calibration objects and plane calibration objects. The three-dimensional calibration method is easy to operate and reliable in accuracy. However, the production cost of the three-dimensional calibration object is expensive, and it is difficult to process and maintain. The production of the plane calibration object is simple, and the required accuracy can be guaranteed through the improved algorithm. Therefore, some calibration methods in recent years are based on the development and improvement of the plane calibration. .
The calibration accuracy of the camera calibration method based on the calibration object has high requirements on the calibration object, and some workplaces are not suitable for its placement, which also limits the scope of application.

1) Tsai's two-step method

Tsai's two-step method first obtains the camera parameters linearly, then considers the distortion factor to obtain the initial parameter values, and obtains the final camera parameters through nonlinear optimization. Tsai's two-step method is faster, but only considers the radial distortion, which is not applicable when the camera distortion is severe.

2) Zhang's calibration method

See next section.

4 Introduction to Zhang's Calibration Method

"Zhang's calibration method" is a paper " Flexible Camera Calibration By Viewing a Plane From Unknown Orientations " ( article PDF download ) published by Dr. Zhang Zhengyou on the top international conference ICCV in 1999. It uses a plane checkerboard for camera calibration. practical method.

4.1 Principle

The basic principle of Zhang’s calibration method: use a calibration board composed of two-dimensional squares for calibration, collect pictures of different poses of the calibration board, extract the pixel coordinates of the corner points in the picture, and calculate the initial value of the internal and external parameters of the camera through the homography matrix. The distortion coefficients are estimated by nonlinear least squares method, and finally the parameters are optimized by maximum likelihood estimation method.
This method is between the photographic calibration method and the self-calibration method, which not only overcomes the shortcomings of the high-precision three-dimensional calibration object required by the photographic calibration method, but also solves the problem of poor robustness of the self-calibration method. The calibration process only needs to use a printed checkerboard and take several groups of pictures from different directions. Anyone can make the calibration pattern by themselves. It is not only flexible and convenient to use, but also has high precision and good robustness. Therefore, it was widely adopted all over the world soon, which greatly promoted the process of 3D computer vision from the laboratory to the real world.

4.2 Calibration board

An important part of Zhang’s calibration method is the calibration board. On the calibration board, the topological structure is printed, and checkerboard and dot grids are widely used. Why are checkerboard or origin grids commonly used? This is because: the topological structure of the checkerboard or the origin lattice is clear and uniform, and the image processing algorithm for detecting its topological structure is simple and effective.

In addition to these two commonly used patterns, there are many calibration board formats, and each calibration software has a corresponding calibration board format, but no matter what style of calibration board, its basic functions are similar:

1. Determine the correspondence between object points and image points;
2. Transform the corner points in the calibration board to the coordinates in the camera coordinate system, because the position of each corner point in the calibration board in the calibration board coordinate system is determined and known.

5 How to complete camera calibration

The above discusses some basic knowledge of camera calibration, so if we want to actually calibrate, what should we do? The following methods are commonly used:

5.1 Using forming tools

Many commercial software or tools have integrated camera calibration functions, such as the following two typical software:

5.1.1 Falcon

The calibration assistant in the Halcon software can complete the camera calibration, as shown in the figure below is the format of the Halcon calibration board.

Using the Halcon calibration assistant, after completing the steps, you can get the calibration results of the internal and external parameters of the camera as shown in the figure.

5.1.2 Matlab

Camera calibration tools are also integrated in Matlab.

Among them, Camera Calibrator is used for monocular camera calibration, and Stereo Camera Calibrator is used for binocular camera calibration.

Calibration results show:

5.2 Programming implementation

5.2.1 Implementation using Opencv programming

OpenCV integrates related functions such as checkerboard corner detection, homography matrix, and internal and external parameter calculations, which can be used to build your own camera calibration tools. Here are some references to commonly used functions

findChessboardCorners();//棋盘格角点检测
drawChessboardCorners();//棋盘格角点绘制
find4QuadCornerSubpix();//粗提取角点精确化
findHomography();//获取单应矩阵
cornerSubPix();//亚像素检测
calibratCamera();//求解相机内外参
initUndistortRectifyMap();//计算畸变参数

5.2.2 Fully self-programmed implementation

The above tools and programming interfaces have their own advantages and disadvantages. In fact, the calibration results of each method have more or less deviations, and the accuracy is not the same.
Therefore, if you understand the principle of camera calibration, you can program it by yourself if necessary. Of course, this method is also the most difficult, and the result of camera calibration (accuracy, speed, etc.) depends entirely on your implementation principle and program. Performance, but this method can be optimized and adjusted according to your own needs, and the adaptability is better.

reference

Actual Combat | Camera Calibration
http://www.imrobotic.com/news/detail/5302