Checkerboard calibration for Opencv camera calibration

1. Image distortion

Camera imaging can be divided into four steps: rigid body transformation, perspective projection, distortion correction, and digitizing the image.

The image of the camera sometimes has serious distortion. Distortion refers to the offset between the real imaging point and the ideal imaging point. The reason is that the lens technology is not perfect, which leads to irregular refraction.

Correcting an image requires two camera parameters:

• Internal parameters of the camera . Such as the focal length of the lens, the optical center and the radial distortion coefficient.
• Extrinsic parameters : this refers to the orientation of the camera relative to some world coordinate system (rotation and translation)

Distortion can be improved through calibration, the image is not distorted, and it is close to the real image. In addition, the relationship between the camera's natural unit (pixel) and the actual unit (such as millimeters) can be determined, so that the size of the object in the image can be known after calibration.

2. Process steps of camera calibration

• Step 1: Collect checkerboard images (more than 10) and preprocess
• Step 2: Find the corner coordinates of the checkerboard
• Step 3: Further extract sub-pixel corner information
• Step 4: Calculate the camera intrinsic parameter matrix and distortion coefficient
• Step 5: Distorted Image Calibration

Step 1: Collect checkerboard images (more than 10) and preprocess

For image collection, the method is to hold a checkerboard printed on A4 paper in your hand, change different angles to the camera, and collect more than ten images; or put the checkerboard on the table and take a shot from different angles with the camera.
In order to demonstrate, here are fifteen photos taken with a mobile phone:

each photo is 3968x2976 (width x height), and the preprocessing is performed to compress the size by four times:

#include "stdafx.h"
#include<opencv2/opencv.hpp>
#include<iostream>
using namespace cv;
using namespace std;

Mat image;
int main()
{
    
    
	vector<String> images_path;//创建容器存放读取图像路径
	string image_path = "D:/work_c++/image/chessboardimage/*.jpg";//待处理图路径
	
	glob(image_path, images_path);//读取指定文件夹下图像
	char saveimage[100];
	for (int i = 0; i < images_path.size(); i++)
	{
    
    
		image = imread(images_path[i]);
		resize(image, image, Size(992, 744));//调整大小
		sprintf_s(saveimage, "D:/work_c++/image/temp/%d.jpg", i + 1);//数据格式化输出到字符串，图像存储路径
		imwrite(saveimage, image);//保存
	}

    return 0;
}

result:

Step 2: Find the corner coordinates of the checkerboard

Here the number of interior angles in each row and column of the checkerboard is the (5, 7)

main function:

• findChessboardCorners function

//! finds checkerboard pattern of the specified size in the image  
CV_EXPORTS_W bool findChessboardCorners( InputArray image, Size patternSize,  
                                         OutputArray corners,  
                                         int flags=CALIB_CB_ADAPTIVE_THRESH+CALIB_CB_NORMALIZE_IMAGE );

/*
第一个参数Image，传入拍摄的棋盘图Mat图像，必须是8位的灰度或者彩色图像；

第二个参数patternSize，每个棋盘图上内角点的行列数，一般情况下，行列数不要相同，便于后续标定程序识别标定板的方向；

第三个参数corners，用于存储检测到的内角点图像坐标位置，一般用元素是Point2f的向量来表示：vector<Point2f> image_points_buf;

第四个参数flage：用于定义棋盘图上内角点查找的不同处理方式，有默认值。

*/

Step 3: Further extract sub-pixel corner information

Main function:

• cornerSubPix function

//! adjusts the corner locations with sub-pixel accuracy to maximize the certain cornerness criteria
CV_EXPORTS_W void cornerSubPix( InputArray image, InputOutputArray corners,
                                Size winSize, Size zeroZone,
                                TermCriteria criteria );

/*
第一个参数image，输入的Mat矩阵，最好是8位灰度图像，检测效率更高；

第二个参数corners，初始的角点坐标向量，同时作为亚像素坐标位置的输出，所以需要是浮点型数据，一般用元素是Pointf2f/Point2d的向量来表示：vector<Point2f/Point2d> iamgePointsBuf；

第三个参数winSize，大小为搜索窗口的一半；

第四个参数zeroZone，死区的一半尺寸，死区为不对搜索区的中央位置做求和运算的区域。它是用来避免自相关矩阵出现某些可能的奇异性。当值为（-1，-1）时表示没有死区；

第五个参数criteria，定义求角点的迭代过程的终止条件，可以为迭代次数和角点精度两者的组合；

*/

Step 4: Calculate the camera intrinsic parameter matrix and distortion coefficient

Main function:

• calibrateCamera function

//! finds intrinsic and extrinsic camera parameters from several fews of a known calibration pattern.  
CV_EXPORTS_W double calibrateCamera( InputArrayOfArrays objectPoints,  
                                     InputArrayOfArrays imagePoints,  
                                     Size imageSize,  
                                     CV_OUT InputOutputArray cameraMatrix,  
                                     CV_OUT InputOutputArray distCoeffs,  
                                     OutputArrayOfArrays rvecs, OutputArrayOfArrays tvecs,  
                                     int flags=0, TermCriteria criteria = TermCriteria(  
                                         TermCriteria::COUNT+TermCriteria::EPS, 30, DBL_EPSILON) 
 
/*
第一个参数objectPoints，为世界坐标系中的三维点。在使用时，应该输入一个三维坐标点的向量的向量，即vector<vector<Point3f>> object_points。需要依据棋盘上单个黑白矩阵的大小，计算出（初始化）每一个内角点的世界坐标。

第二个参数imagePoints，为每一个内角点对应的图像坐标点。和objectPoints一样，应该输入vector<vector<Point2f>> image_points_seq形式的变量；

第三个参数imageSize，为图像的像素尺寸大小，在计算相机的内参和畸变矩阵时需要使用到该参数；

第四个参数cameraMatrix为相机的内参矩阵。输入一个Mat cameraMatrix即可，如Mat cameraMatrix=Mat(3,3,CV_32FC1,Scalar::all(0));

第五个参数distCoeffs为畸变矩阵。输入一个Mat distCoeffs=Mat(1,5,CV_32FC1,Scalar::all(0))即可；

第六个参数rvecs为旋转向量；应该输入一个Mat类型的vector，即vector<Mat>rvecs;

第七个参数tvecs为位移向量，和rvecs一样，应该为vector<Mat> tvecs；

第八个参数flags为标定时所采用的算法。有如下几个参数：

CV_CALIB_USE_INTRINSIC_GUESS：使用该参数时，在cameraMatrix矩阵中应该有fx,fy,u0,v0的估计值。否则的话，将初始化(u0,v0）图像的中心点，使用最小二乘估算出fx，fy。 
CV_CALIB_FIX_PRINCIPAL_POINT：在进行优化时会固定光轴点。当CV_CALIB_USE_INTRINSIC_GUESS参数被设置，光轴点将保持在中心或者某个输入的值。 
CV_CALIB_FIX_ASPECT_RATIO：固定fx/fy的比值，只将fy作为可变量，进行优化计算。当CV_CALIB_USE_INTRINSIC_GUESS没有被设置，fx和fy将会被忽略。只有fx/fy的比值在计算中会被用到。 
CV_CALIB_ZERO_TANGENT_DIST：设定切向畸变参数（p1,p2）为零。 
CV_CALIB_FIX_K1,…,CV_CALIB_FIX_K6：对应的径向畸变在优化中保持不变。 
CV_CALIB_RATIONAL_MODEL：计算k4，k5，k6三个畸变参数。如果没有设置，则只计算其它5个畸变参数。

第九个参数criteria是最优迭代终止条件设定。

在使用该函数进行标定运算之前，需要对棋盘上每一个内角点的空间坐标系的位置坐标进行初始化，标定的结果是生成相机的内参矩阵cameraMatrix、相机的5个畸变系数distCoeffs，另外每张图像都会生成属于自己的平移向量和旋转向量。
*/                                        

Step 5: Distorted Image Calibration

Main function:

• undistort function

//! corrects lens distortion for the given camera matrix and distortion coefficients  
CV_EXPORTS_W void undistort( InputArray src, OutputArray dst,  
                             InputArray cameraMatrix,  
                             InputArray distCoeffs,  
                             InputArray newCameraMatrix=noArray() ); 

/*
第一个参数src，输入参数，代表畸变的原始图像；

第二个参数dst，矫正后的输出图像，跟输入图像具有相同的类型和大小；

第三个参数cameraMatrix为之前求得的相机的内参矩阵；

第四个参数distCoeffs为之前求得的相机畸变矩阵；

第五个参数newCameraMatrix，默认跟cameraMatrix保持一致；

方法一相比方法二执行效率更高一些，推荐使用
*/

3. Code implementation

environment:

• OpenCV3.4.1
• vs2015

// CheckerBoardDemo.cpp : 定义控制台应用程序的入口点。
//

#include "stdafx.h"
#include<opencv2/opencv.hpp>
#include<iostream>
using namespace cv;
using namespace std;

Mat image,img_gray;
int BOARDSIZE[2]{
    
     5,7 };//棋盘格每行每列角点个数
int main()
{
    
    
	vector<vector<Point3f>> objpoints_img;//保存棋盘格上角点的三维坐标
	vector<Point3f> obj_world_pts;//三维世界坐标
	vector<vector<Point2f>> images_points;//保存所有角点
	vector<Point2f> img_corner_points;//保存每张图检测到的角点
	vector<String> images_path;//创建容器存放读取图像路径

	string image_path = "D:/work_c++/image/temp/*.jpg";//待处理图路径	
	glob(image_path, images_path);//读取指定文件夹下图像

	//转世界坐标系
	for (int i = 0; i < BOARDSIZE[1]; i++)
	{
    
    
		for (int j = 0; j < BOARDSIZE[0]; j++)
		{
    
    
			obj_world_pts.push_back(Point3f(j, i, 0));
		}
	}

	for (int i = 0; i < images_path.size(); i++)
	{
    
    
		image = imread(images_path[i]);
		cvtColor(image, img_gray, COLOR_BGR2GRAY);
		//检测角点
		bool found_success = findChessboardCorners(img_gray, Size(BOARDSIZE[0], BOARDSIZE[1]), 
			img_corner_points,
			CV_CALIB_CB_ADAPTIVE_THRESH | CV_CALIB_CB_FAST_CHECK | CV_CALIB_CB_NORMALIZE_IMAGE);
		
		//显示角点
		if (found_success)
		{
    
    
			 //迭代终止条件
			 TermCriteria criteria(CV_TERMCRIT_EPS | CV_TERMCRIT_ITER, 30, 0.001);

			 //进一步提取亚像素角点
			 cornerSubPix(img_gray, img_corner_points, Size(11, 11),
				 Size(-1, -1), criteria);

			 //绘制角点
			 drawChessboardCorners(image, Size(BOARDSIZE[0], BOARDSIZE[1]),img_corner_points,
				 found_success);

			 objpoints_img.push_back(obj_world_pts);//从世界坐标系到相机坐标系
			 images_points.push_back(img_corner_points);
		}
		char *output = "image";
		imshow(output, image);
		waitKey(200);
	}
	
	/*
	计算内参和畸变系数等
	*/

	Mat cameraMatrix, distCoeffs, R, T;//内参矩阵，畸变系数，旋转量，偏移量
	calibrateCamera(objpoints_img, images_points, img_gray.size(), 
		cameraMatrix, distCoeffs, R, T);

	cout << "cameraMatrix:" << endl;
	cout << cameraMatrix << endl;

	cout << "*****************************" << endl;
	cout << "distCoeffs:" << endl;
	cout <<  distCoeffs << endl;
	cout << "*****************************" << endl;

	cout << "Rotation vector:" << endl;
	cout << R << endl;

	cout << "*****************************" << endl;
	cout << "Translation vector:" << endl;
	cout << T << endl;


	/*
	畸变图像校准
	*/
	Mat src, dst;
	src = imread("D:/work_c++/image/test/test.jpg");
	undistort(src, dst, cameraMatrix, distCoeffs);
	
	char *dst_output = "image_dst";
	imshow(dst_output, dst);
	waitKey(100);
	imwrite("D:/work_c++/image/test/result.jpg", dst);
	
	destroyAllWindows();//销毁显示窗口
	system("pause");
    return 0;
}

4. Results

4.1 Detecting corner effect

4.2 Internal reference, distortion coefficient results

4.3 Distortion map calibration test results

Original test image:

Test result image:

Note: The checkerboard picture taken with a mobile phone, when printing the checkerboard, checked Adapt to A4 paper, and did not print according to the actual size, resulting in the checkerboard not being a square grid, 3.5cm on one side, and one side on the other. 3.7cm, no re-printing, just for testing.
In addition, one square is used as the unit here, and the unit length is not considered. To calculate the actual parameters, you need to multiply the unit length.
If changed in program:

	//转世界坐标系
	realsize = Size(x,y);//测得实际真实尺寸
	for (int i = 0; i < BOARDSIZE[1]; i++)
	{
    
    
		for (int j = 0; j < BOARDSIZE[0]; j++)
		{
    
    
			obj_world_pts.push_back(Point3f(j*realsie.with, i*realsize.height, 0));
		}
	}
//与计算内参矩阵无关，与计算外参旋转、平移量有关

At present, the distortion of industrial cameras is basically very small. The manufacturers have done a lot to eliminate the effects of distortion. Users don’t need to do distortion correction specially. Generally, cheap network cameras have particularly large distortion and need to be corrected.

Reference：
https://yq.aliyun.com/articles/62472
http://www.opencv.org.cn/opencvdoc/2.3.2/html/doc/tutorials/calib3d/camera_calibration_square_chess/camera_calibration_square_chess.html
https://blog.csdn.net/Crystal_YS/article/details/86582414