Table of contents
【1】Related introduction to calibration
【2】Introduction to algorithm flow and related operators
(1) The algorithm process mainly consists of five parts:
(2) Introduction to related operators
1. Find the corner points on the chessboard calibration board
【1】Related introduction to calibration
(1) Purpose of calibration
Before performing camera calibration, you must understand the purpose of camera calibration. Simply put, camera calibration is mainly to find the internal and external parameters of the camera based on the different positions of the calibration plate under the camera. Internal parameters are inherent properties of the camera, which means that no matter how you place the calibration plate, your camera is not fixed and the camera's internal parameters are not affected. However, the accuracy of the internal parameters is related to the number of calibration plate images. Generally speaking, the calibration plate The more comprehensive the placement, the more accurate the internal parameters of the camera and the smaller the error. For the external parameters of the camera, it is the spatial transformation (rotation matrix and translation matrix) of the world coordinate system relative to the camera coordinate system.
(2) Introduction to internal parameters
: The number of pixels in the X direction within the unit millimeter;
: The number of pixels in the Y direction within the unit millimeter;
,
: represents the x and y coordinates of the image center;
The internal parameter matrix (3 x 3) is as shown below:
In practical applications, a telecentric lens is used. Within the depth of field range, the field of view of the image is constant, which means that the telecentric lens will not be distorted, and the internal parameters are certain. When used, it can be configured according to the parameters of the camera. But for ordinary cameras, similar to fisheye lenses, the internal parameters need to be corrected when using it, and it doesn’t matter whether the camera is fixed or not.
(3) Distortion coefficient
The distortion coefficient mainly has 5 parameters: k1, k2, k3 radial distortion coefficients, p1, p2 are tangential distortion coefficients. Radial distortion occurs in the process of converting the camera coordinate system to the image physical coordinate system. Tangential distortion occurs during the camera manufacturing process because the plane of the photoreceptor is not parallel to the lens. Radial distortion, that is, the distortion caused by the different focal lengths of different areas of the lens, is divided into pincushion distortion and barrel distortion, as shown in the figure below. The distortion is more obvious closer to the edge of the lens. The deformation is as shown in the figure:
(4) Camera external parameters
The external parameters of the camera are the description of the world coordinate system in the camera coordinate system. R is the rotation parameter that is the product of the rotation matrix of each axis, where the rotation parameter of each axis ( ϕ , ω , θ ), the translation parameter ( Tx , Ty , Tz ), the matrix is (4 x 4 homogeneous matrix) as follows :
【2】Introduction to algorithm flow and related operators
(1) The algorithm process mainly consists of five parts:
1. Extract the corner points of the calibration plate.
2. Sub-pixel corner precision.
3. Visualize the corners.
4. Camera calibration.
5. Error calculation (reprojection error).
(2) Introduction to related operators
1. Find the corner points on the chessboard calibration board
bool findChessboardCorners(
InputArray image,
Size patternSize,
OutputArray corners,
int flags = CALIB_CB_ADAPTIVE_THRESH + CALIB_CB_NORMALIZE_IMAGE
);The first parameter is the incoming image, the image type is 8UC1 and 8UC3.
The second parameter is the number of interior corner points in the chessboard calibration board (how many in one row, how many in one column), for example
Size size（9,6）
//一行九个内角点，一列6个内角点
The third parameter is used to record the pixel coordinates of the corner points to be searched.
Its data type is vector<point2f>, which stores the corner point coordinates of a picture. The storage of corner points of multiple calibration plates is defined.
vector<vector<point2f>>
The fourth parameter:
CALIB_CB_ADAPTIVE_THRESH = 1, find the calibration plate through the adaptive threshold method;
CALIB_CB_NORMALIZE_IMAGE = 2, if this flag is set, use CV::equalizeHist() to normalize the image before applying the threshold; CALIB_CB_FILTER_QUADS = 4, when the calibration plate image has distortion , the quadrilateral application will be constrained to prevent wrong quadrilaterals;
CALIB_CB_FAST_CHECK = 8, the image will be quickly scanned to ensure that there are corner points in the image. If there are no corner points, the image will be skipped directly.
The return value of this operator is whether the corner point is successfully found.
(Finding the circular grid calibration plate is similar to this and will not be introduced yet.)
2. Sub-pixel corner accuracy
Mainly the two operators cv::cornerSubPix and find4QuadCornerSubpix have the same parameters.
bool cv::find4QuadCornerSubpix ( InputArray img,
InputOutputArray corners,
Size region_size
)
img: The input Mat matrix, preferably an 8-bit grayscale image, has higher detection efficiency; corners: The initial corner point coordinate vector is also used as the output of the sub-pixel coordinate position, so it needs to be floating point data, generally using elements Pointf2f/Point2d. represented by vectors. region_size: The size of the corner point search window, the neighborhood range considered when optimizing coordinates.
3. Visualize corner points
The visualization operator for corner points is drawChessboardCorners.
void cv::drawChessboardCorners ( InputOutputArray image,
Size patternSize,
InputArray corners,
bool patternWasFound
) image: The target image where the corner points need to be drawn, it must be an 8-bit color image.
patternSize: The number of rows and columns of interior corner points on each calibrated chessboard.
corners: Array of detected corner point coordinates.
patternWasFound: Draw a corner point style flag to show whether a complete calibration plate is found.
When patternWasFound=true, connect each interior corner point in sequence.
When patternWasFound=false, the corner position is marked with a (red) circle.
4.Camera calibration
calibrateCamera is the operator of camera calibration. The specific function is as follows:
double calibrateCamera(
InputArrayOfArrays objectPoints, //输入：目标点位 的集合的集合
InputArrayOfArrays imagePoints, //输入：图像点位 的集合的集合
Size imageSize, //输入：图像尺寸
InputOutputArray cameraMatrix, //输出：相机内参矩阵（fx,fy,cx,cy）
InputOutputArray distCoeffs, //输出：畸变矫正多项式（k1,k2,p1,p2,k3...）
OutputArrayOfArrays rvecs, //输出：旋转矩阵 的集合
OutputArrayOfArrays tvecs, //输出：平移矩阵 的集合
int flags = 0, //输入：对标定过程方法进行更精细的控制
TermCriteria criteria = //输入：迭代参数
TermCriteria(TermCriteria::COUNT + TermCriteria::EPS, 30, DBL_EPSILON));
The first parameter, objectPoints, is a three-dimensional point in the world coordinate system. When using, you should enter a vector of three-dimensional coordinate point vectors, that is, vector<vector<Point3f>> object_points. It is necessary to calculate (initialize) the world coordinates of each interior corner point based on the size of a single black and white matrix on the chessboard.
The second parameter, imagePoints, is the image coordinate point corresponding to each interior corner point. Like objectPoints, a variable in the form of vector<vector<Point2f>> image_points_seq should be entered;
The third parameter, imageSize, is the pixel size of the image. This parameter needs to be used when calculating the internal parameters and distortion matrix of the camera;
The fourth parameter cameraMatrix is the internal parameter matrix of the camera. Just enter a Mat cameraMatrix, such as Mat cameraMatrix=Mat(3,3,CV_32FC1,Scalar::all(0));
The fifth parameter distCoeffs is the distortion matrix. Enter a Mat distCoeffs=Mat(1,5,CV_32FC1,Scalar::all(0));
The sixth parameter rvecs is the rotation vector; a vector of Mat type should be entered, that is, vector<Mat>rvecs;
The seventh parameter tvecs is the displacement vector. Like rvecs, it should be vector<Mat> tvecs;
The eighth parameter flags is the algorithm used during calibration. There are several parameters as follows:
CV_CALIB_USE_INTRINSIC_GUESS: When using this parameter, there should be estimated values of fx, fy, u0, v0 in the cameraMatrix matrix. Otherwise, the center point of the (u0, v0) image will be initialized, and fx, fy will be estimated using least squares.
CV_CALIB_FIX_PRINCIPAL_POINT: The optical axis point will be fixed during optimization. When the CV_CALIB_USE_INTRINSIC_GUESS parameter is set, the optical axis point will remain at the center or some input value.
CV_CALIB_FIX_ASPECT_RATIO: Fix the ratio of fx/fy, and only use fy as a variable variable for optimization calculation. When CV_CALIB_USE_INTRINSIC_GUESS is not set, fx and fy will be ignored. Only the fx/fy ratio is used in the calculation.
CV_CALIB_ZERO_TANGENT_DIST: Set the tangential distortion parameters (p1, p2) to zero.
CV_CALIB_FIX_K1,…,CV_CALIB_FIX_K6: The corresponding radial distortion remains unchanged during optimization.
CV_CALIB_RATIONAL_MODEL: Calculate three distortion parameters k4, k5, k6. If not set, only the other 5 distortion parameters are calculated.
The ninth parameter criteria is the optimal iteration termination condition setting.
Before using this function for calibration operation, it is necessary to initialize the position coordinates of the spatial coordinate system of each interior corner point on the chessboard. The result of the calibration is to generate the camera's internal parameter matrix cameraMatrix, the camera's five distortion coefficients distCoeffs, and each image Each will generate its own translation vector and rotation vector.
Return value: the root mean square error of the reprojection
5. Error calculation
Reprojection error projectPoints, the method to evaluate the calibration results is to re-project the three-dimensional points in the space through the obtained internal and external parameters of the camera, obtain the coordinates of the new projection points of the three-dimensional points in the space on the image, and calculate the projection coordinates and sub-pixels The deviation between corner point coordinates. The smaller the deviation, the better the calibration result.
void projectPoints( InputArray objectPoints,
InputArray rvec, InputArray tvec,
InputArray cameraMatrix, InputArray distCoeffs,
OutputArray imagePoints,
OutputArray jacobian=noArray(),
double aspectRatio=0 );The first parameter, objectPoints, is the three-dimensional point coordinates in the camera coordinate system;
The second parameter rvec is the rotation vector, and each image has its own selection vector;
The third parameter tvec is the displacement vector, and each image has its own translation vector;
The fourth parameter cameraMatrix is the internal parameter matrix of the camera obtained;
The fifth parameter distCoeffs is the distortion matrix of the camera;
The sixth parameter imagePoints is the coordinate point on the image corresponding to each interior corner point;
The seventh parameter jacobian is the Jacobian;
The eighth parameter aspectRatio is an optional parameter related to the photosensitive unit of the camera sensor. If set to non-0, the function defaults to a fixed dx/dy of the photosensitive unit, and the Jacobian matrix will be adjusted accordingly;
【3】Complete code
#include<opencv2\opencv.hpp>
#include<opencv2\highgui\highgui.hpp>
#include<opencv2\imgproc\imgproc.hpp>
#include<iostream>
#include<fstream>
#include<io.h>
#include<string>
using namespace cv;
using namespace std;
const char* path = "E:\\乔大花进度\\11-24\\相机标定\\calibration2\\*.jpg";//标定板存放地址
string pic_path = "E:\\乔大花进度\\11-24\\相机标定\\calibration2\\";
int main(int argc, char** argv)
{
//【1】 从文件夹中读取标定图片
intptr_t handle; //用于文件查找的句柄
struct _finddata_t fileinfo; //定义文件存储的结构体
handle = _findfirst(path, &fileinfo); //用于第一次查找，handle记录查找结果，未找到返回-1；
if (handle==-1)
{
return -1; //未成功读取图片结束程序
}
if (fileinfo.attrib!= _A_SUBDIR)
{
printf("%s\n",fileinfo.name); //如果不是子文件夹则输出名字
}
ofstream outdata; //定义输出流
outdata.open(pic_path + "list.txt",ios::trunc);//打开存放标定板名字的txt文件，并且清空其内容
outdata << fileinfo.name << endl;
outdata.close();
while (!_findnext(handle,&fileinfo))//继续在文件夹下搜索标定板jpg格式文件，未找到返回-1，找到返回0；
{
ofstream outdata; //定义输出流
if (fileinfo.attrib != _A_SUBDIR)//
printf("%s\n", fileinfo.name);
outdata.open(pic_path + "list.txt", ios::app);
outdata << fileinfo.name << endl;
}
outdata.close();//关闭输出流
_findclose(handle);//关闭查找句柄
//定义vector用于存放图片名字；
vector<string> pic_name;
ifstream indata;
indata.open(pic_path + "list.txt", ios::in);
if (!indata)//未打开成功返回0
{
cout << "读取txt文件失败" << endl;
return -1;
}
while (!indata.eof())
{
string str;
getline(indata,str);
if (str.length()>2)
{
pic_name.push_back(str);
}
}
cout << "标定板照片的数量为" << pic_name.size() << endl;
//【2】 依次读取每张照片，提取角点
cout << "开始提取角点……" << endl;
int image_nums = 0; //图片数量
Size image_size; //图片尺寸
int points_per_row = 9; //每行的内点数
int points_per_col = 6; //每列的内点数
Size corner_size = Size(points_per_row, points_per_col); //标定板每行每列角点个数，共12*12个角点
vector<Point2f> points_per_image; //缓存每幅图检测到的角点
vector<vector<Point2f>> points_all_images; //用一个二维数组保存检测到的所有角点
string image_file_name; //声明一个文件名的字符串
for (int i = 0; i < pic_name.size(); i++)
{
image_nums++;
//读入图片
Mat image_raw = imread(pic_path + pic_name[i]);
if (image_nums == 1)
{
// cout<<"channels = "<<image_raw.channels()<<endl;
// cout<<image_raw.type()<<endl; //CV_8UC3
image_size.width = image_raw.cols; //图像的宽对应着列数
image_size.height = image_raw.rows; //图像的高对应着行数
cout << "image_size.width = " << image_size.width << endl;
cout << "image_size.height = " << image_size.height << endl;
}
//角点检测
Mat image_gray;
cvtColor(image_raw, image_gray,COLOR_BGR2GRAY); //将BGR图转化为灰度图
//step1 提取角点,并且返回布尔值
bool success = findChessboardCorners(image_gray, corner_size, points_per_image, CALIB_CB_FAST_CHECK);
if (!success)
{
cout << pic_name[i]<<":图片未找到角点" << endl;
continue;
}
else
{
//亚像素精确化（两种方法）
//step2 亚像素角点
find4QuadCornerSubpix(image_gray, points_per_image, Size(5, 5));
// cornerSubPix(image_gray,points_per_image,Size(5,5));
points_all_images.push_back(points_per_image); //保存亚像素角点
//step3 角点可视化,//在图中画出角点位置
drawChessboardCorners(image_raw, corner_size, points_per_image, success); //将角点连线
imshow("Camera calibration", image_raw);
waitKey(0); //等待按键输入
cout << "照片的序号为：" << image_nums << endl;
}
}
//destroyAllWindows();
int nums = points_all_images.size();
cout << "处理标定照片的数量为：" << nums << endl;
cout << "开始进行相机标定........." << endl;
//【3】将标定板中二维像素点位转为三维空间点
//开始相机标定
Size2f block_size(0.023, 0.023); //每个小方格实际大小, 只会影响最后求解的平移向量t(标定板中每个小方块的实际大小)
Mat camera_K(3, 3, CV_32FC1, Scalar::all(0)); //内参矩阵3*3
Mat distCoeffs(1, 5, CV_32FC1, Scalar::all(0)); //畸变矩阵1*5
vector<cv::Mat> rotationMat; //旋转矩阵
vector<cv::Mat> translationMat; //平移矩阵
//初始化角点三维坐标,从左到右,从上到下!!!
vector<Point3f> points3D_per_image;
for (int i = 0; i < corner_size.height; i++)
{
for (int j = 0; j < corner_size.width; j++)
{
points3D_per_image.push_back(Point3f(block_size.width * j, block_size.height * i, 0));
}
}
//生成与图片数量相同的标定板照片向量，也就是一张拍摄照片和一张标准标定板照片对应
vector<vector<Point3f>> points3D_all_images(nums, points3D_per_image); //保存所有图像角点的三维坐标, z=0
int point_counts = corner_size.area(); //每张图片上角点个数
cout << "标定板的角点数量为：" << point_counts << endl;
for (int i = 0; i < points3D_per_image.size(); i++)
{
cout << points3D_per_image[i] << endl;
}
//【4】开始进行标定
calibrateCamera(points3D_all_images, points_all_images, image_size, camera_K, distCoeffs, rotationMat, translationMat, 0);
//【5】对标定结果进行评价，计算标定误差
ofstream fout;
//将标定出来的误差存到该文件下；
fout.open(pic_path+"calibration.txt", ios::app);
double total_err = 0.0; //所有图像平均误差总和
double err = 0.0; //每幅图像的平均误差
vector<cv::Point2f> points_reproject; //重投影点
cout << "\n\t每幅图像的标定误差:\n";
fout << "每幅图像的标定误差：\n";
for (int i = 0; i < nums; i++)
{
vector<cv::Point3f> points3D_per_image = points3D_all_images[i];//标准的标定板图像
//通过之前标定得到的相机内外参，对三维点进行重投影
cv::projectPoints(points3D_per_image, rotationMat[i], translationMat[i], camera_K, distCoeffs, points_reproject);
//计算两者之间的误差,转为双通道，一个通道存储x，一个通道存储y
vector<cv::Point2f> detect_points = points_all_images[i]; //提取到的图像角点
cv::Mat detect_points_Mat = cv::Mat(1, detect_points.size(), CV_32FC2); //变为1*144的矩阵,2通道保存提取角点的像素坐标
cv::Mat points_reproject_Mat = cv::Mat(1, points_reproject.size(), CV_32FC2); //2通道保存投影角点的像素坐标
for (int j = 0; j < detect_points.size(); j++)
{
detect_points_Mat.at<cv::Vec2f>(0, j) = cv::Vec2f(detect_points[j].x, detect_points[j].y);
points_reproject_Mat.at<cv::Vec2f>(0, j) = cv::Vec2f(points_reproject[j].x, points_reproject[j].y);
}
err = cv::norm(points_reproject_Mat, detect_points_Mat, cv::NormTypes::NORM_L2);
total_err += err /= point_counts;
cout << "第" << i + 1 << "幅图像的平均误差为： " << err << "像素" << endl;
fout << "第" << i + 1 << "幅图像的平均误差为： " << err << "像素" << endl;
}
cout << "总体平均误差为： " << total_err / nums << "像素" << endl;
fout << "总体平均误差为： " << total_err / nums << "像素" << endl;
cout << "评价完成！" << endl;
//将标定结果写入txt文件
Mat rotate_Mat = Mat(3, 3, CV_32FC1, cv::Scalar::all(0)); //保存旋转矩阵
cout << "\n相机内参数矩阵:" << endl;
cout << camera_K << endl << endl;
fout << "\n相机内参数矩阵:" << endl;
fout << camera_K << endl << endl;
cout << "畸变系数：\n";
cout << distCoeffs << endl << endl << endl;
fout << "畸变系数：\n";
fout << distCoeffs << endl << endl << endl;
for (int i = 0; i < nums; i++)
{
cv::Rodrigues(rotationMat[i], rotate_Mat); //将旋转向量通过罗德里格斯公式转换为旋转矩阵
fout << "第" << i + 1 << "幅图像的旋转矩阵为：" << endl;
fout << rotate_Mat << endl;
fout << "第" << i + 1 << "幅图像的平移向量为：" << endl;
fout << translationMat[i] << endl
<< endl;
}
fout << endl;
fout.close();
system("pause");
return 0;
}The code running shows as:
