[Image] [OpenCV fisheye correction] 1. Interpretation of the main API

One, the code flow

There are many codes for fisheye camera correction with OpenCV. You can search on the Internet or download the code of the blogger .

However, no matter which program you refer to, the main APIs used are always these three:

- fisheye::calibrate();  //标定相机，获取内参、外参、畸变系数   
- fisheye::initUndistortRectifyMap();  //利用内参和畸变系数，获取重投影矩阵
- remap(); //根据重投影矩阵，对畸变图像进行重投影，获得矫正图

The function of each function will be briefly introduced below.

1. fisheye::calibrate();

fisheye::calibrateThe function declaration in OpenCV is as follows:

double cv::fisheye::calibrate	(	InputArrayOfArrays 	  objectPoints,
									InputArrayOfArrays 	  imagePoints,
									const Size & 	      image_size,
									InputOutputArray 	  K,
									InputOutputArray 	  D,
									OutputArrayOfArrays   rvecs,
									OutputArrayOfArrays   tvecs,
									int 		          flags = 0,
									TermCriteria 	      criteria = TermCriteria(TermCriteria::COUNT+TermCriteria::EPS, 100, DBL_EPSILON) 
)	

There are many articles about the function of each formal parameter on the Internet, so I won't repeat it in this article.

Here is only a brief introduction to its input and output :

enter:

• One/multiple checkerboard images with fisheye distortion
• Other parameters

Output:

• Internal Reference K
• Rotation vector rvecs
• Translation vector tvecs
• Distortion coefficient D

Regarding the functions of camera internal parameters and external parameters (rotation vector + translation vector) , readers are asked to go online to learn the conversion relationship between world coordinate system, camera coordinate system, image coordinate system, and pixel coordinate system . This article will not repeat them.

Only the following formula is given here:

The above formula is valid for a normal, distortion- free camera, so the distortion coefficient D is not used .

The distortion of the fisheye camera occurs during the conversion process from the camera coordinate system to the pixel coordinate system .

2. fisheye::initUndistortRectifyMap();

fisheye::initUndistortRectifyMap()The function of is to deduce the pixel position (x, y) in the distortion image corresponding to the pixel position (i, j) of the undistorted image, and then copy the (x, y) in the distortion image to the (x, y) in the distortion image. i, j), the corrected image is obtained.

fisheye::initUndistortRectifyMap()The function declaration in OpenCV is as follows:

void cv::fisheye::initUndistortRectifyMap	(	InputArray 	K,
												InputArray 	D,
												InputArray 	R,
												InputArray 	P,
												const cv::Size & 	size,
												int 	m1type,
												OutputArray 	map1,
												OutputArray 	map2 
)	

There are many articles about the function of each formal parameter on the Internet, so I won't repeat it in this article.

Here is only a brief introduction to its input and output :

enter:

• Internal parameter matrix K
• Distortion coefficient D
• R is not a rotation vector/matrix. When correcting a monocular fisheye camera, R defaults to the identity matrix
• If you want to make the internal reference of the corrected image corresponding to the original camera internal reference different, set P as the new internal reference; otherwise, P is the same as the original internal reference
• The type of the output matrix map1 m1type
• Other parameters

Output:

• map1
• map2

When we usually set the parameter m1typeto CV_16SC2 , the most important output at this time is map1 :

According to CV_16SC2, map1 is a 2-channel matrix at this time , and each point (i, j) is a 2-dimensional vector, including:

• (i, j) [0]
• (i, j) [1]

Now we order x = map1(i, j)[0], y= map1(i, j)[1].

Well, iand j, xand ythe meaning is:

(x, y)The pixel with coordinate in the distortion map should be in (i, j)position in the undistorted map .

In this way, the pixels at (x, y) of the distortion map are copied to (i, j), and the corrected image is obtained.

That is as shown in the figure below:

At this time, map2 is used for interpolation to make the corrected image clearer. This article will not introduce it.

---

Note:

If the parameter is m1typeset to CV_32FC1 , then map1 is no longer a two-channel matrix, but a one-channel matrix .

At this time map1 saved m1typeas CV_16SC2 when map1 of 0 channel, MAP2 save m1typeas CV_16SC2 when map1 of a channel.

That is, at this time x = map1(i, j)，y = map2(i, j), then there is:

(x, y)The pixel with coordinate in the distortion map should be in (i, j)position in the undistorted map .

---

3. remap()

remap()The function declaration in OpenCV is as follows:

void cv::remap	(	InputArray 	src,
OutputArray 	dst,
InputArray 	map1,
InputArray 	map2,
int 	interpolation,
int 	borderMode = BORDER_CONSTANT,
const Scalar & 	borderValue = Scalar() 
)	

There are many articles about the function of each formal parameter on the Internet, so I won't repeat it in this article.

Here is only a brief introduction to its input and output :

enter:

• Fisheye distortion map src
• map1
• map2
• Other parameters

Output:

• Corrected image dst

remap()The function is actually fisheye::initUndistortRectifyMap()generated based on it map1. As described in the figure above, the point in the distortion map src is (map1(i, j)[0], map1(i, j)[1])projected to the output image dst (i, j); and then map2some clarification processing is done to complete the correction.