[Image] [OpenCV fish-eye correction] 3. Explanation of the principle of fish-eye correction

Reference materials:

链接：https://pan.baidu.com/s/19BK9HbRBYtFCjdR0qSIv2Q 
提取码：eu2s 

According to the previous two articles, we already know that the most important function of fisheye correction is fisheye::initUndistortRectifyMap()that it can get the map1 matrix, and its function is:

map1 2 is a channel matrix, in which the two-dimensional vector elements (i, j) at the (u, v) = (map1(i, j)[0], map1(i, j)[1])meanings as follows:
the distortion of the image (u, v) = (map1(i, j)[0], map1(i, j)[1])elements to copy (i, j), the distortion-free image is obtained.

So, fisheye::initUndistortRectifyMap()how is it (i, j)calculated (u, v)based on it?
OpenCV officially gives the following formula:

Wherein the (i, j)pixel graph orthoscopic, and the formula ① (i, j, 1)is (i, j)homogeneous coordinates;

The iR of formula ① is the inverse of the camera's internal parameters .

⑦ in the last formula (u, v)is undistorted figure (i, j)corresponding to a position of a distortion point of FIG (u, v).

---

The following is a detailed introduction to the above announcement.

1. Explanation of the principle of fisheye correction

1. Convert pixel coordinates to camera coordinates

Before reading this section, please learn the conversion relationship between the world coordinate system, camera coordinate system, image coordinate system, and pixel coordinate system . This section assumes that the reader has mastered the basic knowledge.

First of all, we know that the conversion relationship from the camera coordinate system to the pixel coordinate system is as follows:

Note:

In the following, “the coordinates of a point in the real three-dimensional world in the camera coordinate system” will be abbreviated as “camera coordinates”; the “coordinates in the pixel coordinate system in a two-dimensional picture” will be abbreviated as “pixel coordinates”.

In the above formula ①, it iRis the inverse of the internal parameter matrix, which can be obtained by formula ① [Xc, Yc, Zc], which is actually the normalized coordinates of the real camera coordinates, namely:

However, we used [Xc, Yc, Zc], but only used the proportional relationship between them. Therefore, it doesn't really matter whether it is normalized or not, because the proportional relationship is the same.

So far, according to the pixel coordinates in the undistorted image, we have (i, j)obtained the normalized camera coordinates that it should correspond to under the undistorted conditions (Xc, Yc, Zc).

2. Correspondence between the undistorted camera coordinates and the distorted camera coordinates

Suppose that in the real world (x, y, z)shown in a position below the camera coordinate system, then, according to the above formula ②③④ , we get theta = atan(r)actually incident angle , i.e., 点(x, y, z)the line connecting the center of the camera and the camera optical axis (Z axis) of the clamp angle. as follows:

In the figure, the lower left corner theta1is atan(r)calculated, and the right theta2is 点(x, y, z)the incident angle, which theta1is equal to the lower left corner .

Some readers may ask, the real camera coordinates are marked in the figure, but the normalized camera coordinates in the above formula are the thetasame. Can they be calculated the same? The answer is yes. Observation shows that what we use is only the proportional relationship between x, y, and z, regardless of whether they are normalized or not, their proportional relationship is the same.

So far, whether it is pixel coordinates (i, j), camera coordinates (x, y, z), or incident angle theta = theta1 = theta2, we have calculated the values ​​without distortion.

Then, the next step will be the dividing point between distortion and distortion!

---

The Juho Kannala and Sami S. Brandt masterpiece "A and the Generic Camera Calibration Method, for the Model Conventional, Wide-Angle, Fish and an Eye-Lenses" , we may be made without distortion of the angle of incidencetheta , with the distortion coefficient after obtain distortion r ! In order to distinguish this r from the undistorted r , we record the distorted r as r_d (d stands for distortion).

即，r_d = k1 · theta + k2 · theta ^ 3 + k3 · theta ^ 5 + k4 · theta ^ 7 + k5 · theta ^ 9

This is the formula ⑤.

At this point, we have got r_d in the case of distortion.

The meaning of the distortion point is as follows:

In the case of no distortion, in the above figure, the black pixels in the normal image should be projected from the black normal points in the real world;

However, in the case of fisheye distortion, the black pixels in the undistorted image in the above figure should be projected to correspond to the red points in the real world.

Then, we only need to find the position of the red point in the distortion map, and then place it in the position of the black undistorted pixel.

Now, we can get the corresponding red dots r_d, and black dots corresponding r, then make a scale factor scale = r_d / r, then scale · ait should be a red dot camera coordinate in the x direction, scale · bit should be red camera coordinate point in the y-direction.

The camera coordinates of the red point are obtained, and based on it 像素坐标 = 内参 x 相机坐标, the pixel coordinates in the distorted image corresponding to the red point in the real world can be obtained , that is, formula ⑦. As follows: **So far, we have started from the very beginning , after a derivation, and calculated its corresponding , as long as the (u, v) pixel is copied to (i, j), the corrected image can be obtained.

无畸变图中的像素点(i, j)畸变图中的像素点(u, v)

Note:

In fact, the blogger is not very clear about the principle and function of the scale factor. As for why " scale · ait should be the camera coordinates of the red point in the x direction, but scale · bit should be the camera coordinates of the red point in the y direction", he didn't understand too much. But there must be a principle to follow, and I hope that readers here can answer their questions in the comment section.