Optical center offset correction based on plane assumption in fisheye camera

This article is mainly aboutHugin open source software<libpano< a Principle analysis and summary of functions such as plane_transfer_to_camera() and plane_transfer_from_camera() in i=4>/math.c>.

       In the article "Panorama Correction of Fisheye Image", we have introduced in detail how to map a fisheye image To the corresponding 360-degree panoramic image, the basic model is shown in Figure 1. Three-dimensional coordinates $OXYZ$ Indication is where we are Instant physical space, inside $The\; Y$ axis points in the direction we are facing, $Z$ 轴oriented sky, $The\; X$ axis points to our right hand side. And take the coordinate origin $O$ The spherical surface with the center of the sphere can be regarded as the fisheye lens used for photography, $The\; O$ point is the optical center of the lens. Expanding the sphere into a plane is the panoramic image we need. The following two-dimensional plane $O^{\prime }{X}_f{AND}_f$ is the imaging plane of the camera. When shooting, start from $The\; ray\; starting\; from\; the\; point\; P$ follows the straight line $OP$ reach the light $O$ point, and refraction occurs. The refraction characteristics are determined by the design model and manufacturing process of the lens. The linear model is usually used, that is, the refraction angle is linearly proportional to the incident angle. Finally, the refracted light ray follows the straight line $O{P}_1$ reaches the imaging plane, because the image on the imaging plane is 180 degrees upside down at this time. For convenience, we usually use the point $P_1$ Point symmetric about the origin $P_2$. All these points $P_2$ constitutes the fisheye image we obtained. The main task of correcting the fisheye image is to correct the spherical $P$ The point coordinates are calculated as the corresponding coordinates on the fisheye image $P_2$, thereby obtaining the point $The\; pixel\; value\; of\; P$.

Figure 1 Schematic diagram of fisheye projection

       As can be seen from Figure 1, the mapping from the spherical surface to the imaging plane has certain limitations, that is, in reality we cannot make a lens with a complete spherical surface, so that we cannot obtain a complete 360 ​​degrees in one shot. perspective. However, it is not difficult to make a fisheye lens with an angle of view greater than 180 degrees. If we use such a lens, let it face $Take\; an\; image\; in\; the\; positive\; direction\; of\; the\; Y$ axis, and then move around $Z$ Rotation 180 degrees, arrival in the morning $Y$ Take another image in the negative direction of the axis. In theory, we can obtain an image with a 360-degree viewing angle covering the entire spherical surface. This is what multi-eye fisheye image stitching does. But this method also has a fatal flaw, that is, we cannot capture panoramic images of sports scenes, because the images from different perspectives in this method come from different moments. At this time, we can easily think of using two fisheye cameras with the same parameters, facing $Y$ is installed in the positive and negative directions of the axis. After synchronization, it seems that the content of the 360-degree view can be obtained at the same time. But the problem arises again, because the optical centers of the two fisheye lenses are not at the same physical space point. We call it the optical center offset. That is, there are actually two spheres at different positions in Figure 1. Somewhere in the space When two points reach the two spheres along the straight line passing through the center of the sphere, their longitude and latitude coordinates relative to the centers of the two spheres are different. Therefore, we cannot directly copy the content on one sphere to another, otherwise there will be a gap at the interface of the images, which is the so-called parallax. This problem is especially obvious in close-up shooting. It is often difficult to perfectly solve the problem of parallax, and in most cases even impossible, because the mapping from a three-dimensional physical space to a two-dimensional image plane itself is a process of information loss. Therefore, we can only solve some specific problems. For example, suppose what we are shooting is all on a flat surface?

Figure 2 Schematic diagram of optical center offset correction

       Figure 2 shows the process of optical center offset correction of the fisheye lens based on the plane assumption, mainly referring to the open source software Hugin The method used, and a rough summary of its principles by reading the code. For ease of understanding, this is simplified to a two-dimensional case, and the principle is also applicable to three-dimensional space. Its point $O$ is the optical center position of the main camera, which is the origin of the physical coordinate system in Figure 1. Point $P_1$ is the position of the optical center of the secondary camera lens, $O{P}_1$ is the optical center offset of the two lenses. Note the point $O$ 和点 $P_1$The direction of the coordinate system established on is consistent. The yaw, pitch and rotation of the lens can be processed through other steps, so only the translation of the optical center is involved here. The coordinates of the points in Figure 2 are all based on point $O$ It's very original.

       Suppose there is a control point (that is, the point used for parameter optimization) on the fisheye image of the secondary camera, and its spherical coordinates obtained after inverse mapping of the fisheye projection model are$P_2$, then the actual perspective projection point corresponding to the control point should be $P_3$. We can only know that the real spatial coordinate point corresponding to the control point is at point $P_1$ Sum $P_3$, but the corresponding distance cannot be known. At this time we need some assumptions, such as assuming that at point $P_0$There is a plane at , such as a wall, with the straight line $O{P}_0$ Vertical, inside $P_0$ From point $O$ is the known latitude and longitude coordinates on the sphere of the center of the sphere. Note that we do not need to know the true distance of the plane. When there is no object blocking in front of the plane, which is the foreground, we can know $P_3$ actually comes from the straight line $P_1{P}_3$The intersection of and the plane $P_5$, where the straight line $P_1{P}_3$ 和 $O{P}_0$The angle between cannot be greater than or equal to 90 degrees. In this way, the point $P_5$ Straight line $O{P}_5$ $O$ The intersection point on the sphere with the center of the sphere $The\; latitude\; and\; longitude\; coordinates\; of\; P$.

       Based on the similarity of triangles, we have

$$P_{1}A=\frac{{\overrightarrow{P_1{P}_3}}^T\cdot \overrightarrow{P_1{P}_4}}{P_1{P}_4}=\frac{{\overrightarrow{O{P}_2}}^T\cdot \overrightarrow{O{P}_0}}{O{P}_0},\text{\hspace{1em}(1)}$$

$$P_{1}B=\frac{{\overrightarrow{P_1{P}_0}}^T\cdot \overrightarrow{P_1{P}_4}}{P_1{P}_4}=\frac{{\left(\overrightarrow{O{P}_0}-\overrightarrow{O{P}_1}\right)}^T\cdot \overrightarrow{O{P}_0}}{O{P}_0},\text{\hspace{1em}(2)}$$

$$\frac{P_1{P}_3}{P_1{P}_5}=\frac{P_{1}A}{P_{1}B}\Rightarrow \overrightarrow{P_1{P}_5}=\frac{P_{1}B}{P_{1}A}\cdot \overrightarrow{P_1{P}_3}.\text{\hspace{1em}(3)}$$

So,

$$\overrightarrow{O{P}_5}=\overrightarrow{O{P}_1}+\overrightarrow{P_1{P}_5}=\overrightarrow{O{P}_1}+\frac{P_{1}B}{P_{1}A}\cdot \overrightarrow{O{P}_2}=\overrightarrow{O{P}_1}+\frac{{\left(\overrightarrow{O{P}_0}-\overrightarrow{O{P}_1}\right)}^T\cdot \overrightarrow{O{P}_0}}{{\overrightarrow{O{P}_2}}^T\cdot \overrightarrow{O{P}_0}}\cdot \overrightarrow{O{P}_2}.\text{\hspace{1em}(4)}$$

Similarly, if we know $P$ Since $O$ The point is the longitude and latitude coordinates on the sphere with the center of the sphere, then the point $P_5$ Where are you $P_1$ The perspective projection point on the sphere whose point is the center of the sphere $P_3$ 向对于Point $P_1$The coordinates are

$$\frac{O{P}_5}{OP}=\frac{O{P}_0}{{\overrightarrow{OP}}^T\overrightarrow{O{P}_0}/O{P}_0}\Rightarrow {\overrightarrow{OP}}_5=\frac{O{P}_0^2}{{\overrightarrow{OP}}^T\overrightarrow{O{P}_0}}\overrightarrow{OP},\text{\hspace{1em}(5)}$$

$$\overrightarrow{P_1{P}_5}=\overrightarrow{O{P}_5}-\overrightarrow{O{P}_1}\Rightarrow \overrightarrow{O{P}_2}=\overrightarrow{P_1{P}_3}=\overrightarrow{P_1{P}_5}/\Vert \overrightarrow{P_1{P}_5}\Vert .\text{\hspace{1em}(6)}$$

       The premise of the plane hypothesis is that all points come from the same plane, which is why we do not need to know the real distance to the plane (but we need to know the specific orientation). However, this is not always true. On the contrary, in many scene applications it is impossible to have a plane that can cover most of the field of view. Therefore, this method is more suitable for multi-camera stitching of large-area flat images such as wall graffiti. Of course, this is also very suitable for estimating the optical center distance of multi-eye fisheye cameras, because the control points appear in the overlapping fields of view of each lens, and this part of the field of view is relatively small. We can use a checkerboard to Provide a large enough surface. However, even if we have obtained a relatively accurate estimate of the optical center distance, in actual fisheye image correction, if we cannot provide the required plane, it is still difficult to avoid parallax due to the complexity of the sources of spatial points.