Camera coordinate system, world coordinate system, pixel coordinate system conversion, and Fov conversion of OPENGLDEFocal Length and Opengl

Purpose: Understand how the camera works, rendering and depth maps captured by the camera

Recently, the parameter relationship between camera parameters and opengl rendered images has been studied. The project matrix will be encountered in the process of going from the camera coordinate system to the pixel coordinate system.

Theoretical Camera Transformation

Camera-based learning theory knowledge. Normally, we use camera parameters to build a project and generally need to go through the following two processes:
camera coordinate system -> image coordinate system -> pixel coordinate system Suppose we have obtained a vertex $p(x_c )$
in the camera coordinate system $,y_c,z_c)$ , transformed into a vertex p on the image $p(x_i,y_i)$ . Its transformation matrix is $M_{proj}$
Below we will introduce a few steps:

Camera coordinate system to image coordinate system $M_{p2c}$
At the same location, different cameras will get different pictures. It can be seen that this process is generally related to the parameters of the camera. Some parameters of the camera are generally in the camera manual, and you can also find the corresponding style of documentation on the Internet, and some even in the configuration file (for example, the camera parameters of the image camera in the scanner). In order to understand the function of these parameters, we need to understand the principle of camera imaging (small hole imaging).

The coordinate system in the image is $(O^{'} x, O^{'} y)$ , the camera coordinate system is $O_cx_c, O_cy_c , O_cz_c)$ , for easy understanding, convert the visual view. Because $(O^{'} x, O^{'} y)$ 和 $O_cx_c, O_cy_c , O_cz_c)$ 在 $O_cx_c, O_cy_c)$ parallel. Therefore, according to the geometric symmetry, the image coordinate system is moved to the mirrored position, as shown in the figure below. where $\angle{O_cAB}$ is a right angle. The picture is not very accurate.

insert image description here
Because similar triangles (1) :
$\bigtriangleup ABO_c \sim \bigtriangleup oCO_c$
Get the following formula:
$\frac{O_co}{O_cZ_c} = \frac{oC}{AB} = \frac{O_cC}{O_cB} \space \space \space (1)$
Another expression (because $O_c$ is the origin , changed the symbol) as follows:
$\frac{O_co}{O_cZ_c} = \frac{f}{Z_c} \space \space \space (2)$
$\frac{oC}{AB} = \frac{x}{X_c} \space \space \space (3)$

Because similar triangles (2) :
$\bigtriangleup ABO_c \sim \bigtriangleup oCO_c$
The following formula is obtained:
$\frac{O_cC}{O_cB} = \frac{Cp}{BP} \space \space \space (4)$
Another expression (because $O_c$ is the origin , changed the symbol) as follows:
$\frac{Cp}{BP} = \frac{y}{Y_c} \space \space \space(5)$
because $(1) (2) (3) (4) (5)$ The formula gets:
$\frac{f}{Z_c} = \frac{x}{X_c} = \frac{ y}{Y_c}$
Further converted into the following formula:
$x=\frac{f*X_c}{Z_c} \space \space \space(6)$
$y=\frac{f*Y_c}{Z_c} \space \space \space(7)$
among them $(6), (7)$ The formula is written in the form of matrix as follows:
$\begin{bmatrix} x\\ y \\ z \end{bmatrix} = \begin{bmatrix} f & 0 & 0 & 0 \\ 0 & f & 0 & 0 \\ 0 & 0 & 1 & 0 \end{bmatrix} \begin{bmatrix} X_c\\ Y_c \\ Z_c \\ 1 \end{bmatrix}$
The final matrix equation $M_{p2c}$ Default:
$M_{p2c}= \begin{bmatrix} f & 0 & 0 & 0 \\ 0 & f & 0 & 0 \\ 0&0&1&0\end{bmatrix}$
The shorthand formula can be obtained:
$\begin{bmatrix} x\\ y \\ z \end{bmatrix} = M_{p2c}\begin{bmatrix} X_c \\ Y_c \\ Z_c \\ 1 \end{bmatrix}$
$\begin{bmatrix} x\\ y \\ z \end{bmatrix}$ obtained at this time $⎣ ⎡ x y z ⎦ ⎤$ is a physical unit. It is not in units of pixels, so it needs to be converted into units of pixels, it needs the specific physical length of each pixel. The sensor parameter of the camera is required. The camera coordinates to pixel coordinates are described below.

Convert image coordinate system to pixel coordinate system $M_{i2p}$
It is necessary to convert the camera coordinate system into a pixel coordinate system, which involves two issues,
1) Two coordinate system issues. In the image coordinate system, generally the origin is at the center. However, the origin of our traditional pixel display coordinate system is the upper left corner of the image, and this vector needs to be translated.
2) Unit conversion problem: It is necessary to know how many physical units (usually mm) each pixel has, and the physical unit is millimeter. For both conversions, you need to know $d_x,d_y$ Indicate how many mm each column and each row represent, and represent the width of a column $\space pixel = d_x \space mm$ ; the width of a line $\space pixel = d_y \space mm$ , in general $d_x=d_y$ . Therefore The pixel represented by the $x$ $\frac{x}{dx}$ , the corresponding The pixel represented by $the y$ $\frac{y}{dy}$
The two coordinate systems shown in the figure are as follows:

the following formula is obtained by translating the coordinate system:
$\frac{x}{d_x}+u_0 \space \space \space (8)$
$\frac{y}{d_y}+v_0 \space \space \space (9)$
among them $(8), (9)$ The formula is written in the form of a matrix as follows:
$\begin{bmatrix} u\\ v \\1 \end{ bmatrix}= \begin{bmatrix} \frac{1}{d_x}&0&u_0 \\ \frac{1}{d_y}&0&v_0 \\0&0&1 \end{bmatrix}\begin{bmatrix} u\\ v \\1 \end {bmatrix}$
The obtained matrix formula is:
$M_{i2p}= \begin{bmatrix} \frac{1}{d_x}&0&u_0 \\ \frac{1 }{d_y}&0&v_0 \\0&0&1 \end{bmatrix}$

Summarize the above two transformations to get:
$M_{proj}= \begin{bmatrix} \ frac{1}{d_x}&0&u_0 \\ \frac{1}{d_y}&0&v_0 \\0&0&1 \end{bmatrix} \begin{bmatrix} f & 0 & 0 & 0 \\ 0 & f & 0 & 0 \\ 0 & 0 & 1 & 0 \end{bmatrix}$
Finally, the formula projection matrix is obtained:
$M_{proj}= \begin{bmatrix } \frac{f}{d_x}& 0 & u_0 & 0 \\ 0 & \frac{f}{d_y} & v_0 & 0 \\ 0 & 0 & 1 & 0 \end{bmatrix} = \begin{bmatrix } f_x & 0 & u_0 & 0 \\ 0 & f_x & v_0 & 0 \\ 0 & 0 & 1 & 0 \\end{bmatrix}$
Understand the relationship through the formula, get $f_x=\frac{f}{d_x}$ , it indicates how many pixels the focal length has (this is the pixel unit), similarly $f_y=\frac{f}{d_y}$ It also indicates how many pixels there are. In general, $d_x=d_y$ in the above figure $f$ 。

The summary is as follows:
$M_{proj}= \begin{bmatrix} f_x & 0 & u_0 & 0 \\ 0 & f_x & v_0 & 0 \\ 0 & 0 & 1 & 0 \end{bmatrix}$

Opengl coordinate rendering theory

Camera-based OPENGL theoretical knowledge. Normally, we use camera parameters to build a project that requires more fov, aspect ratio, near, and far. Among them, we found that there are 4 parameters in the specific rendering, which can accurately render the parameters required by the space, and they are easy to understand, and they cut a space for normalization operations and easy rendering calculations. For details, please Baidu learnopengl about camera settings. It sets about the above 4 parameters. It has two kinds of projects, and I will introduce their principles below:
insert image description here
If you don't convert them into fov, aspect ratio, near, far, you can directly project the matrix and complete it through glFrustum. But in learning opengl rendering, gluPerspective is usually used, and the parameters it uses are fov, aspect ratio, near, far. We can directly set the project matrix in the shader to get the rendered project matrix. It also imitates gluPerspective to generate the project matrix. At the same time, it can also be converted into the above four, and opengl's gluPerspective can help you generate the project matrix. Either way works fine.
1) The parameters of the camera are converted into four parameters of fov, aspect ratio, near, and far.
First understand these four parameters:
1) Field of view (FOV), aspect ratio Aspect
ratio is the x/y ratio of the final displayed image
FOV: Indicates the angle of the open field of view, see the figure below.
2) near and far represent the distance between the far and near tangent planes and the origin, see the two parallel planes in the figure.
insert image description here
The relationship in the above picture can be seen:

if you need to render a picture that is consistent with the image, the picture is expressed as img, and we set the aspect ratio and FOV through the image
According to the definition, ratio is the x/y ratio of the final displayed image, and
the calculated formula is as follows:
$aspect=\frac{img.cols}{img.rows}$
Bringing into the above formula, top corresponds to half of the y-axis, that is, the pixelwidth in the pixel coordinate system, which is
$\frac{pixelwidth}{2.0}$
Among them, near corresponds to the focal length (pixel coordinate system), which is in the pixel coordinate system. The unit is unified.
Later converted to the following formula:
$FOV=2.0*atanf({\frac{pixelwidth}{2.0*focalLength}})*\frac{\pi}{180.0 }$
The distance set for near and far can be as close or as far as possible.
Its matrix is as follows:
insert image description here
After setting, the rendered image is consistent with the image captured by the camera.

Subsequent to join the derivation process.