Machine Vision Model - Projection Matrix

1 Overview

Machine vision is to use machines instead of human eyes and brains for measurement and judgment. The basic process of the machine vision system is to obtain the image of the target, perform analysis operations such as recognition, feature extraction, classification, and mathematical operations on the image, and control or make decisions on the corresponding system according to the analysis and calculation results of the image.
In many machine vision applications, machine vision measurement is required, that is, to obtain the physical position of the target in the actual space according to the image of the target, such as grabbing manipulators, walking robots, SLAM, etc.
To obtain the physical space position of the target according to the target pixel position in the image, we need to first have a mapping relationship between the image pixel coordinates and the physical space coordinates, that is, abstract the optical imaging process into a mathematical formula, which can express the spatial position The mathematical formula of how to map to the pixel position of the image is the so-called machine vision imaging model. This article discusses the mechanism of this model.

2 small hole imaging

Machine vision imaging adopts a small hole imaging model, as shown in the following figure
Figure 1:

Simplified again to the following figure
Figure 2: XX

in the figure$X$ is a point in space,$x$ is the imaging point of the space point in the image,$C$ is the optical center of the lens (camera centre), as can be seen from the figure,$C$、$x$、$The\; three\; points\; of\; X$ are collinear.
Optical center$The\; distance\; from\; C$ to the image plane is the focal length$f$ .
The following coordinate systems and their interrelationships are derived based on this pinhole imaging model.

3 coordinate system

When it comes to machine vision measurement models, it is necessary to first understand several coordinate systems involved in the entire model.

3.1 Pixel coordinate system uov

That is, the coordinate system where each pixel point in the image sits, as shown in the figure below uov.
Figure 3:

This coordinate system is a two-dimensional coordinate system, the abscissa is the image width direction, the ordinate is the image height direction, the origin is in the upper left corner, and the coordinate axis unit is pixel, which corresponds to the pixel point of the image.

3.2 Image coordinate system xoy

That is, the image sensor (such as CMOS, CCD) coordinate system, as shown in the figure below xoy.
Figure 4:

This coordinate system is also a two-dimensional coordinate system, the abscissa is the width direction of the sensor, the ordinate is the height direction of the sensor, the origin is at the center of the sensor, and the unit of the coordinate axis is mm (set according to actual needs, m, mm, ... …), and the following coordinate systems are also in the same unit, which will not be explained here.
Combined with the pixel coordinate system, we can get the following figure Figure
5:

From this figure, we can get the mapping relationship between the pixel coordinate system uov and the image coordinate system xoy, namely:
$$\begin{gathered} u=x/dx+u_0 \\ v=-y/dy+v0 \end{gathered}$$
in formula:
$u_0$、$v_0$——The pixel coordinates of the center of the image (usually half of the horizontal and vertical resolution of the image, but if the position of the lens and the sensor is misaligned, it is not half), the unit is pixel; dx
$dx$、$dy$ ——The horizontal and vertical dimensions of the sensor unit (that is, the pixel size), in mm/pixel, usually the pixel is a square, so$dx=dy$
The above formula is written in the form of a homogeneous matrix, which isthe conversion relationship between the pixel coordinate system and the image coordinate system
$$\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=\left[\begin{array}{ccc}1/dx& 0& u_0\\ 0& -1/dy& v_0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{c}x\\ y\\ 1\end{array}\right]$$
Note that a negative sign is added to the y direction in the above formula, because the pixel coordinate system uov is a left-handed coordinate system, but the three-dimensional coordinate system to be discussed later uses a right-handed coordinate system, so here the image coordinate system xoy is directly set to the right-handed coordinate system The negative sign is used to transform the y-axis direction.

3.3 Camera coordinate system $O_C{X}_C{Y}_C{Z}_C$

Set a three-dimensional coordinate system on the camera lens, as shown in the figure below, the origin is at the optical center, the X-axis and Y-axis are parallel to the x-axis and y-axis of the image coordinate system respectively, and the Z-axis points to the object side (Note: This is what I am used to One camera coordinate system definition, another more common one is that Z points to the object space, and XY points to the opposite of what I have in the picture. The advantage is that there are no two negative signs of my internal reference in the internal reference).
Figure 6:

According to the above small hole imaging model, we can get the projection relationship in the YOZ (YCZ) plane, as shown in the figure below (XOZ plane is the same)
Figure 7:

In the above figure, according to similar triangles, $\frac{f}{Z_C}=\frac{y}{Y_C}$以及$\frac{f}{Z_C}=\frac{x}{X_C}$, so we can write the conversion relationship between the camera coordinate system and the image coordinate system , we directly write it as the homogeneous coordinate form
$$\left[\begin{array}{c}x\\ y\\ 1\end{array}\right]=\left[\begin{array}{cccc}f/Z_C& 0& 0& 0\\ 0& f/Z_C& 0& 0\\ 0& 0& 1/Z_C& 0\end{array}\right]\left[\begin{array}{c}{X}_C\\ Y_C\\ Z_C\\ 1\end{array}\right]=\frac{1}{Z_C}\left[\begin{array}{cccc}f& 0& 0& 0\\ 0& f& 0& 0\\ 0& 0& 1& 0\end{array}\right]\left[\begin{array}{c}{X}_C\\ Y_C\\ Z_C\\ 1\end{array}\right]$$
Where: $f$ - the focal length of the lens, some formulas in the literature will divide the focal length into fx f_xin the X and Y directions$f_x$、$f_y$
Determine the equivalent of the equivalent equation of the equation
$$\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=\left[\begin{array}{ccc}1/dx& 0& u_0\\ 0& -1/dy& v_0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{c}x\\ y\\ 1\end{array}\right]=\frac{1}{Z_C}\left[\begin{array}{ccc}1/dx& 0& u_0\\ 0& -1/dy& v_0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{cccc}f& 0& 0& 0\\ 0& f& 0& 0\\ 0& 0& 1& 0\end{array}\right]\left[\begin{array}{c}{X}_C\\ Y_C\\ Z_C\\ 1\end{array}\right]$$
We can get the conversion relationship between the camera coordinate system and the pixel coordinate system as follows
$$\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=\frac{1}{Z_C}\left[\begin{array}{cccc}f/dx& 0& u_0& 0\\ 0& -f/dy& v_0& 0\\ 0& 0& 1& 0\end{array}\right]\left[\begin{array}{c}{X}_C\\ Y_C\\ Z_C\\ 1\end{array}\right]$$
We use $M_1$Represents the matrix in the formula. At the same time, we can see from Figure 6 above that the image is actually observed from the back of the image sensor in the figure. Therefore, a negative sign needs to be added to the X direction, and M 1 = [ − f / dx 0
$$M_1=\left[\begin{array}{cccc}-f/dx& 0& u_0& 0\\ 0& -f/dy& v_0& 0\\ 0& 0& 1& 0\end{array}\right]$$
The parameters in this matrix are only related to the lens focal length $f$ , pixel size$dxdy$ , center pixel$u_0{v}_0$Related, these are the internal parameters of the camera and lens. After the camera and lens are determined, the matrix is ​​determined, so it is called the internal reference matrix .

3.4 OWXWYWZW $O_W{X}_W{Y}_W{Z}_W$

The world coordinate system is the absolute coordinate system of the system, and it is also a three-dimensional coordinate system. The origin and coordinate axis directions are selected according to our needs.
Figure 8:

As a rigid body, the camera has a pose in the world coordinate system - position and pose, the position is the translation of the camera (the origin of the camera coordinate system) relative to the origin of the world coordinate system, using a 3×1 translation vector TC $T_C$Expression, attitude is the rotation of the camera (camera coordinate system) relative to the world coordinate system, using a 3×3 rotation matrix $R_C$Expression
Then we can get the relationship between the camera coordinate system and the world coordinate system
$$\left[\begin{array}{c}{X}_W\\ Y_W\\ Z_W\\ 1\end{array}\right]=\left[\begin{array}{cc}{R}_C& T_C\\ 0_{1\times 3}& 1\end{array}\right]\left[\begin{array}{c}{X}_C\\ Y_C\\ Z_C\\ 1\end{array}\right]$$
反过来
$$\left[\begin{array}{c}{X}_C\\ Y_C\\ Z_C\\ 1\end{array}\right]={\left[\begin{array}{cc}{R}_C& T_C\\ 0_{1\times 3}& 1\end{array}\right]}^{-1}\left[\begin{array}{c}{X}_W\\ Y_W\\ Z_W\\ 1\end{array}\right]=M_2\left[\begin{array}{c}{X}_W\\ Y_W\\ Z_W\\ 1\end{array}\right]$$
This is the transformation relationship between the world coordinate system and the camera coordinate system , where the matrix $M_2$It is related to the pose of the camera and is called the extrinsic parameter matrix .
Substituting the previous conversion formula between the pixel coordinate system and the camera coordinate system, we get
$$\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=\frac{1}{Z_C}\left[\begin{array}{cccc}-f/dx& 0& u_0& 0\\ 0& -f/dy& v_0& 0\\ 0& 0& 1& 0\end{array}\right]\left[\begin{array}{c}{X}_C\\ Y_C\\ Z_C\\ 1\end{array}\right]=\frac{1}{Z_C}\left[\begin{array}{cccc}-f/dx& 0& u_0& 0\\ 0& -f/dy& v_0& 0\\ 0& 0& 1& 0\end{array}\right]M_2\left[\begin{array}{c}{X}_W\\ Y_W\\ Z_W\\ 1\end{array}\right]$$

4 Machine Vision Projection Matrix

So far, we have obtained the mapping relationship between the pixel coordinate system and the world coordinate system , that is, the machine vision projection matrix
$$\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=\frac{1}{Z_C}{M}_1{M}_2\left[\begin{array}{c}{X}_W\\ Y_W\\ Z_W\\ 1\end{array}\right]$$
Where:
$Z_C$——Z coordinate M 1 M_1 of the space point in the camera coordinate system
$M_1$——Horizontal matrix,3×4 matrix, $M_1=\left[\begin{array}{cccc}-f/dx& 0& u_0& 0\\ 0& -f/dy& v_0& 0\\ 0& 0& 1& 0\end{array}\right]$
$M_2$——Specialized square,4×4 square, $M_2=\left[\begin{array}{cccc}{r}_{11}& r_{12}& r_{13}& t_x\\ r_{21}& r_{22}& r_{23}& t_y\\ r_{31}& r_{32}& r_{33}& t_z\\ 0& 0& 0& 1\end{array}\right]$
After the model is established, how to obtain the parameters in it? This involves the next question: "Machine Vision-Camera Calibration"