[Multi-view geometry series in computer vision] Understanding the pinhole camera model in simple terms

If you review the past and learn the new, you can become a teacher!

1. Reference materials

"Multiple-view Geometry in Computer Vision - Chapter 5" -Richard Hartley, Andrew Zisserman.

2. Introduction to the pinhole model

1. Important concepts

1.1 Projection center/camera center/optical center

projection centercalledcamera center, also known asoptical center. The center of projection is located at the origin of a Euclidean coordinate system.

1.2 Image plane/focus plane

平面 $Z=f$ is calledimage planeorfocus plane。

1.3 Principal axis/primary ray

The vertical line from the center of the camera to the image plane is called the camera'sSpindleorchief ray。

1.4 Main points

The intersection of the principal axis and the image plane is calledmain point。

1.5 Main plane (camera)

The plane parallel to the image plane through the center of the camera is calledmain plane of camera。

2. Camera projection

The reduction from a 3D world to a 2D image is a projection process, in which we lose one dimension. A common way to model this process is to usecentral projection, a ray is drawn from a point in space from the 3D world point to a fixed point ( projection center ) in space. This ray will intersect the specific plane selected as the image plane in space. The intersection of a ray with the image plane represents the image at that point.

Under the pinhole camera model, the 3-dimensional spatial coordinates are $X=(X,Y,Z{)}^T$ 's point$X$ is projected to a point on the image plane, which is the connection point$The\; intersection\; of\; the\; straight\; line\; between\; X$ and the projection center and the image plane. According to similar triangles, you can quickly calculate the point$(X,Y,Z{)}^T$ is mapped to the point$(fX/Z,fY/Z,f{)}^{T.}$ _ After omitting the last image coordinate, the center projection from world coordinates to image coordinates is:
$$(X,Y,Z{)}^T\mapsto (fX/Z,fY/Z{)}^T\left(1\right)$$
This isFrom the 3-dimensional Euclidean space ${\text{AND}}^3$ to 2-dimensional Euclidean space${\text{AND}}^{}$A mapping of${}^2$。

3. Projection matrix

The concept of homogeneous coordinates: Homogeneous coordinates use N+1 dimensions to describe an N-dimensional coordinate.
Homogeneous coordinates of a point $x=(x_1,x_2,x_3{)}^T$ (which is a 3-dimensional vector) and non-homogeneous coordinates$(x,y{)}^T$ (it is a 2-dimensional vector).

If the world and image points are represented by homogeneous vectors, then the central projection can simply be expressed as a linear mapping between homogeneous coordinates. Specifically, $Substitute\left(1\right)$ for the following variable:
$$[\begin{array}{c}X\\ \mathbf{Y}\\ Z\\ 1\end{array}]\mapsto \left[\begin{array}{c}f\mathbf{x}\\ f\mathbf{y}\\ \mathbf{z}\end{array}\right]=\left[\begin{array}{cccc}f& & & 0\\ & f& & 0\\ & & 1& 0\end{array}\right]\left[\begin{array}{c}X\\ Y\\ Z\\ 1\end{array}\right]\left(2\right)$$
index$\left[\begin{array}{cccc}f& & & 0\\ & f& & 0\\ & & 1& 0\end{array}\right]$means $3\ast 4$ homogeneous camerasprojection matrix, recorded as $P$。 $P$ can be written as$diag(f,f,1)[I|0]$ , where$diag(f,f,1)$ isa diagonal matrix, and$[I|0]$ indicates that the matrix is ​​divided into a$3\ast 3$ ​Identity matrixplus azero column vector. Then, the camera projection matrix of the centrally projected pinhole model can be expressed as:
$$P=diag(f,f,1)[I|0]$$

The concept of identity matrix: The identity matrix, also known as the identity matrix , is a square matrix whoseThe elements on the diagonal are 1, and the remaining elements are 0, Recording $I$ or$E.$ _ The size of the identity matrix is ​​determined by its dimensions. For example, the third-order identity matrix is ​​a 3x3 matrix.

Identity matrices have many important properties in linear algebra. For example,For any matrix A, the product of the identity matrix 1 and A is equal to A itself. This is because each element of the identity matrix is ​​multiplied by the corresponding element of A and added together, and the result is A itself. This property has important applications in matrix transposition and inverse operations.

Identity matrices also play an important role in deep learning. In neural networks,Identity matrices are often used as initialization weight matrices. When initializing the weight matrix, setting it to an identity matrix can make the initial state of the neural network more stable. This is because the identity matrix has a certain degree of symmetry and balance, which can avoid problems such as gradient disappearance or gradient explosion, and helps improve the training effect of the model .

Identity matrices can also be used to measure similarity of matrices. In image processing and pattern recognition, we often need to compare the similarity of two matrices. By calculating the difference between two matrices, a measure of their similarity can be obtained. As a special matrix, the identity matrix has obvious differences compared with other matrices and can be used to measure the similarity between matrices.

We now introduce the following notation: world point $X$ uses a 4-dimensional homogeneous vector$(X,Y,Z,1)$ means; image point$x$ is expressed as a 3-dimensional homogeneous vector. Then$Formula\left(2\right)$ can be compactly written as:
$$x=PX$$

4. Main point offset

$Formula\left(1\right)$ assumes that the coordinate origin of the image plane is at the principal point. The actual situation may not be like this, as shown in the figure below:

Camera coordinate system $(x_{cam},y_{cam}{)}^{The\; coordinate\; origin\; of\; T}$ is the camera center, and the projection of this origin on the image plane is the principal point p. Image coordinate system$(x,y{)}^{The\; coordinate\; origin\; of\; T}$ is the lower left corner of the image.

Therefore, the mapping in the general case is:
$$(X,Y,Z{)}^T\mapsto (fX/Z+p_x,fY/Z+p_y{)}^T$$
其中 $(p_x,p_y{)}^{Let}$ 's use the graphical coordinate system:
$$[\begin{array}{c}\mathbf{X}\\ \mathbf{Y}\\ \mathbf{Z}\\ 1\end{array}]\mapsto \left[\begin{array}{c}f\mathbf{x}+\mathbf{Z}{\mathbf{p}}_{\mathbf{x}}\\ f\mathbf{y}+Z{p}_{\mathbf{y}}\\ \mathbf{z}\end{array}\right]=\left[\begin{array}{cccc}f& & p_x& 0\\ & f& p_x& 0\\ & & 1& 0\end{array}\right]\left[\begin{array}{c}X\\ Y\\ Z\\ 1\end{array}\right]\left(3\right)$$
DefinitionK
$$K=\left[\begin{array}{ccc}f& & p_x\\ & f& p_x\\ & & 1\end{array}\right]\left(4\right)$$
then$Formula\left(3\right)$ has a concise form:
$$x=K[I|0]X_{cam}\left(5\right)$$
Matrix$K$ is called the camera calibration matrix, in$In\; formula\left(5\right)$ we record$(X,Y,Z,1{)}^T$ 为 $X_{cam}$is to emphasize that the camera is set at the origin of a Euclidean coordinate system and the main axis is along $The\; direction\; of\; the\; z$ -axis, while point$X_{cam}$Expressed according to this coordinate system. Such a coordinate system can be calledcamera coordinate system。

The origin of the camera coordinate system is the camera center , $The\; z-$ axis direction points tothe main axis。

5. Camera rotation and displacement

Generally, 3-dimensional space points are represented by different Euclidean coordinate systems, called world coordinate systems .The camera coordinate system and the world coordinate system are connected through rotation and translation .。

Euclidean transformation between world coordinate system and camera coordinate system

if $\tilde{X}$is a 3-dimensional non-homogeneous vector, representing the coordinates of a point in the world coordinate system , and ${\tilde{X}}_{cam}$is the same point represented by the camera coordinate system , then we can record ${\tilde{X}}_{cam}=R\left(\tilde{X}-\tilde{C}\right)$ , where$\tilde{C}$Represents the coordinates of the camera center in the world coordinate system, $R$ is a$3\ast The\; rotation\; matrix\; of\; 3$ represents the orientation of the camera coordinate system. This equation can be written as:
$$X_{cam}=\left[\begin{array}{cc}R& -R\tilde{C}\\ 0^T& 1\end{array}\right]\left[\begin{array}{c}X\\ Y\\ Z\\ 1\end{array}\right]=\left[\begin{array}{cc}R& -R\tilde{C}\\ 0^T& 1\end{array}\right]X\left(6\right)$$
Compare it with$Formulas\left(5\right)$ are combined to form the formula:
$$x=KR\left[I|-\tilde{C}\right]X\left(7\right)$$
where$X$ is expressed in the world coordinate system. This is a general mapping given by a pinhole model.

6. Camera internal parameters and external parameters

From $It\; can\; be\; seen\; from\; formula\left(7\right)$ that,General pinhole camera $P=KR\left[I|-\tilde{C}\right]$ has 9 degrees of freedom: 3 from$K(elementf,p_x,p_y)$ , 3 from$R$ , 3 from$\tilde{C}$. Contained in KKThe parameters in$K\; are\; called$Camera internal parametersorCamera internal calibration. Contained in $R$ and$\tilde{C}$The parameters in are related to the orientation and position of the camera in the world coordinate system, and are calledexternal parametersorexternal calibration。

For convenience, usually the camera center is not clearly marked, and the transformation from the world coordinate system to the image coordinate system is expressed as ${\tilde{X}}_{cam}=R\tilde{X}+t$ . In this case, the camera matrix is ​​simplified to:
$$P=k[R|t](8)$$
where according to$Formula\left(7\right)$ ,$t=-R\tilde{C}$。

7. CCD camera

For the basic pinhole model, the image coordinates are assumed to have equal-scale Euclidean coordinates in both axes. But the pixels of a CCD camera may not be square. If the image coordinates are measured in pixels, then non-equal scaling factors need to be introduced in each direction . Specifically, if in $x$和yyThe number of pixels per unit distance of the image coordinates in the$y$$m_x$ 和 $m_y$, then the transformation from world coordinates to pixel coordinates is given by $Formula\left(4\right)$ is left multiplied by an additional factor$diag(m_x,m_y,1)$ and get. Therefore, the general form of a CCD camera calibration matrix is:
$$K=\left[\begin{array}{ccc}{a}_x& & x_0\\ & a_y& y_0\\ & & 1\end{array}\right]\left(9\right)$$
where$a_x=f{m}_x$和$a_y=f{m}_y$Convert the focal length of the camera to $x$和$The\; pixel\; dimension\; in\; the\; y$ direction. In the same way,${\tilde{x}}_0=(x_0,y_0{)}^T$ is the principal point expressed in pixel dimension, and its coordinates are$x_0=m_x{p}_x$和$y_0=m_y{p}_y$. therefore,A CCD camera has 10 degrees of freedom。