[Stereo Vision (1)] Imaging Principle and Camera Distortion

This is a personal study note. I have borrowed a lot of good pictures and articles (references are at the end of the article), mainly to sort out relevant knowledge in order to form my own system. The text expression is pieced together, and the symbolic formula is manually entered. Please point out if there are any mistakes.

1. Imaging principle

1) Pinhole model

In scientific research, the internal process of a phenomenon is always complicated and difficult to see clearly, but smart scholars always use the simplest model for initial description and propose a relatively simple model. The imaging process is no exception. The process of mapping the coordinate points in the three-dimensional world to the two-dimensional image plane is described by a geometric model. There are many kinds of models, the simplest of which is called the pinhole model.

Note that in exploring imaging, a virtual imaging plane is often used for analysis.

2) Coordinate system conversion

From a point in the real world (world coordinate system) to an imaging point on a photo (pixel coordinate system) involves conversion between the following four coordinate systems.

• World Coordinate System (WCS): The position of a point in the real world, describing the camera position, in m.
• Camera Coordinate System (CCS): The center of the camera sensor is the origin, and the camera coordinate system is established in m.
• Image Coordinate System (Film Coordinate System, FCS): A two-dimensional coordinate system with the principal point as the origin, the unit is mm.
• Pixel Coordinate System (PCS): A two-dimensional coordinate system whose origin is the upper left corner of the photo, and the unit is none or pixel.

Next, follow the steps below to introduce the transformation of a point in each coordinate system during the imaging process of the camera.

1. World coordinate system to camera coordinate system

From the world coordinate system to the camera coordinate system (World to Camera, W2C), it is a simple three-dimensional coordinate system transformation (the change of the direction of the three coordinate axes and the origin). A point P w on the world coordinate system ( U , V , W ) {P_{w}(U,V,W)}$P_w(U,V,W)$ to a point P c ( X , Y , Z ) {P_{c}(X,Y,Z)}on the camera coordinate system$P_c(X,Y,Z)$：

$$\left[\begin{array}{c}X\\ Y\\ Z\end{array}\right]=\left[\begin{array}{cc}{R}_{3\times 3}& T_{3\times 1}\end{array}\right]\left[\begin{array}{c}U\\ V\\ W\\ 1\end{array}\right]$$

or using the second matrix expression:

$$\left[\begin{array}{c}X\\ Y\\ Z\\ 1\end{array}\right]=\left[\begin{array}{cc}{R}_{3\times 3}& T_{3\times 1}\\ O& 1\end{array}\right]\left[\begin{array}{c}U\\ V\\ W\\ 1\end{array}\right]$$

Among them, $\left[\begin{array}{cc}{R}_{3\times 3}& T_{3\times 1}\end{array}\right]$ (或 $\left[\begin{array}{cc}{R}_{3\times 3}& T_{3\times 1}\\ O& 1\end{array}\right]$ ) is called the camera’sextrinsic matrix (Extrinsic Matrix), which describes the position of the camera in world coordinates ($T_{3\times 1}$), and its pointing direction ( $R_{3\times 3}$）。

---

2. Camera coordinate system to image coordinate system

Camera coordinate system to image coordinate system (Camera to Film, C2F), here consists of three-dimensional coordinates $P_c(X,Y,Z)$ dimensionality reduction to two-dimensional coordinates$p(x,y)$ 。

According to similar triangles, we get:

$$\frac{x}{X}=\frac{y}{Y}=\frac{f}{Z}$$

则：
$$\left[\begin{array}{c}x\\ y\\ 1\end{array}\right]=\left[\begin{array}{ccc}\frac{f}{Z}& 0& 0\\ 0& \frac{f}{Z}& 0\\ 0& 0& \frac{1}{Z}\end{array}\right]\left[\begin{array}{c}X\\ Y\\ Z\end{array}\right]$$

---

3. Image coordinate system to pixel coordinate system

The image coordinate system to pixel coordinate system (Film to Pixel, F2P) is a simple affine transformation (Affine Transformation) process, mainly including the movement of the origin and the transformation of the scale.

There are:
$$u=\frac{x}{d_x}+u_0,v=\frac{y}{d_y}+v_0$$

矩阵形式：
$$\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=\left[\begin{array}{ccc}\frac{1}{d_x}& 0& u_0\\ 0& \frac{1}{d_y}& v_0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{c}x\\ y\\ 1\end{array}\right]$$

---

4. Camera coordinate system to pixel coordinate system

Now start backtracking, following the previous step, the camera coordinate system to pixel coordinate system (Camera to Film, C2F) is:
$$\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=\left[\begin{array}{ccc}\frac{1}{d_x}& 0& u_0\\ 0& \frac{1}{d_y}& v_0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{ccc}\frac{f}{Z}& 0& 0\\ 0& \frac{f}{Z}& 0\\ 0& 0& \frac{1}{Z}\end{array}\right]\left[\begin{array}{c}X\\ Y\\ Z\end{array}\right]=\frac{1}{Z}\left[\begin{array}{ccc}\frac{f}{d_x}& 0& u_0\\ 0& \frac{f}{d_y}& v_0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{c}X\\ Y\\ Z\end{array}\right]$$

其中，$\frac{f}{d_x}、\frac{f}{d_y}$That is, the focal length is divided by the pixel size, and the unit is pixel, indicating the pixel unit value of the focal length in the direction of two pixels. Since f , dx , dy {f}, {d_x}, {d_y} during camera calibration$f、d_x、d_y$It cannot be directly measured, and its combined value $\frac{f}{d_x}、\frac{f}{d_y}$It can be obtained by calibration, so $f_x=\frac{f}{d_x}、f_y=\frac{f}{d_y}$to represent two combined values. Then:
$$\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=\frac{1}{Z}\left[\begin{array}{ccc}{f}_x& 0& u_0\\ 0& f_y& v_0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{c}X\\ Y\\ Z\end{array}\right]$$

If the two coordinate axes in the pixel coordinate system are not vertical (at this time, the pixel is not a rectangle, but a parallelogram), then there is a skew coefficient $s$（skew coefficient），$s=f_{x}tan\left(\alpha \right)$ , if you are interested, you can deduce it.

此时：
$$\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=\frac{1}{Z}\left[\begin{array}{ccc}{f}_x& s& u_0\\ 0& f_y& v_0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{c}X\\ Y\\ Z\end{array}\right]$$

For example, $\left[\begin{array}{ccc}{f}_x& s& u_0\\ 0& f_y& v_0\\ 0& 0& 1\end{array}\right]$Intrinsics Matrix of the camera , which describes the transformation relationship from the camera coordinate system to the pixel coordinate system, reflects the properties of the camera itself, usually with the symbol $K$ means (note thatthe internal reference matrix in many versions does not contain a skew coefficient, that is, it is 0 by default). In the camera coordinate system,$Z$ corresponds to the depth of a three-dimensional point, often called the scale factor$\lambda$ , then the camera coordinate system to the pixel coordinate system:
$$l\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=\left[\begin{array}{ccc}{f}_x& s& u_0\\ 0& f_y& v_0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{c}X\\ Y\\ Z\end{array}\right]$$
Example:
$$K=\left[\begin{array}{ccc}{f}_x& s& u_0\\ 0& f_y& v_0\\ 0& 0& 1\end{array}\right]$$
有:
$$p\_=K{P}_c$$

Supplement: $Z=1$ is called the normalized plane, and the coordinates on the normalized plane are called normalized coordinates.

---

5. World coordinate system to pixel coordinate system

World coordinate system to pixel coordinate system (World to Pixel, W2P), the whole string:
$$l\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=\left[\begin{array}{ccc}{f}_x& s& u_0\\ 0& f_y& v_0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{c}X\\ Y\\ Z\end{array}\right]=\left[\begin{array}{ccc}{f}_x& s& u_0\\ 0& f_y& v_0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{cc}{R}_{3\times 3}& T_{3\times 1}\end{array}\right]\left[\begin{array}{c}U\\ V\\ W\\ 1\end{array}\right]$$

The conversion from the world coordinate system to the pixel coordinate system actually expresses the projection relationship from the spatial point to the image point in the perspective projection, so the conversion matrix is ​​called the projection matrix (Projection Matrix) , commonly used $M$ indicates that the projection matrix is ​​a 3x4 matrix through matrix operations, which is the product of the internal reference matrix and the external reference matrix.

$$p\_=K\left[\begin{array}{cc}{R}_{3\times 3}& T_{3\times 1}\end{array}\right]\left[\begin{array}{c}{P}_w\\ 1\end{array}\right]=M\left[\begin{array}{c}{P}_w\\ 1\end{array}\right]$$

In addition, if the second matrix expression is used, the matrix dimension will change.

---

2. Camera Distortion

Due to the inherent characteristics of the camera lens (convex lens converges light, concave lens diverges light), the imaging straight line will become a curve.

Camera distortion mainly includes radial distortion and tangential distortion.

Distortion characteristics:

• Radial distortion is mainly due to the geometry of the lens changing the shape of the line.
• Tangential distortion is mainly due to the fact that the lens is not mounted perfectly parallel to the image plane.
• In practical experience, image geometry pays more attention to radial distortion, so sometimes the process of distorted image correction may ignore tangential distortion.
  

1) Radial distortion

Radial distortion mainly includes:

• barrel distortion
• pincushion distortion
• mustache distortion
  

It can be seen that the characteristics of the radially distorted image are:

• Centrosymmetric
• Straight Variation

distortion	features	Scenes
barrel distortion	The center is enlarged, and the farther away from the optical center, the smaller the image magnification	Fisheye lens, wide-angle/panoramic pictures
pincushion distortion	Pincushion distortion squeezes the picture, (like a squashed pillow)	Telephoto lenses often suffer from pincushion distortion to eliminate the spherical effect
mustache distortion	A mixture of the two types above, which starts with barrel distortion near the optical axis and progresses to pincushion distortion toward the edges of the image

Radial Distortion Correction

In general, the radial distortion of an image can be represented by a low-order polynomial model:

$$\begin{gathered} x_{undist}=x_{dist}(1+k_1{r}^2+k_2{r}^4+k_3{r}^6) \\ {y}_{undist}=y_{dist}(1+k_1{r}^2+k_2{r}^4+k_3{r}^6) \end{gathered}$$

其中 $r^2=x_{dist}^2+y_{dist}^2$，$(x_{dist}，y_{dist})$ is the point of the normalized camera coordinate system, that is, the origin of the coordinates has been moved to the main point, and the pixel coordinates are divided by the focal length,$x_{dist}=\frac{X}{Z}=\frac{u-u_o}{f_x}$ ，$y_{dist}=\frac{Y}{Z}=\frac{v-v_o}{f_y}$, which can be obtained from the distorted image. $k_1、k_2、k_3$It is the radial distortion parameter (generally, the first two terms of the polynomial can be used, and the third term will be used for cameras with large distortion such as fisheye, and the three terms are always correct by default), which can be obtained through checkerboard calibration.

2) Tangential Distortion

Tangential distortion is caused by the camera sensor not being parallel to the lens.

Tangential Distortion Correction

The tangential distortion of the image can also be represented by a low-order polynomial model:
$$\begin{gathered} x_{\text{undist }}=x_{\text{dist }}+\left[2p_1{x}_{\text{dist }}{y}_{\text{dist }}+p_2\left(r^2+2x{\_}_{\text{dist }}^2\right)\right] \\ {y}_{\text{undist }}=y_{\text{dist }}+\left[p_1\left(r^2+2y_{\text{dist }}^2\right)+2p_2{x}_{\text{dist }}{y}_{\text{dist }}\right] \end{gathered}$$

$p_1$and $p_2$is the tangential distortion parameter.

3) Total Distortion Correction

Consider both radial and tangential distortions:

$$\begin{gathered} x_{undist}=x_{dist}(1+k_1{r}^2+k_2{r}^4+k_3{r}^6)+\left[2p_1{x}_{dist}{y}_{dist}+p_2\left(r^2+2x{\_}_{dist}^2\right)\right] \\ {y}_{undist}=y_{dist}(1+k_1{r}^2+k_2{r}^4+k_3{r}^6)+\left[p_1\left(r^2+2y_{dist}^2\right)+2p_2{x}_{dist}{y}_{dist}\right] \end{gathered}$$
There are five distortion parameters$k_1、k_2、k_3、p_1、p_2$, these 5 distortion parameters together with the internal reference matrix need to be calibrated for the camera. But the output order in OpenCV camera calibration is $k_1、k_2、p_1、p_2、k_3$, because $k_3$Not important, not needed in many cases. (Isn't it true?)

---

References:
[1] CSE/EE486 Computer Vision I
[2] A detailed explanation of camera calibration algorithm principle
[3] Stereo Vision Getting Started Guide (1): Coordinate System and Camera Parameters
[4] Camera calibration (Camera calibration) principle and steps