[Digital Image Processing] Geometric transformation of images

---

Two images containing the same content may have completely different appearances due to geometric distortion caused by imaging angle, perspective relationships, or even the lens itself.
Through appropriate geometric transformation, the negative effects of these geometric distortions can be eliminated to the greatest extent, which is beneficial to focusing on the content of the image itself in subsequent processing and recognition work.

General ideas of image geometric transformation

A geometric transformation requires two parts of the operation:

The first is the operations required for spatial transformation, such as translation, rotation, and mirroring, which need to be used to represent the mapping relationship between the output image and the input image;

In addition, a grayscale interpolation algorithm needs to be used to calculate according to this transformation relationship. The pixels of the output image may be mapped to non-integer coordinates of the input image.

Assume the original image $f(x_0,y_0)$ The target image generated by geometric transformation is$g(x_1,y_1)$ , then the spatial transformation mapping relationship can be expressed as:
$$x_1=s(x_0,y_0)\text{\hspace{1em}(1)}$$

$$y_1=t(x_0,y_0)\text{\hspace{1em}(2)}$$

Among them, $s(x_0,y_0)$和$t(x_0,y_0)$ is given by$f(x_0,y_0)$到$g(x_1,y_1)$ coordinate transformation function.

For example, $x_1=s(x_0,y_0)=2x{\_}_0$， $y_1=t(x_0,y_0)=2y_0$When, the transformed image $g(x_1,y_1)$ just simply at$x$和yyIn the two spatial directions of$y$$f(x_0,y_0)$ is doubled in size.

Mastered the transformation function $s(x_0,y_0)$和$t(x_0,y_0)$ , you can follow the steps below to implement geometric transformation.

Algorithm 4.1 steps:

• According to the mapping relationship of the spatial transformation, determine the size of the transformed target image, that is, the row and column range, because some transformations may change the image size;
• Calculate the inverse transformation $s^{-1}(j_1,i_1)$和$t^{-1}(j_1,i_1)$；
• Progressively scan the target image $g(x_1,y_1)$ , for$g(x_1,y_1)$ in each point$(j_0,j_0)$：
  • According to the mapping relationship of spatial transformation, it is calculated:
    • $j_0^{\prime }=s^{-1}(j_1,i_1);$ // The abscissa calculated directly through the mapping relationship may not be an integer;
    • $i_0^{\prime }=t^{-1}(j_1,i_1);$ // The vertical coordinate calculated directly through the mapping relationship may not be an integer;
  • Depending on the chosen interpolation method:
    • $(j_0,i_0)=interp(j_0^{\prime },i_0^{\prime });$ // for non-integer coordinates$(j_0^{\prime },i_0^{\prime })$ requires interpolation
    • $if$ $(j_0,i_0)$ in imageffwithin $f$
      • Copy the corresponding pixel: $g(j_1,i_1)=f(j_0,i_0);$
    • $else$
      • $g(j_1,i_1)=255;$

The restoration (correction) process of geometrically distorted images is exactly the inverse process of the above transformation.
$$x_0=s^{-1}(x_1,y_1)\text{\hspace{1em}(3)}$$

$$y_0=t^{-1}(x_1,y_1)\text{\hspace{1em}(4)}$$

Equations (3) and (4) represent the corresponding $g(x_1,y_1)$ to$f(x_0,y_0)$ inverse transformation. At this time, the image g ( x 1 , y 1 ) g(x_1, y_1)distorted by geometric transformation$g(x_1,y_1)$ is the object to be restored, the original image$f(x_0,y_0)$ is the goal of recovery.

When image normalization is used to eliminate changes in image appearance caused by geometric factors, it is called image geometric normalization. It can eliminate the differences in geometric relationships between objects and find those geometric invariants in the image, thereby knowing these objects. Are originally the same or belong to the same category.

---

image pan

Image translation is to move all points in the image horizontally or vertically according to the specified translation amount.
Let $(x_0,y_0)$ is a point on the original image, and the horizontal translation amount of the image is$T_x$, the vertical translation amount is $T_y$,
the point coordinates after translation become $(x_1,y_1)$ , becomes
$$x_1=x_0+t_x$$
$$y_1=y_0+t_y$$

Define the function
$$\left[\begin{array}{ccc}{x}_1& y_1& 1\end{array}\right]=\left[\begin{array}{ccc}{x}_0& y_0& 1\end{array}\right]\left[\begin{array}{ccc}1& 0& 0\\ 0& 1& 0\\ T_x& T_y& 1\end{array}\right]\text{\hspace{1em}( 5 )}$$
Invert the transformation matrix to get the inverse transformation:
$$\left[\begin{array}{ccc}{x}_0& y_0& 1\end{array}\right]=\left[\begin{array}{ccc}{x}_1& y_1& 1\end{array}\right]\left[\begin{array}{ccc}1& 0& 0\\ 0& 1& 0\\ -T_x& -T_y& 1\end{array}\right]\text{\hspace{1em}(6)}$$
即
$$\{\begin{array}{c}{x}_0=x_1-T_x\\ y_0=y_1-T_y\end{array}\text{\hspace{1em}( 7 )}$$
From the above formula, we can get that every point in the translated target image can find the corresponding point in the original image.

For example, for (i, j) (i,j) in the new graph$(i,j)$ pixel, into the formula you can get its corresponding pixel in the original image$(i-T_x,j-T_y)$ . At this time, if$T_x>i$ 或$T_y>j$，点$(i-T_x,j-T_y)$ is beyond the scope of the original image, and its pixel values ​​can be directly set to$0$ or$255$。

There are usually two processing methods for the points in the original image that have been moved out of the image display area. They can be discarded directly, or the width of the newly generated image can be increased by $T_x$, the height increases $T_y$The method enables these points to be included in the new image.

imtransformFunctions are used to complete general two-dimensional space transformations.
Function prototype:B = imtransform(A, TFORM, method);

• A is the image to be geometrically transformed;
• TFORM specifies the specific transformation type;
• method allows you to select the interpolation method for the imtransform function. The optional interpolation methods are: bicubic, bilinear, nearest

TFORM structures can be created in two ways, using maketformfunctions and cp2tformfunctions.
cp2tform is a data fitting function that requires the corresponding point pairs between the original image and the target image as input to determine the geometric transformation relationship based on the control point pair.

maketform function calling form:T = maketform(transformtype, Matrix);

• The parameter transformtype specifies the type of transformation;
• The parameter Matrix is ​​the corresponding affine transformation matrix. For example, for translation transformation, the matrix is ​​[1,0,0;0,1,0;T_x,T_y,1]

---

image mirroring

Image mirroring transformation is divided into horizontal mirroring and vertical mirroring. Horizontal mirroring converts the left and right halves of the image with the vertical central axis of the image as the central axis; vertical mirroring converts the upper and lower half of the image with the horizontal central axis of the image as the central axis.

Determine the range of the equations:
$$\left[\begin{array}{ccc}{x}_1& y_1& 1\end{array}\right]=\left[\begin{array}{ccc}{x}_0& y_0& 1\end{array}\right]\left[\begin{array}{ccc}-1& 0& 0\\ 0& 1& 0\\ Width& 0& 1\end{array}\right]=\left[\begin{array}{ccc}Width-x_0& y_0& 1\end{array}\right]\text{\hspace{1em}( 8 )}$$
Specific graph:
$$\left[\begin{array}{ccc}{x}_0& y_0& 1\end{array}\right]=\left[\begin{array}{ccc}{x}_1& y_1& 1\end{array}\right]\left[\begin{array}{ccc}-1& 0& 0\\ 0& 1& 0\\ Width& 0& 1\end{array}\right]=\left[\begin{array}{ccc}Width-x_1& y_1& 1\end{array}\right]\text{\hspace{1em}( 9 )}$$
The vertical mirror transformation relationship can be formally described as follows:
$$\left[\begin{array}{ccc}{x}_1& y_1& 1\end{array}\right]=\left[\begin{array}{ccc}{x}_0& y_0& 1\end{array}\right]\left[\begin{array}{ccc}1& 0& 0\\ 0& -1& 0\\ 0& Height& 1\end{array}\right]\text{\hspace{1em}( 10 )}$$
Define:
$$\left[\begin{array}{ccc}{x}_0& y_0& 1\end{array}\right]=\left[\begin{array}{ccc}{x}_1& y_1& 1\end{array}\right]\left[\begin{array}{ccc}1& 0& 0\\ 0& -1& 0\\ 0& Height& 1\end{array}\right]\text{\hspace{1em}(11)}$$

---

image transpose

Image transpose is to transform $x$ coordinate and$The\; y$ coordinates are swapped, the size of the image will change after transposition, and the height and width will be swapped.

The transpose transformation formula is as follows:
$$\left[\begin{array}{ccc}{x}_1& y_1& 1\end{array}\right]=\left[\begin{array}{ccc}{x}_0& y_0& 1\end{array}\right]\left[\begin{array}{ccc}0& 1& 0\\ 1& 0& 0\\ 0& 0& 1\end{array}\right]\text{\hspace{1em}( 12 )}$$
The inverse matrix of the transposed matrix is ​​still itself.

---

Image zoom

Image scaling means that the image is enlarged or reduced according to a specified ratio.

Assume image $The\; scaling\; ratio\; in\; the\; x-$ axis direction is$S_x$，$The\; scaling\; ratio\; in\; the\; y-$ axis direction is$S_y$, the corresponding transformation expression is:
$$\left[\begin{array}{ccc}{x}_1& y_1& 1\end{array}\right]=\left[\begin{array}{ccc}{x}_0& y_0& 1\end{array}\right]\left[\begin{array}{ccc}{S}_x& 0& 0\\ 0& S_y& 0\\ 0& 0& 1\end{array}\right]=\left[\begin{array}{ccc}{x}_0\cdot S_x& y_0\cdot S_y& 1\end{array}\right]\text{\hspace{1em}( 13 )}$$
Derive the equation:
$$\left[\begin{array}{ccc}{x}_0& y_0& 1\end{array}\right]=\left[\begin{array}{ccc}{x}_1& y_1& 1\end{array}\right]\left[\begin{array}{ccc}\frac{1}{S_x}& 0& 0\\ 0& \frac{1}{S_y}& 0\\ 0& 0& 1\end{array}\right]=\left[\begin{array}{ccc}\frac{x_1}{S_x}& \frac{y_1}{S_y}& 1\end{array}\right]\text{\hspace{1em}( 14 )}$$
In the target image calculated directly according to the scaling formula, some mapping source coordinates may not be integers, so the corresponding pixel position cannot be found. For example, when the image is magnified 2 times, that is,$S_x=S_y=2$ , enlarge the pixels in the image$(0,1)$ Corresponds to the pixel (0, 0.5) (0, 0.5)in the original image$(0,0.5)$ , which is not an integer coordinate position and its grayscale value cannot be extracted.

Some kind of approximation has to be done, a simple strategy is to just use the nearest nearest integer coordinate position $(0,0)$ or$(0,$The pixel gray value at$1$$)\; ,\; which\; is\; also\; the\; nearest\; neighbor\; interpolation\; algorithm.$

---

image rotation

Rotation generally refers to rotating an image by a certain angle around a specified point. Rotation also usually changes the size of the image. As shown below

点$P(x_0,y_0)$ rotates counterclockwise around the origin through an angle$\theta$ to$P1(x_1,y_1)$，令$L=|OP|=\sqrt{x^2+y^2}$,有: $sin\alpha =y_0/L$， $cos\alpha \_=x_0/L$。

Arrive at $P1$ ,equation:
$$\begin{gathered} sin(a+i)=y_1/L=cos\theta sin\alpha +sin\theta cosa \\ cos(a+i)=x_1/L=cos\theta cos\alpha -sin\theta sina\text{\hspace{1em}( 15 )} \end{gathered}$$
Denote:
$$\begin{gathered} x_1=cos\theta x_0-sin\theta y_0 \\ {y}_1=cos\theta y_0+sin\theta x_0\text{\hspace{1em}( 16 )} \end{gathered}$$
Let us derive the matrix:
$$\left[\begin{array}{ccc}{x}_1& y_1& 1\end{array}\right]=\left[\begin{array}{ccc}{x}_0& y_0& 1\end{array}\right]\left[\begin{array}{ccc}cos\theta & sin\theta & 0\\ -sin\theta & cos\theta & 0\\ 0& 0& 1\end{array}\right]\text{\hspace{1em}( 17 )}$$
Specifies:
$$\left[\begin{array}{ccc}{x}_0& y_0& 1\end{array}\right]=\left[\begin{array}{ccc}{x}_1& y_1& 1\end{array}\right]\left[\begin{array}{ccc}cos\theta & -sin\theta & 0\\ sin\theta & cos\theta & 0\\ 0& 0& 1\end{array}\right]\text{\hspace{1em}( 18 )}$$
The above discussion is based on the origin, which is the lower left corner of the image, as the center. How to rotate around any specified point?

Just combine the translation and rotation operations. First translate the coordinate system, then rotate around the new coordinate origin, and then translate the new origin back to the origin of the original coordinate system. This process can be summarized into the following 3 steps:

1. Change coordinate system I to II;
2. Rotate the point clockwise $\theta$ angle;
3. Change coordinate system II back to I;

Taking rotation around the center of the image as an example, as shown in the figure below, coordinate system I has the origin of the upper left corner of the image, and $The\; positive\; direction\; of\; the\; x-$ axis, downward is$The\; positive\; direction\; of\; the\; y-$ axis; while the coordinate system II takes the center of the image as the origin, and$The\; positive\; direction\; of\; the\; x-$ axis, downward is$Positive\; y-$ axis direction. The conversion relationship between coordinate system I and coordinate system II is as follows:

Assume that the width of the image is $w$ , high bit$h$ ,denote by:
$$\left[\begin{array}{c}{x}_{Ⅰ}\\ y_{Ⅰ}\\ 1\end{array}\right]=\left[\begin{array}{c}{x}_{Ⅱ}\\ y_{Ⅱ}\\ 1\end{array}\right]\left[\begin{array}{ccc}1& 0& 0\\ 0& -1& 0\\ 0.5w& 0.5& 1\end{array}\right]\text{\hspace{1em}( 19 )}$$
The corresponding inverse transformation is:
$$\left[\begin{array}{c}{x}_{Ⅱ}\\ y_{Ⅱ}\\ 1\end{array}\right]=\left[\begin{array}{c}{x}_{Ⅰ}\\ y_{Ⅰ}\\ 1\end{array}\right]\left[\begin{array}{ccc}1& 0& 0\\ 0& -1& 0\\ -0.5w& 0.5& 1\end{array}\right]\text{\hspace{1em}( 20 )}$$
Steps 1 and 3 of the above three steps have been implemented here. Together with the rotation transformation of step 2, the final transformation matrix rotated around the center point of the image is obtained. This matrix is ​​actually a cascade of three transformation matrices used in each of the three transformation steps. In the formula,$W_{old}$、$H_{old}$、$W_{new}$、$H_{new}$represent the width and height of the original image and the new image respectively.

$$\begin{array}{rl}\left[\begin{array}{c}{x}_1\\ y_1\\ 1\end{array}\right]& =\left[\begin{array}{c}{x}_0\\ y_0\\ 1\end{array}\right]\left[\begin{array}{ccc}1& 0& 0\\ 0& -1& 0\\ -0.5W_{old}& 0.5H{\_}_{old}& 1\end{array}\right]\left[\begin{array}{ccc}cos\theta & -sin\theta & 0\\ sin\theta & cos\theta & 0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{ccc}1& 0& 0\\ 0& -1& 0\\ 0.5W_{new}& 0.5H{\_}_{new}& 1\end{array}\right]\\ & =\left[\begin{array}{c}{x}_0\\ y_0\\ 1\end{array}\right]\left[\begin{array}{ccc}cos\theta & sin\theta & 0\\ -sin\theta & cos\theta & 0\\ 0.5\cdot (-W_{old}\cdot cos\theta +H_{old}\cdot sin\theta +W_{new})& 0.5(-W_{old}\cdot sin\theta -H_{old}\cdot cos\theta +H_{new})& 1\end{array}\right]\end{array}\text{\hspace{1em}(21)}$$

The inverse transformation of the above formula is:

$$\begin{array}{rl}\left[\begin{array}{c}{x}_0\\ y_0\\ 1\end{array}\right]& =\left[\begin{array}{c}{x}_1\\ y_1\\ 1\end{array}\right]\left[\begin{array}{ccc}1& 0& 0\\ 0& -1& 0\\ -0.5W_{new}& 0.5H{\_}_{new}& 1\end{array}\right]\left[\begin{array}{ccc}cos\theta & sin\theta & 0\\ -sin\theta & cos\theta & 0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{ccc}1& 0& 0\\ 0& -1& 0\\ 0.5W_{old}& 0.5H{\_}_{old}& 1\end{array}\right]\\ & =\left[\begin{array}{c}{x}_1\\ y_1\\ 1\end{array}\right]\left[\begin{array}{ccc}cos\theta & -sin\theta & 0\\ sin\theta & cos\theta & 0\\ -0.5W_{new}\cdot cos\theta -0.5H{\_}_{new}\cdot sin\theta +0.5W_{old}& 0.5W_{new}\cdot sin\theta -0.5H{\_}_{new}\cdot cos\theta +0.5H{\_}_{old}& 1\end{array}\right]\end{array}\text{\hspace{1em}(22)}$$

According to the inverse transformation formula, the rotation transformation around the center of the image can be implemented as described in Algorithm 4.1. Similarly, the rotation transformation centered on any point can be further obtained.

---

interpolation algorithm

When implementing geometric operations, there are two methods: forward mapping and backward mapping.

• The principle of the forward mapping method is to transfer the grayscale of the input image to the output image sequentially according to pixels, that is, the target image coordinates are calculated from the original image coordinates: $g(x_1,y_1)=f(a(x_0,y_0),b(x_0,y_0))$ . For example, operations such as image translation and mirroring can use this method.
• The backward mapping method is the inverse of the forward mapping transformation, that is, the output pixels are mapped back to the input image in turn. If an output pixel is mapped to a pixel point that is not at the integer coordinates of the sampling grid of the input image, then its gray value It is necessary to make inferences based on the grayscale values ​​​​of integer coordinates, which is interpolation. The backward mapping method generates the output image pixel by pixel without causing computational waste. This method is often used in operations such as scaling and rotation.

Commonly used interpolation algorithms include: nearest neighbor interpolation, bilinear interpolation, and high-order interpolation.

nearest neighbor interpolation

Nearest neighbor interpolation algorithm, the value of the output pixel is the pixel value of its nearest sampling point in the input image.

Nearest neighbor interpolation can be expressed as
$$f(x,y)=g(round(x),round(y))\text{\hspace{1em}( 23 )}$$
The calculation of nearest neighbor interpolation is simple, and the output effect is acceptable in most cases. However, nearest neighbor interpolation can produce artifacts in the image.

bilinear interpolation

Bilinear interpolation, also known as first-order interpolation, is an application of linear interpolation extended to two dimensions and can be obtained through a series of first-order linear interpolation.

Linearity refers to the proportional and straight-line relationship between quantities. Mathematically, it can be understood as a function whose first-order derivative is a constant.

Linear interpolation refers to linearly determining the value of a point on the line connecting the two points based on the values ​​of the two points.

The value of the output pixel is the 2 × 2 2 \times 2 closest to it in the input image$2\times 2$ Weighted average of pixel gray value of sampling points in the neighborhood.

Assume that the vertex coordinates of the known unit square are $f(0,0),f(1,0),f(0,1),f(1,1)$ , as shown in the figure below, any point$f(x,y)$ value.

First, linearly interpolate the two points at the upper end to obtain $f(x,0)$：$f(x,0)=f(0,0)+x[f(1,0)-f(0,0)]$；

Then perform linear interpolation on the two lower vertices to obtain $f(x,1)$：$f(x,1)=f(0,1)+x\left[f\right(1,1,)-f(0,1)]$；

Finally, linear interpolation is performed in the vertical direction to obtain $f(x,y)$：$f(x,y)=f(x,0)+y[f(x,1)-f(x,0)]$；

Arrange to get $f(x,y)$：

$$f(x,y)=[f(1,0)-f(0,0)]x+[f(0,1)-f(0,0)]y+\left[f\right(1,1)+f(0,0)-f(0,1)-f(1,0)]xy+f(0,0)\text{\hspace{1em}(24)}$$

The smoothing effect of bilinear interpolation can degrade image detail, while discontinuities in its slope can lead to undesirable transformation results.

higher order interpolation

The hot spots of bilinear interpolation can be compensated by high-order interpolation, which is usually implemented using convolution.

The value of the output pixel is the closest $4\times The\; weighted\; average\; of\; the\; pixel\; values\; of\; the\; sampling\; points\; in\; the\; 4$ neighborhoods.

Taking cubic interpolation as an example, it uses the following cubic polynomial to approximate the theoretical optimal interpolation function $sinc(x)$。
$$S(x)=\{\begin{array}{cc}1-2|x{|}^2+|x{|}^3,& 0\le |x|<1\\ 4-8|x|+5|x{|}^2-|x{|}^3,& 1\le |x|<2\\ 0& |x|\ge 2\end{array}\text{\hspace{1em}( 25 )}$$
In the above formula$|x|$ is the surrounding pixels along$The\; distance\; between\; the\; x$ direction and the origin. Pixel to be found$(x,The\; gray\; value\; of\; y)$ is obtained by weighted interpolation of the gray values ​​of 16 points around it. The calculation formula is as follows:
$$f(x,y)=f(i+u,j+v)=ABC\text{\hspace{1em}(26)}$$
其中
$$\begin{gathered} A={\left[\begin{array}{c}S(1+v)\\ S(v)\\ S(1-v)\\ S(2-v)\end{array}\right]}^T \\ C=\left[\begin{array}{c}S(1+u)\\ S(u)\\ S(1-u)\\ S(2-u)\end{array}\right] \\ B=\left[\begin{array}{cccc}f(i-1,j-1)& f(i-1,j)& f(i-1,j+1)& f(i-1,j+2)\\ f(i,j-1)& f(i,j)& f(j,j+1)& f(i,j+2)\\ f(i+1,j-1)& f(i+1,j)& f(i+1,j+1)& f(i+1,j+2)\\ f(i+2,j-1)& f(i+2,j)& f(i+2,j+1)& f(i+2,j+2)\end{array}\right] \end{gathered}$$
The cubic interpolation method is usually used in raster displays, which allows scaling operations at any ratio while maintaining image details well.

To be continued~