[CV & Unity] Principle of Affine Transformation

What is an affine transformation?

Affine transformation (Affine Transformation) is the transformation of the space Cartesian coordinate system, from one two-dimensional coordinates to another two-dimensional coordinates, affine transformation is a linear transformation, it maintains the "parallelism" and "straightness" of the image ", that is, the original straight lines and parallel lines in the image, after the transformation, the original straight lines and parallel lines are still maintained. The special transformations commonly used in affine transformation are Translation, Scale, Flip, and Rotation ) and Shear.

The difference between linear transformation and affine transformation

linear transformation	affine transformation
It is a straight line before the transformation, and it is still a straight line after the transformation. The ratio of the line remains unchanged. It was the origin before the transformation, and it is still the origin after the transformation.	It is a straight line before the transformation, and it is still a straight line after the transformation. The ratio of the line remains unchanged

Does the difference between linear and affine transformations change at the origin

Two main categories of spatial transformations

affine transformation

Affine transformations: translation, rotation, scaling, shearing, reflection

Formula expression of affine transformation

$\left(\begin{array}{c}{X}_2\\ Y_2\\ 1\end{array}\right)=\left[\begin{array}{ccc}{A}_{11}& A_{12}& B_{11}\\ A_{21}& A_{22}& B_{12}\\ 0& 0& 1\end{array}\right]\times \left(\begin{array}{c}{X}_1\\ Y_1\\ 1\end{array}\right)$

Through the above matrix transformation, the image can be translated, scaled, rotated and cut, as shown in the following figure

The application of affine transformation in the field of ML machine learning

The idea of ​​affine transformation corresponds to the homography matrix in the field of machine learning "image processing"

In three-axis coordinates $XYZ，Z=1$ This is somewhat similar to three-dimensional homogeneous coordinates. The homography matrix is ​​mainly used to solve two problems,

1. Express the perspective transformation of a plane in the real world (main world camera perspective in Unity) and its corresponding image
2. Transforming an image from one view to another through perspective transformation
   

The middle zero points in the above figure respectively represent any two points in the two planes, a1, a2 and b1, b2 are linear vectors in the two directions corresponding to these two points. For the relationship between these two planes, we can use these points to determine the direct relationship between the two planes in one step, and the relationship between the two planes is described by the homography matrix as follows: ph → = H q ⃗ h
$\overrightarrow{p^h}=H{\vec{q}}^{\vec{h}}$
where $-\overrightarrow{p^h}，{\vec{q}}^{\vec{h}}$A homogeneous coordinate vector representing three dimensions
This relationship is called a planar homography. There are some derivations of mathematical knowledge in this, and I will not prove it here, just to understand how this concept came about.

• In addition, two computer graphics application scene distributions are texture rendering and computing flat shadows.
• It is used to solve the alignment problem when implementing image stitching

projection transformation

$\left(\begin{array}{l}{x}_1^t\\ x_2^t\\ x_3^t\end{array}\right)=\left[\begin{array}{lll}{h}_{11}& h_{12}& h_{13}\\ h_{21}& h_{22}& h_{23}\\ h_{31}& h_{32}& h_{33}\end{array}\right]\left(\begin{array}{l}{x}_1\\ x_2\\ x_3\end{array}\right)$
Difference Between Projective and Affine Transformations

projection transformation	affine transformation
It is a straight line before the transformation, and it is still a straight line after the transformation. The ratio of the two lines that are parallel before the transformation is not necessarily parallel after the transformation.	It is a straight line before the transformation, and it is still a straight line after the transformation. The ratio of the two lines that are parallel before the transformation is not necessarily parallel after the transformation.

The only difference between these affine and projective transformations is in the last row of the transformation matrix. For affine transformations, the first two elements of the row are zero. This results in the following differences in operational properties:

• Projective transformations do not preserve parallelism, length, and angle.

• Affine transformations, unlike projective transformations, preserve parallelism.

A projective transformation can be expressed as the transformation of an arbitrary quadrilateral (i.e. a four-point system) to another quadrilateral. Affine transformations are transformations of triangles. The following diagram illustrates this:

Intuitive example:

Affine transformation:

Affine transformation C# code implementation

To apply an affine transformation in Graphics Mill, follow these steps:

1. Specify source and destination triangles.

2. Use Matrix.CreateFromAffinePoints(PointF[], PointF[]) to create an affine transformation matrix. Pass the previously specified point as a method argument.

3. Transforms are created using the MatrixTransform.#ctor constructor. Here the previously created matrix is ​​the parameter of the constructor.

4. Transformation is applied by calling the MatrixTransform.Apply method.

   using (var bitmap = new Bitmap(@"Images\in.jpg"))
  {
    
    
    System.Drawing.PointF[] source = {
    
    
        new System.Drawing.PointF(0f, 0f),
        new System.Drawing.PointF(0f, 80f),
        new System.Drawing.PointF(80f, 0f)
    };
    


    System.Drawing.PointF[] target = {
    
    
        new System.Drawing.PointF(20, 0f),
        new System.Drawing.PointF(0f, 80f),
        new System.Drawing.PointF(80f, 0f)
    };

    using (var matrix = Matrix.CreateFromAffinePoints(source, target))
    {
    
    
        using (var transform = new MatrixTransform(matrix))
        {
    
    
            using (var result = transform.Apply(bitmap))
            {
    
    
                result.Save(@"Images\Output\out.jpg");
            }
        }
    }
}

Projective transformation:

C# code implementation of projection transformation

To apply a projective transformation in Graphics Mill, follow these steps:

1. Specifies the source and destination quads.
2. Use Matrix.CreateFromProjectivePoints(PointF[], PointF[]) to create a projection transformation matrix. Pass the previously specified point as a method argument.
3. Transforms are created using the MatrixTransform.#ctor constructor. Here the previously created matrix is ​​the parameter of the constructor.
4. Transformation is applied by calling the MatrixTransform.Apply method.

 
using (var bitmap = new Bitmap(@"Images\in.jpg"))
{
    
    
    System.Drawing.PointF[] source = {
    
    
        new System.Drawing.PointF(0f, 0f),
        new System.Drawing.PointF(0f, bitmap.Height),
        new System.Drawing.PointF(bitmap.Width, bitmap.Height),
        new System.Drawing.PointF(bitmap.Width, 0f)
    };

    System.Drawing.PointF[] target = {
    
    
        new System.Drawing.PointF(0f, 0f),
        new System.Drawing.PointF(0f, bitmap.Height),
        new System.Drawing.PointF(bitmap.Width * 0.75f, bitmap.Height - 50f),
        new System.Drawing.PointF(bitmap.Width * 0.75f, 80f)
    };

    using (var matrix = Matrix.CreateFromProjectivePoints(source, target))
    {
    
    
        using (var transform = new MatrixTransform(matrix))
        {
    
    
            using (var result = transform.Apply(bitmap))
            {
    
    
                result.Save(@"Images\Output\out.jpg");
            }
        }
    }
}

Application of Affine Transformation in Machine Learning

In the recognition of MINST handwritten data set with CNN model, the method of affine transformation is introduced to improve the recognition accuracy.
Paper: https://arxiv.org/abs/1506.02025

References:

Affine and Projective Transformation - Graphics Mill
Review: STN — Spatial Transformer Network (Image Classification)
How to explain the concept of "affine transformation" in a popular way? - Ma's answer - Zhihu