Lesson4-3:OpenCV image feature extraction and description---SIFT/SURF algorithm

learning target

理解 $Principle of S I FT / S U RF$ algorithm,
Ability to use $S I FT / S U RF$ detect key points

SIFT/SURF algorithm

1.1 SIFT principle

In the previous two sections we introduced $H a rr i s$ 和 $S hi - T o ma s i$ corner point detection algorithm. These two algorithms have rotation invariance but not scale invariance. Take the following figure as an example. Corners can be detected in the small image on the left, but after the image is enlarged, the Using the same window, corner points cannot be detected.
Insert image description here
Therefore, below we will introduce a computer vision algorithm, scale-invariant feature transform, namely $S I FT (S c a l e - in v a r ian t f e a t u re t r an s f or m)$ . It is used to detect and describe local features in images. It finds extreme points in the spatial scale and extracts their position, scale, and rotation invariants. This algorithm was published by David Lowe in 1999 and perfected in 2004. Summarize. The application scope includes object recognition, robot map perception and navigation, image stitching, 3D model building, gesture recognition, image tracking and action comparison, etc.

The essence of the SIFT algorithm is to find key points (feature points) on different scale spaces and calculate the direction of the key points. The key points found by SIFT are some very prominent points that will not change due to factors such as lighting, affine transformation, and noise, such as corner points, edge points, bright spots in dark areas, and dark points in bright areas.

1.1.1 Basic process

$L o w e$ 将 $The S I FT$ algorithm is decomposed into the following four steps:

Scale space extreme value detection: Search image locations at all scales. Potential key points that are invariant to scale and rotation are identified through Gaussian difference functions.
Key point positioning: At each candidate position, the position and scale are determined through a well-fitted model. Key points are chosen based on their stability.
Key point direction determination: Based on the local gradient direction of the image, one or more directions are assigned to each key point position. All subsequent operations on the image data are transformed relative to the direction, scale and position of the key points, thus ensuring invariance to these transformations.
Keypoint description: In the neighborhood around each keypoint, the local gradient of the image is measured at a selected scale. These gradients serve as keypoint descriptors, which allow relatively large local shape deformations or illumination changes.

Let's go along $Low steps for$ SIFT $_$ $The implementation process of SIFT algorithm$ $is$ introduced:

1.1.2 Scale space extreme value detection

In different scale spaces, the same window cannot be used to detect extreme points. Small windows are used for small key points, and large windows are used for large key points. In order to achieve the above purpose, we use scale space filters.

Gaussian kernel is the only kernel function that can generate multi-scale space. -"Scale-space theory: A basic tool for analyzing structures at different scales".

The scale space L ( x , y , σ ) of an image $L (x, y, σ)$ , defined as the original image $I (x, y)$ with a variable scale of $2-$ dimensional Gaussian function $G (x, y, σ)$ convolution operation, that is:

$L (x, y, s) = G (x, y, s) * I (x, y)$

in:

$\frac{1}{2πσ^{2}}e^{\frac{x ^{2}+y^{2}}{2σ^{2}}}$

$σ$ is the scale space factor, which determines the degree of blur of the image. At large scales ( $(large σ$ value) represents the general information of the image. In small scale ( $(small σ$ value) represents the detailed information of the image.

When computing the discrete approximation of a Gaussian function, at approximately Pixels outside the 3σ distance can be regarded as ineffective, and the calculation of these pixels can be $ignored$ $.$ Therefore, in practical applications, only calculate $(6 p + 1) * (6 p + The Gaussian convolution kernel of 1)$ can ensure the influence of relevant pixels.

Next, we construct a Gaussian pyramid of the image, which is obtained by blurring and downsampling the image using a Gaussian function. During the construction of the Gaussian pyramid, the image is first doubled, a Gaussian pyramid is constructed based on the expanded image, and then the Gaussian pyramid is constructed. The image at this size is subjected to Gaussian blur, and the collection of several blurred images constitutes an $O c t a v e$ , and then $Select an image under Octave for downsampling . The$ length and width are doubled respectively, and the image area becomes a quarter of the original one $.$ This image is the next $The initial image of Octave is completed$ based on the initial image and belongs to this $Octave$ $Gaussian blur processing of O c t a v e$ , and so on to complete all the octave construction required by the entire algorithm, so that the Gaussian pyramid is constructed. The entire process is shown in the figure below:

Insert image description here
Utilize $L o G$ (Laplacian of Gaussian method), which is the second derivative of the image, can detect key point information of the image at different scales to determine the feature points of the image. But $L o G$ has a large amount of calculation and low efficiency. by subtracting the images of two adjacent Gaussian scale spaces. $Do G$ (difference of Gaussians) to approximate $L o G$ 。

To calculate $Do GWe$ in the Gaussian pyramid $The subtraction of two adjacent layers in Octave forms a Gaussian$ difference $pyramid$ $.$ As shown below: Insert image description here

Gaussian Difference Pyramid No. Group $1$ $Level 1$ is obtained by subtracting level 1 of group 1 from level 2 of group 1 of the Gaussian pyramid. By analogy, each differential image is generated group by group and layer by layer, and all differential images constitute a differential pyramid. Summed up as $D OG$ Pyramid No. $o$ Group No. $The l-$ with Gaussian pyramid $Group o$ No. $l + 1st$ floor minus $o$ Group No.Obtained from layer $l .$ Follow-up $S if t feature$ points are extracted inConducted on $D$ $OG Pyramid$

In $After Do G$ is completed, local maxima can be searched in different scale spaces. For a pixel in the image, it needs to be consistent with $8$ neighborhood, and adjacent $Compared to 18 (2 x 9)$ points. If it is a local maximum, it may be a critical point. Basically the keypoint is the best representation of the image in the corresponding scale space. As shown in the figure below:
Insert image description here
the search process starts from the second layer of each group, with the second layer as the current layer, and theEach point in the $Do$ $G$ $3 \times The cube of 3$ , the upper and lower layers of the cube are the first and third layers. In this way, the extreme points obtained through the search have position coordinates ( $Do G$ 's image coordinates), and spatial scale coordinates (layer coordinates). After the second layer search is completed, the third layer is used as the current layer. The process is similar to the second layer search. When $S = 3$ , search for $3$ floors, so inin $D$ $OG$ $S + Level 2$ in the pyramid built by the first user $S + 3$ floors.

1.1.3 Key point positioning

Due to $Do G$ is sensitive to noise and edges, so the local extreme points detected in the Gaussian difference pyramid above need to be further inspected before they can be accurately positioned as feature points.

Use the Taylor series expansion of the scale space to obtain the exact location of the extreme value. If the gray value of the extreme point is less than the threshold (usually $0.03$ or $0.04$ ) will be ignored. In OpenCV this kind of threshold is calledcontrastThreshold.

$The Do G$ algorithm is very sensitive to boundaries, so we must remove the boundaries. In addition to corner point detection, $H$ $a$ $r$ $i$ $s algorithm can also be used to detect boundaries.$ From $In the H arris corner$ detection algorithm, a boundary is detected when one feature value is much larger than another feature value $.$ That's in $The poor key points in the Do G$ algorithm have a larger principal curvature in the direction parallel to the edge and a smaller curvature in the direction perpendicular to the edge. If the ratio between the two is higher than a certain threshold (called the boundary in OpenCV threshold), the key point is considered to be a boundary and will be ignored. Generally, the threshold is set to $10$ 。

By removing low contrast and boundary key points, we get the key points we are interested in.

1.1.4 Determining the direction of key points

After the above two steps, the key points of the image have been completely found, and these key points have scale invariance. In order to achieve rotation invariance, it is also necessary to assign a direction angle to each key point, that is, to obtain a direction reference based on the neighborhood structure of the Gaussian scale image where the detected key point is located.

For any key point, we collect the gradient features (amplitude and angle) of all pixels in the area of the Gaussian pyramid image with r as the radius, the radius $r$ 的：
$r = 3 \times 1.5 p$

where σ is the key point $The scale of the image of oc t a v e$ can be used to obtain the corresponding scale image.

The calculation formula for the magnitude and direction of the gradient is:

$,y)=\sqrt{(L(x+1,y)-L(x-1,y)^{2}+(L(x,y+1)-L(x,y-1))^ {2}}$

$θ(x,y)=arctan(\frac{L(x,y+1)-L(x,y-1)}{L(x+1,y)-L(x-1,y)})$

The calculation results of the neighborhood pixel gradient are shown in the figure below:
Insert image description here

After completing the key point gradient calculation, use the histogram to count the gradient amplitude and direction of the pixels in the key point neighborhood. The specific method is to convert $360°$ is divided into $36$ columns, every $10°$ is a column, and then in the area with r as the radius, find the pixels with the gradient direction in a certain column, and then add their amplitudes together as the height of the column. Because the contribution of the gradient amplitude of pixels to the central pixel in the area with radius r is different, the amplitude needs to be weighted, using Gaussian weighting with a variance of 1.5σ $1.5σ$ . $_$ is drawn to simplify the figure. $Histogram in 8$ directions.
Insert image description here
Each feature point must be assigned a main direction and one or more auxiliary directions. The purpose of adding auxiliary directions is to enhance the robustness of image matching. The definition of the auxiliary direction is that when the height of a cylinder is greater than 80% of the height of the main direction cylinder, the direction represented by the cylinder is the auxiliary direction for the feature point.

The peak of the histogram, that is, the direction represented by the highest column is the main direction of the image gradient in the neighborhood of the feature point, but the angle represented by the column is a range, so we also need to perform interpolation fitting on the discrete histogram, to get a more accurate direction angle value. Use a parabola to fit the discrete histogram, as shown in the figure below:

Insert image description here
After obtaining the main direction of the key points of the image, each key point has three pieces of information $(x, y, s, θ)$ : position, scale, direction. From this we can determine a $S I FT$ characteristic area. is usually represented by a circle with an arrow or an arrow directly $There are three values in the S I FT$ area: the center represents the location of the feature point, the radius represents the scale of the key point, and the arrow represents the direction. As shown below:
Insert image description here

1.1.5 Key point description

Through the above steps, each key point is assigned position, scale and orientation information. Next we build a descriptor for each keypoint that is both distinguishable and invariant to certain variables, such as lighting, viewing angle, etc. Moreover, the descriptor includes not only the key points, but also the pixels surrounding the key points that contribute to it. The main idea is to abstract the image information by dividing the image area around the key point into blocks, calculating the gradient histogram within the block, and generating a feature vector.

The descriptor is related to the scale at which the feature point is located, so we generate the corresponding descriptor on the Gaussian scale image where the key point is located. Taking the feature point as the center, divide its nearby neighborhood into $d * d$ sub-region (generally $d = 4$ ), each sub-region is a square with side length $3 σ$ , considering that in actual calculation, cubic linear interpolation is required, so the feature point neighborhood is $3 σ (d + 1) * 3 σ (d + 1)$ , as shown in the figure below:
Insert image description here
In order to ensure the rotation invariance of the feature point, take the feature point as the center and rotate the coordinate axis to the main direction of the key point, as shown in the figure below: Calculate the gradient of the

pixels in the sub-region, And according to $p =$ Gaussian weighting is performed at $0.5$ $d , and then interpolation calculation is performed to obtain the gradient in eight directions of each seed point. The interpolation method is shown in the figure below:$
Insert image description here

covering it $It is obtained by interpolation of 4$ sub-regions. As the red point in the picture falls at $Line 0$ and $1$ row, contributes to both rows. RightLine $0$ $The contribution factor of the 3$ columns of seed points is $d r$ , for No. $The contribution factor of row 1$ and column 3 is $1 - d r$ , similarly, the contribution factor to the two adjacent columns is $d c$ 和 $1 - d c$ , the contribution factor to the two adjacent directions is $d o$ 和 $1 - d o$ . Then the final gradient accumulated in each direction is:

$weight=w*dr^{k}(1-dr)^{1-k}dc^{m}(1-dc)^{1-m}do^{n}(1-do)^{1-n}$

where $k, m, n$ are $0$ or $1$ . The above statistics of 4*4*8=128 gradient information are the feature vectors of the key points. By sorting the feature vectors of each key point according to the feature points, the SIFT feature description vector is obtained.

1.1.6 Summary

$SIFT has unparalleled advantages in extracting invariant features of images, but it is not perfect. It still has shortcomings such as low real-time performance, sometimes fewer feature points,$ and the inability to accurately extract feature points for targets with smooth edges. Since SIFT $Since the advent of the SIFT algorithm, people have been$ optimizing and improving it, the most famous of which is $S U RF$ algorithm.

1.2 SURF principle

Using $The execution speed of the SIFT$ algorithm for key point detection and description is relatively slow, and a faster algorithm is needed $.$ In 2006 Bay proposed $S U RF$ algorithm, is $An enhanced version of the SIFT$ algorithm. It has a small amount of calculation and fast operation speed. The extracted features are similar to $SIFT$ $S I FT$ is almost the same, compare it with $The comparison of S I FT$ algorithm is as follows:

Insert image description here

1.3 Implementation

The process of using SIFT to detect key points in OpenCV is as follows:

1. Instantiate sift

sift = cv.xfeatures2d.SIFT_create()

2. Use sift.detectAndCompute() to detect key points and calculate

kp,des = sift.detectAndCompute(gray,None)

parameter:

gray: Image for key point detection, note that it is a grayscale image

return:

kp: Key point information, including position, scale, and direction information
des: Key point descriptor, each key point corresponds to 128 feature vectors of gradient information

3. Draw the key point detection results on the image

cv.drawKeypoints(image, keypoints, outputimage, color, flags)

parameter:

image: The original image
keypoints: Key point information, drawn on the image
outputimage: Output picture, which can be the original image
color: Color setting, （b,g,r）change the color of the brush by modifying the value, b=blue, g=green, r=red.
flags: Logo setting of drawing function
1. cv2.DRAW_MATCHES_FLAGS_DEFAULT:Create an output image matrix, use the existing output image to draw matching pairs and feature points, and only draw the intermediate points for each key point
2. cv2.DRAW_MATCHES_FLAGS_DRAW_OVER_OUTIMG: does not create an output image matrix, but instead draws matching pairs on the output image
3. cv2.DRAW_MATCHES_FLAGS_DRAW_RICH_KEYPOINTS: Draw a key point graphic with size and direction for each feature point
4. cv2.DRAW_MATCHES_FLAGS_NOT_DRAW_SINGLE_POINTS: Single point feature points are not drawn.

$The application of the S U RF$ algorithm is consistent with the above process and will not be described in detail here.

Example:

Leveraging $The SIFT algorithm$ detects key points on CCTV pictures and plots them:

import cv2 as cv 
import numpy as np
import matplotlib.pyplot as plt
# 1 读取图像
img = cv.imread('./image/tv.jpg')
gray= cv.cvtColor(img,cv.COLOR_BGR2GRAY)
# 2 sift关键点检测
# 2.1 实例化sift对象
sift = cv.xfeatures2d.SIFT_create()

# 2.2 关键点检测：kp关键点信息包括方向，尺度，位置信息，des是关键点的描述符
kp,des=sift.detectAndCompute(gray,None)
# 2.3 在图像上绘制关键点的检测结果
cv.drawKeypoints(img,kp,img,flags=cv.DRAW_MATCHES_FLAGS_DRAW_RICH_KEYPOINTS)
# 3 图像显示
plt.figure(figsize=(8,6),dpi=100)
plt.imshow(img[:,:,::-1]),plt.title('sift检测')
plt.xticks([]), plt.yticks([])
plt.show()

result:

Insert image description here

Summarize

SIFT principle:

Scale space extreme value detection: Construct Gaussian pyramid, Gaussian difference pyramid, and detect extreme points.
Key point positioning: remove the influence of small contrast and edges on extreme points.
Key point direction determination: Use gradient histogram to determine the direction of key points.
Key point description: Divide the image area around the key point into blocks, calculate the gradient histogram within the block, generate a feature vector, and describe the key point information.

API：cv.xfeatures2d.SIFT_create()

SURF algorithm:

Improvements to the SIFT algorithm include improvements in scale space extreme value detection, key point direction determination, and key point description, improving efficiency.