OpenCV (image processing) - based on Python - feature detection - feature point matching

1. Image features

Image features refer to meaningful image regions that are unique and easy to identify, such as corners, spots, and high-density areas. And why do corners have important features?
Look at the picture below:

Observing the three pictures of ABD, it is not easy for us to know the position of the images, and it is especially easy for us to find their corresponding positions in the original picture for the three pictures of CEF. This is because ABD is relatively smooth, and it is not easy for us to find out their characteristics , while the three images of CEF have obvious contour features, and there are many edge features that are corner points , so the corner points in an image are very important.
In order for computers to have the ability to recognize pictures, they must first learn to extract the features of the picture and extract the corners of the image.

corner feature

• The most important of the features are the corners
• is the pixel corresponding to the maximum value of the gray gradient
• intersection of two lines
• Extreme point (maximum first derivative but zero second derivative)

2. Harris Assay

Principle:

Figure 1 does not change when moving up, down, left, or right, indicating that Figure 1 is not a corner point; Figure 2 does not change up and down, but changes when moving left and right, indicating that Figure 2 is an edge; Figure 3 has changes when moving up, down, left, and right, indicating that Figure 3 is a corner point.

cornerHarris()

dst = cv2.cornerHarris(img, blockSize, ksize, k)
img: the image to be detected, preferably a grayscale image
blockSize: the size of the detection window, generally 2
ksiez: Sobel convolution kernel, generally 3
k: weight Coefficient, empirical value, generally between 0.02 and 0.04
Return value: dst is not an image, but a collection of points
shortcoming:k is an experience value that needs to be adjusted by yourself

import cv2
import numpy as np

# 读取图片，并转为灰度图
img = cv2.imread('./image/chess.png')
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

# Harris角点检测
dst = cv2.cornerHarris(gray, 2, 3, 0.02)

# 表示img中dst值大于0.01*dst.max()的像素点索引出来，然后将这些像素点的值赋为 [0,0,255]。
img[dst > 0.01*dst.max()] = [0, 0, 255]
cv2.imshow('img', img)

cv2.waitKey(0)

3. Shi-Tomasi corner detection

Shi-Tomasi is an improvement of Harris corner detection, because the stability of Harris corner detection is related to k, and k is an empirical value, which is related to human subjective factors. Shi-Tomasi avoids this problem.

goodFeaturesToTrack()

dst = cv2.goodFeaturesToTrack(img, maxCorners, qualityLevel, minDistance)
img: the image to be detected, preferably a grayscale image
maxCorners: the maximum number of corner points, a value of 0 means unlimited
qualityLevel: a positive number less than 1.0, generally between 0.01-1.0
minDistance: minimum Euclidean distance between corners, ignore points smaller than this distance
mask: area of ​​interest, do not fill in default all images
blockSize: detection window size
useHarrisDetector: whether to use Harris algorithm, True/False
k: The default is 0.04

import cv2
import numpy as np

# 读取图片，并转为灰度图
img = cv2.imread('./image/chess.png')
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

# Harris角点检测
# dst = cv2.cornerHarris(gray, 2, 3, 0.02)

maxCorners = 100
ql = 0.01
minDistance = 10
# ShiTomasi角点检测
corners = cv2.goodFeaturesToTrack(gray, maxCorners, ql, minDistance)
# corners为float32类型，需要转化为整型
corners = np.int0(corners)
# 表示img中dst值大于0.01*dst.max()的像素点索引出来，然后将这些像素点的值赋为 [0,0,255]。
# img[dst > 0.01*dst.max()] = [0, 0, 255]

for i in corners:
    x, y = i.ravel() # 将迭代的数组，转化成一维的
    cv2.circle(img, (x, y), 3, (0, 0, 255), -1)

cv2.imshow('img', img)

cv2.waitKey(0)

4.Scale-Invariant Feature Transform(SIFT)

This algorithm is a feature detection that has nothing to do with zooming. For example, it used to be a corner point, but after the image is enlarged or reduced, due to the change of the image scale, it turned out to be a corner. After zooming in, it may not be a corner; it was originally a corner, but after zooming out Maybe it's the horns. And this algorithm just solves this problem.

---

SIFT can calculate key points and descriptors, key points: position, size and direction; key point descriptors: a set of vector values ​​​​recording the pixels that contribute to it around the key point, which are not affected by affine transformation and illumination transformation Impact.

SIFT syntax

• First create the SIFT object
• To detect, kp = sift.detect(img,...)
• Draw key points, drawKeypoints(gray, kp, img)
• Detected from the grayscale image gray, the key point kp is drawn in the original image img

Create a SIFT object
sift = cv2.xfeatures2d.SIFT_create(do not write anything)
kp = sift.detect(img, mask)
img: is to detect pictures, try to use grayscale image
mask: mask, indicating which part of the picture to detect , generally fill in None (full image)

import cv2
import numpy as np

# 读取图片，并转为灰度图
img = cv2.imread('./image/chess.png')
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

# 创建sift对象
sift = cv2.xfeatures2d.SIFT_create()
# 关键点检测
kp = sift.detect(gray, None)
# 使用关键点画图
cv2.drawKeypoints(gray, kp, img)
cv2.imshow('img', img)

cv2.waitKey(0)

---

This interface can calculate key points and descriptors at once
kp, des = sift.detectAndCompute(img, mask)
img: is to detect pictures, try to use grayscale image
mask: mask, indicating which part of the picture to detect, generally fill in None (full picture)

import cv2
import numpy as np

# 读取图片，并转为灰度图
img = cv2.imread('./image/chess.png')
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

# 创建sift对象
sift = cv2.xfeatures2d.SIFT_create()
# kp = sift.detect(gray, None)

kp, des = sift.detectAndCompute(gray, None)

cv2.drawKeypoints(gray, kp, img)
cv2.imshow('img', img)

cv2.waitKey(0)

---

SIFT Cons: Slow

5.SURF

Due to the slow speed of SIFT, the SURF algorithm was produced, which is faster than SIFT.

• First create the SURF object
• to detect, kp, des = surf.detectAndCompute(img, mask)
• Draw key points, drawKeypoints(gray, kp, img)
• Detected from the grayscale image gray, the key point kp is drawn in the original image img

Create a surf object
surf = cv2.xfeatures2d.SURF_create()
kp, des = surf.detectAndCompute(img, mask)
img: is to detect pictures, try to use grayscale image
mask: mask, indicating which part of the picture to detect, generally Fill in None (full image)
kp: the detected keyword
des: is a descriptor, used for feature matching

import cv2
import numpy as np

# 读取图片，并转为灰度图
img = cv2.imread('./image/chess.png')
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

# # 创建sift对象
# sift = cv2.xfeatures2d.SIFT_create()

# 创建surf对象
surf = cv2.xfeatures2d.SURF_create()

# 计算关键点和描述子
kp, des = surf.detectAndCompute(gray, None)

cv2.drawKeypoints(gray, kp, img)
cv2.imshow('img', img)

cv2.waitKey(0)

Note: This algorithm needs to use the 3.4.2.17 version of the opencv library due to the payment

6.ORB(Oriented FAST and Rotated BRIEF)

Among them, FAST is used for feature detection, and BRIEF describes the detected feature points, which speeds up the establishment of feature descriptors and greatly reduces the time of feature matching.
The advantages of ORB are: fast speed, can be used for real-time monitoring; free of charge.

First create the ORB object
orb = cv2.ORB_create()
calculate feature points and descriptor
kp, des = orb.detectAndCompute(img, mask)
use kp to draw feature points
cv2.drawKeypoints(gray, kp, img)

img: It is to detect the image, try to use the grayscale image
mask: Mask, indicating which part of the image to detect, generally fill in None (full image)

import cv2
import numpy as np

# 读取图片，并转为灰度图
img = cv2.imread('./image/chess.png')
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

# 创建orb对象
orb = cv2.ORB_create()
# 对ORB进行检测
kp, des = orb.detectAndCompute(gray, None)

cv2.drawKeypoints(gray, kp, img)
cv2.imshow('img', img)

cv2.waitKey(0)

---

The detected feature points are getting fewer and fewer, so while the detection speed is improved, the accuracy is also lost.

7. Feature matching

Matching is performed according to the descriptor of the feature point. The feature matching methods include:

• BF (Brute-Force), violent feature matching method
• FLANN Fastest Neighborhood Feature Matching Method

7.1 Violent feature matching

It uses the descriptor of each feature point in the first set to match the descriptors of all feature points in the second set, calculates the gap between them, and returns the closest match

• Create a matcher, bf = cv2.BFMatcher(normType, crossCheck)
• Perform feature matching, match = bf.match(des1, des2)
• Draw matching points, cv2.drawMatches(img1, kp1, img2, kp2,…)

Create a matcher
bf = cv2.BFMatcher(normType, crossCheck)
normType: NORM_L1 (SIFT), NORM_L2 (SURF), HAMMING1 (ORB)
crossCheck: Whether to perform cross-matching, the default is false
to match the descriptor
match = bf.match(dst1 , dst2)
dst1: the descriptor of the graph to be searched
dst2: the descriptor that can be matched from the library
Draw the same place
img3 = cv2.drawMatchers(img1, kp1, img2, kp2, match, None)
img1, kp1: to search The graph, and the feature point
img2, kp2: the graph and the feature point matched from the library match
: the matched descriptor

import cv2
import numpy as np

# 读取图片，并转为灰度图
img1 = cv2.imread('./image/opencv_search.png')
img2 = cv2.imread('./image/opencv_orig.png')
gray1 = cv2.cvtColor(img1, cv2.COLOR_BGR2GRAY)
gray2 = cv2.cvtColor(img2, cv2.COLOR_BGR2GRAY)

# 创建orb对象
orb = cv2.ORB_create()
# 对ORB进行检测
kp1, dst1 = orb.detectAndCompute(gray1, None)
kp2, dst2 = orb.detectAndCompute(gray2, None)

# 创建匹配器
bf = cv2.BFMatcher(cv2.NORM_HAMMING)
# 描述子进行匹配
match = bf.match(dst1, dst2)

img3 = cv2.drawMatches(img1, kp1, img2, kp2, match, None)

cv2.imshow('img3', img3)

7.2 FLAN

• FLANN is faster when doing batch feature matching
• Less accurate since it uses a close approximation

Create a matcher
flann = cv2.FlannBasedMatcher()
for feature matching
match = flann.match()
draw matching points
img3 = cv2.drawMatches()

import cv2
import numpy as np

# 读取图片，并转为灰度图
img1 = cv2.imread('./image/opencv_search.png')
img2 = cv2.imread('./image/opencv_orig.png')
gray1 = cv2.cvtColor(img1, cv2.COLOR_BGR2GRAY)
gray2 = cv2.cvtColor(img2, cv2.COLOR_BGR2GRAY)

# 创建orb对象
orb = cv2.ORB_create()
# 对ORB进行检测
kp1, dst1 = orb.detectAndCompute(gray1, None)
kp2, dst2 = orb.detectAndCompute(gray2, None)
# 判断描述子的数据类型，若不符合，则进行数据替换
if dst1.dtype != 'float32':
    dst1 = dst1.astype('float32')
if dst2.dtype != 'float32':
    dst2 = dst2.astype('float32')

# 创建匹配器(FLANN)
flann = cv2.FlannBasedMatcher()
# 描述子进行匹配计算
matches = flann.match(dst1, dst2)
img3 = cv2.drawMatches(img1, kp1, img2, kp2, matches, None)

cv2.imshow('img3', img3)