10. Top-level operations of opencv-python image processing (2) - corner detection

Harris and Shi-Tomas algorithm

learning target

Understand the principles of Harris and Shi-Tomasi algorithms

Ability to utilize Harris and Shi-Tomasi for corner detection

1. Harris corner detection

1. Principle

The idea of ​​Harris corner point detection is to observe the image through a small local window of the image. The characteristic of the corner point is that moving the window in any direction will cause obvious changes in grayscale, as shown in the following figure: Convert the above ideas into mathematical form, that is, the

local The window moves in all directions (u, v) and the sum of all grayscale differences is calculated. The expression is as follows:

where I(x, y) is the image grayscale of the local window, and I(x+u, y+v) is the translation. After the grayscale of the image, w(x, y) is the window function, which can be a rectangular window or a Gaussian window that assigns different weights to each pixel, as shown in the figure below: In corner detection, E(u,

v ) has the largest value, using the first-order Taylor expansion:

where Ix and Iy are the derivatives along the x and y directions, which can be calculated with the sobel operator.
The derivation is as follows:

The M matrix determines the value of E(u, v). Below we use M to find the corner points. M is the quadratic term function of Ix and Iy, which can be expressed in the shape of an ellipse. The long and short semi-axes of the ellipse are given by Eigenvalue of M $l_1$and $l_2$Determined, the direction is determined by the eigenvector, as shown in the following figure:

The relationship between the elliptic function eigenvalues ​​and the corner points, straight lines (edges) and planes in the image is as shown in the following figure: As can be

seen from the above figure, it is divided into three situation:

• A straight line in the image, one eigenvalue is large, one eigenvalue is small, $l_1$>> $l_2$or $l_1$<< $l_2$. Elliptic function values ​​are large in one direction and small in other directions.
• plane in the image. Both eigenvalues ​​are small and approximately equal; the elliptic function values ​​are small in all directions.
• Corner points in the image. Both eigenvalues ​​are large and approximately equal, and the elliptic function value increases in all images.

The corner point calculation method given by Harris does not require the calculation of specific feature values, but calculates a corner response value R to determine the corner point. The calculation formula of R is:

where, detM is the determinant of matrix M; traceM is the trace of matrix M; α is a constant with a value range of 0.04~0.06. In fact, the characteristics are implicit in detM and traceM, because :

Then we judge whether it is a corner point like this: As shown in the figure below:

• Corner points when R is a large positive number.
• Boundary when R is a large negative number.
• When R is a decimal, it is considered a flat area.

2. Specific implementation

The API used to implement Harris detection in OpenCV is:

dst=cv.cornerHarris(src, blockSize, ksize, k)

parameter

• img: Input image with data type float32.
• blockSize: Neighborhood size to be considered in corner detection.
• ksize: The kernel size used for sobel derivation.
• k: The free parameter in the corner detection equation, the value parameter is [0.04, 0.06].
  Code example:

import cv2 as cv
import numpy as np 
import matplotlib.pyplot as plt
# 1 读取图像，并转换成灰度图像
img = cv.imread('./image/chessboard.jpg')
gray = cv.cvtColor(img, cv.COLOR_BGR2GRAY)
# 2 角点检测
# 2.1 输入图像必须是 float32
gray = np.float32(gray)

# 2.2 最后一个参数在 0.04 到 0.05 之间
dst = cv.cornerHarris(gray,2,3,0.04)
# 3 设置阈值，将角点绘制出来，阈值根据图像进行选择
img[dst>0.001*dst.max()] = [0,0,255]
# 4 图像显示
plt.figure(figsize=(10,8),dpi=100)
plt.imshow(img[:,:,::-1]),plt.title('Harris角点检测')
plt.xticks([]), plt.yticks([])
plt.show()

The results obtained are as follows:

Advantages and disadvantages of Harris corner detection:

advantage:

• Rotation invariance, the ellipse rotates through a certain angle but its shape remains unchanged (eigenvalues ​​remain unchanged)
• The affine transformation of image grayscale has partial invariance. Since only the first derivative of the image is used, it is invariant to image grayscale translation transformation and invariant to image grayscale changes.

shortcoming:

• Very sensitive to scale and not geometrically scale invariant
• The extracted corner points are pixel-level.

2. Shi-Tomas corner detection

1. Principle

The Shi-Tomasi algorithm is an improvement on the Harris corner detection algorithm and generally yields better corner points than the Harris algorithm. The corner response function of the Harris algorithm is to subtract the determinant value of the matrix M from the trace of M, and use the difference to determine whether it is a corner point. Later, Shi and Tomasi proposed an improved method: if the smaller of the two eigenvalues ​​of the matrix M is greater than the threshold, it is considered to be a corner point , that is: as

shown

in the following figure: From this figure, it can be seen that only $l_1$and $l_2$When both are greater than the minimum value, it is considered a corner point.

2. Realize

Implement Shi-Tomasi corner detection in OpenCV using API:

corners = cv2.goodFeaturesToTrack ( image, maxcorners, qualityLevel, minDistance )

parameter

• image: input grayscale image
• maxCorners: Get the number of corner points
• qualityLevel: This parameter indicates the lowest acceptable corner quality level, between 0-1.
• minDistance: The minimum Euclidean distance between corner points to avoid adjacent feature points.
  return:
• Corners: The searched corner points, where all corner points below the quality level are excluded, and then the qualified corner points are sorted by quality, and then the corner points near the corners with better quality (less than the minimum Euclidean distance) are Delete, and finally find maxCorners corners and return.
  Code example:

import numpy as np 
import cv2 as cv
import matplotlib.pyplot as plt
# 1 读取图像
img = cv.imread('./image/tv.jpg') 
gray = cv.cvtColor(img,cv.COLOR_BGR2GRAY)
# 2 角点检测
corners = cv.goodFeaturesToTrack(gray,1000,0.01,10)  
# 3 绘制角点
for i in corners:
    x,y = i.ravel()
    cv.circle(img,(x,y),2,(0,0,255),-1)
# 4 图像展示
plt.figure(figsize=(10,8),dpi=100)
plt.imshow(img[:,:,::-1]),plt.title('shi-tomasi角点检测')
plt.xticks([]), plt.yticks([])
plt.show()

The resulting processed image looks like this:

Summarize

Harris algorithm

Idea: Observe the image through a small local window of the image. The characteristic of the corner point is that moving the window in any direction will cause obvious changes in the grayscale of the image.
API:

cv2.cornerHarris()

Shi-Tomasi arithmetic

Improvements to the Harris algorithm can better detect corner points.
API:

cv2.goodFeatureToTrack()