Reference

https://blog.csdn.net/lbaihao/article/details/78919700
https://docs.opencv.org/4.x/dc/d0d/tutorial_py_features_harris.html

1 Mathematical derivation

Harris corner detection principle:

The Harris algorithm uses the autocorrelation of the image gray in the window to determine the corner point, sets a window, and moves in the image, and calculates the autocorrelation coefficient of the image of the area where the window is located before and after the movement.

insert image description here
The autocorrelation function is calculated as follows, (x, y) is the center position of the window, w(u, v) is the weight (generally a Gaussian function), L represents the window, and (u, v) represents the image position in the window.
$\mathrm{c}(\mathrm{x}, \mathrm{y}, \Delta \mathrm{x}, \Delta \mathrm{y})=\sum_{(\mathrm{u}, \mathrm{v} ) \in \mathrm{L}} \mathrm{w}(\mathrm{u}, \mathrm{v}) \cdot[\mathrm{I}(\mathrm{u}, \mathrm{v})-\ mathrm{I}(\mathrm{u}+\Delta \mathrm{x}, \mathrm{v}+\Delta \mathrm{y})]^{2}$
Let the infinitesimal function I of
${\color{Brown} \begin {array}{c} \mathrm{I}(\mathrm{u}+\Delta \mathrm{x}, \mathrm{v}+\Delta\mathrm{y})=\mathrm{I}(\mathrm{ u}, \mathrm{v})+\mathrm{I}_{\mathrm{x}}(\mathrm{u}, \mathrm{v}) \cdot \Delta \mathrm{x}+\mathrm{I }_{\mathrm{y}}(\mathrm{u}, \mathrm{v}) \cdot \Delta \mathrm{y}+\cdots+\mathrm{R}_{\mathrm{n}}(\mathrm {u}, \mathrm{v}) \\\approx \mathrm{I}(\mathrm{u}, \mathrm{v})+\mathrm{I}_{\mathrm{x}}(\mathrm{ u}, \mathrm{v}) \cdot \Delta \mathrm{x}+\mathrm{I}_{\mathrm{y}}(\mathrm{u}, \mathrm{v}) \cdot \Delta\ mathrm{y}\end{array}}$
Substituting into the autocorrelation function, we get: $\begin{aligned} \mathrm{c}(\mathrm{x}, \mathrm{y}, \Delta \mathrm{x}, \Delta\mathrm{y}) &=\sum_{(\mathrm{u}, \mathrm{v }) \epsilon \mathrm{L}} \mathrm{w}(\mathrm{u}, \mathrm{v}) \cdot[\mathrm{I}(\mathrm{u}, \mathrm{v})- \mathrm{I}(\mathrm{u}+\Delta \mathrm{x}, \mathrm{v}+\Delta\mathrm{y})]^{2} \\ & \approx \sum_{(\mathrm {u}, \mathrm{v}) \in \mathrm{L}} \mathrm{w}(\mathrm{u}, \mathrm{v}) \cdot\left\{\mathrm{I}(\mathrm {u}, \mathrm{v})-\left[\mathrm{I}(\mathrm{u}, \mathrm{v})+\mathrm{I}_{\mathrm{x}}(\mathrm{ u},\mathrm{v}) \cdot \Delta \mathrm{x}+\mathrm{I}_{\mathrm{y}}(\mathrm{u}, \mathrm{v}) \cdot \Delta \mathrm{y }\right]\right\}^{2} \\ &=\sum_{(\maths{u}, \maths{v}) \in \maths{L}} \maths{w}(\maths{u }, \mathrm{v})\left[\mathrm{I}_{\mathrm{x}}(\mathrm{u}, \mathrm{v}) \cdot \Delta \mathrm{x}+\mathrm{ I}_{\mathrm{y}}(\mathrm{u}, \mathrm{v}) \cdot \Delta \mathrm{y}\right]^{2} \end{aligned}$
Expanding the square term and writing it in matrix form gives:
$\begin{array}{l} {\left[I_{x}(u, v) \cdot \Delta x+I_{y}(u, v) \cdot \Delta y\right]^{2}} \\ =I^{2}{ }_{x}(u, v) \Delta x^{2}+I^{2}{ }_{y}(u, v) \Delta y^{2}+2 I_{x}(u, v) I_{y}(u, v) \Delta x \Delta y \\ =I^{2}{ }_{x}(u, v) \Delta x^{2}+I^{2}{ }_{y}(u, v) \Delta y^{2}+2 I_{x}(u, v) I_{y}(u, v) \Delta x \Delta y \\ =\left[\begin{array}{lc} I_{x}^{2}(u, v) \Delta x+I_{x}(u, v) I_{y}(u, v) \Delta y & I_{y}^{2}(u, v) \Delta y+I_{x}(u, v) I_{y}(u, v) \Delta x \end{array}\right] \cdot\left[\begin{array}{l} \Delta x \\ \Delta y \end{array}\right] \\ =\left[\begin{array}{ll} \Delta x & \Delta y \end{array}\right] \cdot\left[\begin{array}{cc} I_{x}^{2}(u, v) & I_{x}(u, v) I_{y}(u, v) \\ I_{x}(u, v) I_{y}(u, v) & I_{y}^{2}(u, v) \end{array}\right] \cdot\left[\begin{array}{l} \Delta x \\ \Delta y \end{array}\right] \end{array}$

Substituting into the autocorrelation function, we get:
$\begin{array}{l} c(x, y, \Delta x, \Delta y) \approx \sum_{(u, v) \in L} w(x, y)\left[I_{x}(u,v) \cdot \Delta x+I_{y}(u, v) \cdot \Delta y\right]^{2}\\ \\ =\sum_{(u, v) \in L} w(x, y)\left[I^{2}{ }_{x}(u, v) \Delta x^{2}+I^{2}{ }_{y}(u, v) \Delta y^{2}\right. \left.+2 I_{x}(u, v) I_{y}(u, v) \Delta x \Delta y\right]\\ \\ =\sum_{(u, v) \in L} w(x, y)\left[\begin{array}{ll} \Delta x & \Delta y \end{array}\right]\left[\begin{array}{cc} I_{x}^{2}(u, v) & I_{x}(u, v) I_{y}(u, v) \\ I_{x}(u, v) I_{y}(u, v) & I_{y}^{2}(u, v) \end{array}\right]\left[\begin{array}{l} \Delta x \\ \Delta y \end{array}\right]\\ \\ =\left[\begin{array}{ll} \Delta x & \Delta y \end{array}\right]\left\{\sum_{(u, v) \in L} w(x, y)\left[\begin{array}{cc} I_{x}^{2}(u, v) & I_{x}(u, v) I_{y}(u, v) \\ I_{x}(u, v) I_{y}(u, v) & I_{y}^{2}(u,v) \end{array}\right]\right\}\left[\begin{array}{l} \Delta x \\ \Delta y \end{array}\right]\\ \\ =\left[\begin{array}{ll} \Delta x & \Delta y \end{array}\right]\left[\begin{array}{cc} \sum_{(u, v) \epsilon L} I_{x}^{2}(u, v) & \sum_{(u, v) \epsilon L} I_{x}(u, v) I_{y}(u, v) \\ \sum_{(u, v) \in L} I_{x}(u, v) I_{y}(u, v) & \sum_{(u, v) \epsilon L} I_{y}^{2}(u, v) \end{array}\right]\left[\begin{array}{l} \Delta x \\ \Delta y \end{array}\right]\\ \\ =\left[\begin{array}{ll} \Delta x & \Delta y \end{array}\right] \cdot\left[\begin{array}{ll} A & C \\ C & B \end{array}\right] \cdot\left[\begin{array}{l} \Delta x \\ \Delta y \end{array}\right] \end{array}v) \\ \sum_{(u, v) \in L} I_{x}(u, v) I_{y}(u, v) & \sum_{(u, v) \epsilon L} I_{y}^{2}(u, v) \end{array}\right]\left[\begin{array}{l} \Delta x \\ \Delta y \end{array}\right]\\ \\ =\left[\begin{array}{ll} \Delta x & \Delta y \end{array}\right] \cdot\left[\begin{array}{ll} A & C \\ C & B \end{array}\right] \cdot\left[\begin{array}{l} \Delta x \\ \Delta y \end{array}\right] \end{array}v) \\ \sum_{(u, v) \in L} I_{x}(u, v) I_{y}(u, v) & \sum_{(u, v) \epsilon L} I_{y}^{2}(u, v) \end{array}\right]\left[\begin{array}{l} \Delta x \\ \Delta y \end{array}\right]\\ \\ =\left[\begin{array}{ll} \Delta x & \Delta y \end{array}\right] \cdot\left[\begin{array}{ll} A & C \\ C & B \end{array}\right] \cdot\left[\begin{array}{l} \Delta x \\ \Delta y \end{array}\right] \end{array}$

Determine the dependent variable:
${\color{Red} \begin{array}{l} c (x, y, \Delta x, \Delta y) =\left[\begin{array}{ll} \Delta x & \Delta y \end{array}\right] \cdot M \cdot\left[\begin {array}{l} \Delta x\\\Delta y \end{array}\right] \end{array}}$
$\sum_{x,y} w(x,y) \begin{bmatrix}I_x I_x & I_x I_y \\ I_x I_y & I_y I_y \end{bmatrix}$
ThisriI $I_x$ Sum $I_y$ are the image derivatives of the image in the x and y directions (they can be easily obtained using cv2.Sobel())

The equation of an ellipse

insert image description here
$\frac{x^{2}}{a^{2}}+\frac{y^{2}}{b^{2}}=1$
矩阵表达：
$\left[\begin{array}{ll} x & y \end{array}\right]\left[\begin{array}{ll} \frac{1}{a^{2}} & 0 \\ 0 & \frac{1}{b^{2}} \end{array}\right]\left[\begin{array}{l} x \\ y \end{array}\right]=1$
An nxn matrix can solve its eigenvalues. We solve the eigenvalues of the above-mentioned intermediate matrix, and we can get
$\left\ {\begin{array} { l } { \lambda _ { 1 } = \frac { 1 } { a ^ { 2 } } } \\ { \lambda _ { 2 } = \frac { 1 } { b ^ { 2 } } } \end{array} \quad \left\{\begin{array}{l} a=\frac{1}{\sqrt{\lambda_{1}}} \\ b=\frac{1}{ \sqrt{\lambda_{2}}} \end{array}\right.\right.$
The larger the eigenvalue, the shorter the axis radius.

M matrix

${\color{Red} \begin{array}{l} c(x, y, \Delta x, \Delta y) =\left[\begin{array}{ll} \Delta x & \Delta y \end{array}\right] \cdot M \cdot\left[\begin{array}{l} \Delta x \\ \Delta y \end{array}\right] \end{array}}$
$\sum_{x,y} w(x,y) \begin{bmatrix}I_x I_x & I_x I_y \\ I_x I_y & I_y I_y \end{bmatrix}$
We found that the autocorrelation function is actually a matrix representation of an ellipse.
Then, we can solve $The eigenvalues of the M$ matrix get $\lambda_1$ and $\lambda_2$ 。

$\lambda_1$ and $\lambda_2$ are very small, and the axis lengths are large, which means that the point is in a flat area.
$\lambda_1$ and $\lambda_2$ They are all very large and relatively close to each other, and their axis lengths are very small, which can be considered as corner points.
$\lambda_1$ and $\lambda_2$ One of them is very small, one is very large, one axis is long, and the other axis is short, which can be considered as the edge area at this time.

If we follow the above method, it is obvious that we need to calculate a lot of things for each point, and the speed is too slow, so an empirical formula is proposed.
$R=det(M)-k(trace(M))^2$
of which,

$der(M)=\lambda_1\cdot\lambda_2=AB-C^2$
$trace(M)=\lambda_1+\lambda_2=A+B$
$\lambda_1$ and $\lambda_2$ is Eigenvalues of $M$
$k$ is an empirical value, generally 0.02-0.04

2 Opencv implementation

2.1 Function cv2.cornerHarris() [just a rough understanding]

2.1.1 Parameters

img: input image, should be float32 grayscale image
blockSize: the size of the area considered by corner detection
ksize: Aperture parameter using Sobel derivative
k: Harris detection score in $R$ parameter value of $k$

# -*-coding:utf-8-*-
import numpy as np
import cv2

filename = '1.jpg'
img = cv2.imread(filename)
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

# opencv需要输入是float32类型的
gray = np.float32(gray)
# 返回值是一个跟图片同样大小的矩阵分数图
dst = cv2.cornerHarris(gray, 5, 3, 0.04)

# 通过图像自身的分数图的大小关系，得到哪些像素点应该是角点，
# 这些像素点的值都变为[0,0,255]，也就是红色。
img[dst>0.01*dst.max()]=[0,0,255]
cv2.imshow('dst', img)
cv2.waitKey(0)

insert image description here

2.2 cv2.goodFeaturesToTrack()

The problem with the above API is that what is returned is not the position coordinates of the corner points, but a score map, and we directly process the source image with the score map.
cv2.goodFeaturesToTrack(), not only supports Harris corner detection, but also supports corner detection of Shi Tomasi algorithm.

2.2.1 Parameters

gray_src: 8-bit or 32-bit single-channel grayscale image
corners: position point vector, which saves the coordinates of the detected corner points
max_corners: defined as the maximum number of corner points that can be detected;
qualityLevel: The quality level of the detected corner points, and the points whose eigenvalues are less than the maximum eigenvalue of qualityLevel will be discarded;
minDistance: the minimum distance between two corner points, in pixels;
mask: Specify the detection area, if the entire image is detected, the mask is empty
blockSize: Calculate the window size of the autocorrelation function covariance matrix;
useHarrisDetector: Whether to use Harris corner detection, if false, use Shi-Tomasi operator
k: The hyperparameter of Harris corner detection, generally 0.02 to 0.04, userHarrisDetector is false, it will not work.

# -*-coding:utf-8-*-
import numpy as np
import cv2

filename = '1.jpg'
img = cv2.imread(filename)
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)

gray = np.float32(gray)


corners = cv2.goodFeaturesToTrack(gray, maxCorners=1000, qualityLevel=0.01, minDistance=10)
corners = np.int0(corners)
for i in corners:
    x,y = i.ravel()
    cv2.circle(img, (x, y), 2, (0, 0, 255), -1)

cv2.imshow('out.jpg', img)

cv2.waitKey(0)