In computer vision, the ability to extract and match features from images is critical for tasks such as object recognition, image stitching, and camera localization. A popular approach to achieve this is
ORB (Oriented FAST and Rotated Brief) feature detectors and descriptors. ORB, developed by Ethan Rublee
et al., combines the strengths of two existing technologies—FAST (Accelerated Segmentation Testing Features) and Brief (Binary Robust Independent Basic Features)—to provide a powerful and efficient method for feature extraction and matching. solution.
In this article, we delve into the inner workings of ORB, discussing how it acts as a feature detector and descriptor, its key components, and its advantages over other feature extraction methods.
- F AST feature detector :
The Feature from Accelerated Segment Test (
FAST) algorithm is implemented by analyzing a circular neighborhood of 16 pixels. The FAST algorithm marks each pixel in the neighborhood as brighter or darker than a certain threshold, which is defined relative to the center of the circle. A neighborhood is considered a corner if it contains a series of consecutive pixels marked as brighter or darker.
FAST is a corner detection algorithm that can effectively identify corners or key points in an image. It is designed to be faster than traditional corner detectors by utilizing a simple intensity thresholding scheme and minimization computation. FAST examines a circle of pixels around a candidate pixel and classifies it as a corner if there are a sufficient number of contiguous pixels that are either brighter or darker than the candidate pixel.
- BRIEF
In addition, Binary Robust Independent Elementary Feature (
BRIEF) is not a feature detection algorithm, but a descriptor. Let's delve a little deeper into the concept of descriptors, and then come to BRIEF.
When analyzing images with SIFT and SURF, the core of the whole process is to call
the detectAndCompute function. This function performs two distinct steps - detection and computation, which return 2 distinct results (coupled into a tuple).
The detection result is a set of keypoints, and the computation result is a set of descriptors for those keypoints. This means that OpenCV's cv2.SIFT and cv2.SURF classes both implement detection and description algorithms. Remember, the original SIFT and SURF are not feature detection algorithms.
OpenCV's cv2.SIFT implements DoG feature detection and SIFT description, while OpenCV's cv2.SURF implements fast Hessian feature detection and SURF description.
A keypoint descriptor is a representation of an image and acts as a channel for feature matching, because you
can compare the keypoint descriptors of two images and find their commonalities. BRIEF is currently one of the fastest descriptors. The theory behind BRIEF is rather complex, but it can be said that BRIEF employs a series of optimizations that make it a very good choice for feature matching.
brute force matching
A brute force matcher is a descriptor matcher that compares two sets of keypoint descriptors and generates a list of matches. It is called brute force matching because there is very little optimization involved in this algorithm. For each keypoint descriptor in the first set, the matcher compares it with every keypoint descriptor in the second set. Each comparison produces a distance value, and the best match is chosen based on the smallest distance.
Broadly speaking, in computing, the term "brute force" refers to a method of prioritizing an exhaustive enumeration of all possible combinations (eg, all possible combinations of characters to break a password of known length). Conversely, algorithms that prioritize speed may skip possibilities and try to take shortcuts to find the most plausible solution.
OpenCV provides a cv2.BFMatcher class that supports several methods of brute force feature matching.
- Principle of ORB Algorithm
The ORB algorithm combines the FAST feature point detection method with the BRIEF feature descriptor, and improves and optimizes them on the basis of them.
First, it uses the FAST feature point detection method to detect feature points, and then uses the Harris corner point measurement method to select N feature points with the largest Harris corner point response value from the FAST feature points. The response function of the Harris corner is defined as:
R=detM−k(trace(M))2
In today's life, when we observe an object from different distances, different directions, angles, and different lighting conditions, the size, shape, and brightness of the object will be different. But we can still tell it's an object. An ideal feature descriptor should have these properties, that is, in images with different sizes, directions, and shades, the same feature point should have sufficiently similar descriptors, which is called the reproducibility of the descriptor.
However, ORB does not solve the problem of scale inconsistency. In OpenCV's ORB implementation, image pyramids are used to improve performance in this area. We build Gaussian pyramids and then detect corner points on each layer of pyramid images to achieve scale invariance. . ORB mainly solves the problem that the BRIEF descriptor does not have rotation invariance. The ORB paper proposes a grayscale centroid method to solve this problem. The grayscale centroid method assumes that there is an offset between the grayscale of the corner point and the centroid , this vector can be used to represent a direction. For any feature point p, we define the moment of the neighboring pixels of p as:
Where I(x,y) is the gray value at the point (x,y), q is the centroid, i,j=0,1. Then we can get the centroid of the image as:
Then the angle between the feature point and the centroid is defined as the direction of the FAST feature point:
In order to improve the rotation invariance of the algorithm, it is necessary to ensure that x and y are in a circular area with radius r, that is, x,y∈[−r,r], where r is equal to the radius of the neighborhood.
Description of feature points
ORB chose BRIEF as the feature description method, but we know that BRIEF does not have rotation invariance, so we need to add rotation invariance to BRIEF, and call this method "Steer BRIEF". For any feature point, its BRIEF descriptor is a binary code string with a length of n. This binary code string is generated by n point pairs in the neighborhood of the feature point. We now talk about these 2n points ( xi,yi), i=1,2,...,2n form a matrix S:
Calonder proposed to compute the BRIEF descriptor separately for the set of rotations and projections of each block, but it is expensive. A more efficient method is adopted in ORB: using the neighborhood direction θ and the corresponding rotation matrix Rθ, a corrected version Sθ of S is constructed:
in:
And θ is the main direction we obtained for the feature point.
That is, we rotate the coordinate axis ]theta, and calculate the matching point pairs with the main direction as the coordinate system, as shown in the following figure:
In fact, we can discretize the angle, that is, divide 360 degrees into 12 parts, each of which is 30 degrees, and then we obtain a Sθ for each of the 12 angles, so that we create a lookup table, for each For a θ, we only need to look up the table to quickly get the set Sθ of its points.
Solve the differentiation of descriptors
One of the surprising properties of BRIEF is: for each feature bit of an n-dimensional binary string, the values of all feature points at that bit satisfy a Gaussian distribution with a mean value close to 0.5 and a large variance. The larger the variance, the stronger the discrimination , and the greater the difference between the descriptors of different feature points, it is not easy to mis-match for matching. But when we adjust BRIEF to Steered BRIEF along the direction of feature points, the mean shifts to a more dispersed pattern. It can be understood that the directional corner key points show a more balanced performance for the binary string. Moreover, it is mentioned in the paper that through PCA analysis of each eigenvector, it is known that the variance of Steered BRIEF is small, the discriminability is small, and the correlation between each component is relatively large.
In order to reduce the loss of Steered BRIEF variance and reduce the correlation between binary code strings, ORB uses a learning method to select a smaller set of point pairs. Methods as below:
Firstly, a test set of about 300k keypoints is established, which come from images in the PASCAL2006 set.
For each feature point in these 300k key points, consider its 31×31 neighborhood, we will find some point pairs in this neighborhood, which is different from the Brief that first smooths the points in this Patch, and then A method for selecting point pairs using a Gaussian distribution with the patch center as the origin.
In order to remove the interference of some noise points, ORB selects the average gray level of a 5×5 area to replace the gray level of a single point. Here, the average gray level of the image in the 5×5 area can be calculated using the integral image method. We know that there are N=(31−5+1)×(31−5+1) such windows in the 31×31 Patch, so if we want to select 2 sub-windows from the N sub-windows, there are C2N methods in total. So for each feature point in 300k, we can extract a very long binary string from its 31×31 neighborhood, the length of which is M=C2N, expressed as:
Then when all the 300k key points are extracted from the above features, we get a 300k×M matrix, and the value of each element in the matrix is 0 or 1.
Calculate the mean of each column vector of the matrix, that is, the test results of each point pair on 300k feature points. Reorder all column vectors by mean. After being arranged, a vector T is formed, and each element of T is a column vector. Carry out a greedy search, put the first column from T into R, and there is no test result for this point pair in T, and then compare the next column in T with all elements in R, Calculate their correlation, if the correlation exceeds a pre-set threshold, throw it away, otherwise put it into R.
Repeat the steps above until you have 256 column vector positions in R.
If 256 are not found after searching all T, then we can increase the relevant threshold and try again. In this way, we get 256 point pairs. The above process we call it rBRIEF.
OpenCV implementation
There are many parameters that can be set in ORB, and in OpenCV it can create an ORB detector through ORB.
cv2.ORB_create([,nfeatues[,scaleFactor[,nlevels[,edgeThreshold[,firstLevel[,WTA_K[,[scoreType,[patchSize,fastThreshold]]]]]]]]])
Parameter Description:
- nfeatures: the maximum number of feature points extracted;
- scaleFactor: The scale parameter between pyramid images, similar to k in SIFT;
- nlevels: the number of layers of the Gaussian pyramid;
- edgeThreshold : Edge threshold, this value is mainly determined according to the patchSize later, the pixels close to the edge edgeThreshold are not detected feature points.
- firstLevel-: As we all know after seeing SIFT, we can specify the index value of the first layer, which defaults to 0 here.
- WET_K : The number of point pairs used to generate BIREF descriptors is generally 2, and can also be set to 3 or 4. Then the distance between descriptors cannot be calculated using the Hamming distance, but A variant should be used. In OpenCV, if WET_K = 2 is set, then only 2 points are selected for the point pair, and NORM_HAMMING is selected for the distance parameter when matching. If WET_K is set to 3 or 4, the BIREF descriptor will select 3 or 4 points, then the following The distance parameter that should be selected when matching is NORM_HAMMING2.
- scoreType : The algorithm used to sort the feature points, you can choose HARRIS_SCORE or FAST_SCORE, but it is only a little faster than the former.
- patchSize : The feature point neighborhood size used to calculate the BIREF descriptor.
Example: Brute-Force matching using ORB descriptors:
Sample code:
import cv2
def orb_test():
# 加载图片 灰色
img1 = cv2.imread('images\\quexiao\\2.png')
gray1 = cv2.cvtColor(img1, cv2.COLOR_BGR2GRAY)
img2 = cv2.imread('images\\quexiao\\2-1.png')
gray2 = cv2.cvtColor(img2, cv2.COLOR_BGR2GRAY)
image1 = gray1.copy()
image2 = gray2.copy()
'''
1.使用ORB算法检测特征点、描述符
'''
orb = cv2.ORB_create(128)
keypoints1, descriptors1 = orb.detectAndCompute(image1, None)
keypoints2, descriptors2 = orb.detectAndCompute(image2, None)
# 在图像上绘制关键点
image1 = cv2.drawKeypoints(image=image1, keypoints=keypoints1, outImage=image1, color=(255, 0, 255),
flags=cv2.DRAW_MATCHES_FLAGS_DRAW_RICH_KEYPOINTS)
image2 = cv2.drawKeypoints(image=image2, keypoints=keypoints2, outImage=image2, color=(255, 0, 255),
flags=cv2.DRAW_MATCHES_FLAGS_DRAW_RICH_KEYPOINTS)
# 显示图像
cv2.imshow('orb_keypoints1', image1)
cv2.imshow('orb_keypoints2', image2)
cv2.waitKey(20)
'''
2、匹配
'''
# 使用汉明距离,创建BF匹配器,并进行匹配() 新版本不支持
# matcher = cv2.BFMatcher_create(cv2.HAMMING_NORM_TYPE, crossCheck=True)
matcher = cv2.BFMatcher_create(cv2.NORM_HAMMING, crossCheck=True)
matchePoints = matcher.match(descriptors1, descriptors2)
print(type(matchePoints), len(matchePoints), matchePoints[0])
# 按照距离从小到大排序，选取最优匹配的
sorted(matchePoints, key=lambda x: x.distance)
# 绘制最优匹配点
outImg = None
outImg = cv2.drawMatches(img1, keypoints1, img2, keypoints2, matchePoints[:10], outImg, matchColor=(0, 255, 0),
flags=cv2.DRAW_MATCHES_FLAGS_DEFAULT)
cv2.imshow('matche', outImg)
cv2.waitKey(0)
cv2.destroyAllWindows()
cv2.waitKey(0)
cv2.destroyAllWindows()
if __name__ == '__main__':
orb_test()
The original image of the experiment:
2.png
2-1.png
