A brief description of the SGM algorithm for stereo matching
1 Overview
After reading Li Bo 's series of articles on the theoretical part of the stereo matching SGM algorithm, I feel that I have benefited a lot. In order to sort out the ideas in detail, I will briefly record it.
2. Algorithm steps
2.1 Matching cost calculation
- Computes the binary bit string for each pixel using the Census transform . The main idea is to design a 5*5 or larger window centered on a certain pixel, and obtain the binary bit string of each pixel by comparing the size of the pixel values. It feels a bit like the FASt corner judgment method in the ORB feature point, and the formation form of the BRIEF descriptor.
- Before using the stereo matching algorithm, it is necessary to use stereo correction to correct the left and right images, so that the epipolar lines are in the horizontal position, which is convenient for parallax search.
- After stereo correction, the disparity of the left and right images can be expressed as d=xl-xr. Since the points seen on the left are generally on the right, and the corresponding points seen on the right are generally on the left, d>0, otherwise the common view range is too small, which is not conducive to the solution of the disparity map.
- Assume that the parallax range D of a certain point in the left picture is between the minimum parallax (0) and the maximum parallax (64). For each disparity value, through the binary bit strings of the corresponding points in the left and right images, the Hamming distance (that is, the number of different bits in the bit strings) is calculated as the matching loss of the disparity value.
- After all calculations are completed, a loss cost matrix of image width W × image height H × disparity range D is obtained, which is used for subsequent cost aggregation. Here we borrow a picture from the original text to show the calculation of the Hamming distance:
2.2 Cost Aggregation
- In an image, the disparity of a point is not only related to its corresponding matching point, but also related to its surrounding pixels. Therefore, in order to ensure the correctness and continuity of the disparity value, the disparity is optimized by means of cost aggregation according to the previous cost value .
- The aggregation cost of a certain pixel is mainly calculated by the sum of the costs in all directions. Generally, 4-field aggregation can be used to achieve a better parallax optimization effect, including four directions of up, down, left, and right.
- For the cost value of each direction, it can be calculated by the following formula (taken from the cost aggregation
): in the formula, p represents the pixel, r represents the path, and in the case of left and right paths, p−r is the left side of p (from left to Right aggregation) or adjacent pixels on the right side (right to left aggregation), their row numbers are equal, and the column numbers differ by 1. L is the aggregate cost value, and C is the initial cost value.
2.3 Disparity Calculation and Optimization
- The matching cost value of each disparity value is calculated for each pixel when the disparity is between the minimum disparity (0) and the maximum disparity (64), then the specific disparity value is the disparity value corresponding to the minimum cost value.
- After the disparity calculation is completed, it is necessary to optimize the disparity , including:
(1) left-right consistency check : that is, through the disparity map of the left image, the disparity map of the right image is obtained. For the matching point in the disparity map, if it is in the left image The disparity value of and the disparity value in the right picture are less than 1 pixel, then the disparity calculation is correct, otherwise delete the disparity value corresponding to the changed point.
(2) Uniqueness detection : It refers to calculating the value of the minimum cost and the second minimum cost for each pixel. If the relative difference between the two is less than a certain threshold, it will be eliminated.
(3) Elimination of small connected areas : refers to the elimination of small connected areas in the disparity map, and the difference between the disparity in the same connected area and the disparity in the neighborhood is less than the set threshold (generally 1).
(4) Improving parallax precision : using quadratic interpolation, quadratic curve fitting is performed on the cost value of the optimal parallax and the cost values of the two front and back parallax, and the parallax value corresponding to the extreme point of the curve is the new The subpixel disparity value of .
(5) Median filtering : filter the disparity map.
3 summary
Refer to Li Bo's three-dimensional matching series blog:
[Hengdao Lima] [Theory Hengdao] [Stereoscopic Matching Series] Classic SGM: (2) Census Transformation of Matching Cost Calculation
[Hengdao Lima] [Theory Hengdao] [Stereoscopic Matching] Series] Classic SGM: (3) Cost Aggregation
[Hengdao Lima] [Theory Hengdao] [Stereo Matching Series] Classic SGM: (4) Parallax Calculation, Parallax Optimization