Zynq FPGA binocular vision graduation design (6) master thesis study summary

Brief introduction

The graduation project was almost done for epipolar correction, and then I started to summarize the thesis. This is because the master’s thesis did not discuss this part in detail. It was basically completed by reading other blogs. Of course, I also read a few articles on image distortion correction, but because of the big guys The summary is so good, I don't need to spend time summarizing it. Since the zynq board was accidentally broken, it is impossible to continue to transplant the algorithm after the sale, so I continue to study the stereo matching algorithm for binocular images. Today I will summarize my master's thesis.

text

1) The basic structure of the stereo matching algorithm

Note : The input here is the image after epipolar correction.
Matching cost calculation : measure the correlation between the pixel to be matched and the candidate pixel. Whether two pixels are points with the same name or not, the matching cost can be calculated through the matching cost function. The smaller the cost, the greater the correlation and the greater the probability of being a point with the same name.

• The cost calculation functions include : the sum of absolute value of the cost gray difference (SAD), the sum of squares of the gray difference (SSD), the normalized mutual (NCC), and the Census transformation;
• The function selected in the thesis : non-parametric image transformation (Census).

Cost aggregation : The cost information of a single pixel cannot accurately calculate the disparity value, so the purpose of cost aggregation is to minimize the uncertainty of matching and reduce the mismatch rate. The basic principle is to match pixels near the target pixel (forming a window).

• Several types of aggregation windows : Multiple Windows, Adaptive Windows, and
  Adaptive Support Windows, etc.

Parallax selection : Use the survival of the fittest (WTA) strategy to obtain the optimal parallax value.

Parallax optimization : further optimize the disparity map obtained in the previous step to improve the quality of the disparity map, including the steps of removing false disparity, proper smoothing, and sub-pixel accuracy optimization. Generally, the Left-Right Check algorithm is used to eliminate the reason Error parallax caused by occlusion and noise; small connected area removal algorithm is used to remove isolated abnormal points; Median Filter, Bilateral Filter and other smoothing algorithms are used to smooth the disparity map

2) Hardware implementation of semi-global stereo matching algorithm

The algorithm mainly includes six parts: similarity operator selection, cost calculation, cost aggregation, disparity selection, left and right check, and median filtering.

(1) Census transformation

The basic principle is as shown in the figure below. Compare the pixel value in the center with the pixel values ​​around it. If it is greater than 0, if it is less than 1, get a vector. Then the vectors of the left and right images are XORed to obtain the Hamming distance. Obtaining the least number of 1 in the Hamming distance indicates that the two pixels are more similar, and they are a pair of pixels

(2) Path aggregation

The semi-global matching algorithm in 4 directions is adopted, and the minimum value is obtained in each direction, and finally the different cost values ​​in the four directions are averaged.

(3) Left and right consistency check

At the same time, obtain the two disparity maps obtained by taking the left as the reference image and the right view as the reference image for verification. This will waste a lot of resources. My ZYNQ 7020 estimates that the resources are not enough and reduce the difficulty, so I discarded this Step, only use the right as a reference image to generate a disparity map.

(4) Median filtering

In order to filter out stripes and noise, median filtering is used to eliminate abnormal disparity values.

(5) His final effect

to sum up

Researching this paper, this paper is the best written among many. I understand many places, but I am still a little confused about path aggregation. I will look at other materials later and modify and supplement. The next step is that the board has not arrived yet, so only use matlab to do binocular stereo matching algorithm simulation.
Master's thesis link: Design and Implementation of Real-time Binocular Stereo Vision System Based on FPGA Shang Hai Harbin Institute of Technology

Welcome to follow my public account: Core Kingdom, there are more FPGA & digital IC technology sharing, and you can also get open source FPGA projects!