1. Basic concepts
Put your finger in front of your eyes and close your left and right eyes respectively. We will find that the relative position of the finger and the object behind is different , that is, there is a visual difference between the two images recognized by the two eyes. We use the "parallax" A concept to express this difference .
This process can also be simulated with two cameras in the same plane:
As shown in the figure below, there are two cameras with position offsets on the same horizontal plane , and the pixel coordinate positions on the picture after they take pictures of the same object are different :
For the same feature point P , the imaging points under the cameras O l and O r are respectively p and p' , the two upward arrow lines represent the direction of the camera's main axis of photography (parallel) , and the two red lines represent the imaging plane (along x The lengths in the direction are equal) , the left border of the photo is the zero starting side in the x direction , the x coordinates of points p and p' in the corresponding phase plane are x l and x r respectively , and the parallax (disparity) is defined as x l -xr . _
Second, the conversion of parallax and depth
Through parallax, we can calculate the depth information of point P from the camera by using similar triangles :
Record the distance of O l O r as B, the distance of pp' as b, the depth as z, the focal length as f, and the parallax x l -x r as s, then
b B = z − fz \frac{b}{B}=\ frac{zf}{z}Bb=zz−f
B − s B = z − f z \frac{B-s}{B}=\frac{z-f}{z} BB−s=zz−f
z = f B s z=f\frac{B}{s} z=fsB
Thus, the distance between the object and the camera along the optical axis, that is, the depth information, can be obtained through the parallax.
b=Bs derivation:
b is the distance of pp', that is, B minus the bases of the two small red triangles on the left and right, the base on the left is x r - L/2, and the base on the right is L/2 - x l , so b=B - (x r - L/2) - (L/2 - x l ) = B - x r + x l , i.e. b = Bs
3. The relationship between parallax and depth
At the same time, according to the formula, we can see that the farther the object is, the smaller the parallax is, and the closer the object is, the larger the parallax is. The greater the position deviation of the finger , when placed far enough away, the position deviation of the object close to the finger is basically invisible.
For the human binocular system, it is precisely because of the parallax caused by the horizontal position offset between the two eyes that the human brain can recover the depth information after receiving the two pictures, that is, we can feel the 3D scene How far away are the objects we see below .
3D movies make use of parallax. Two projectors are used to project two images with parallax (so if you don't wear 3D glasses, you will feel that the pictures are overlapped and blurred). Under the action of 3D glasses, the left and right eyes only see Corresponding to an image , this simulates the binocular vision system of the human eye, thereby generating a stereoscopic 3D scene with a sense of depth in the brain .
4. Disparity map and depth map
Each pixel on the disparity map stores the disparity value of the point under the two images ( in order to reflect the difference between the pictures under different viewing angles ), while the depth map records the corresponding real object and the camera optical center along the optical axis direction of each point The distance on ( in order to store real 3D scene information ).
The following are the source images of different viewing angles, the processed disparity map and the processed depth map:
The above steps are not complicated. To obtain the disparity map from two images, you only need to extract the feature points on the two images, and then perform feature matching on the other image to find the corresponding points , and then you can calculate The disparity of the corresponding point can be obtained . After obtaining the disparity map, it can be further converted into a depth map. After having the depth map, 3D reconstruction can be performed to restore the real 3D scene (point cloud and other representation methods), so:
- The disparity map can be regarded as the output carrier of stereo matching (feature extraction, feature matching)
- Depth map is an intermediate bridge between stereo matching and point cloud generation
Reference:
https://www.fpga-china.com/4559.html
https://zhuanlan.zhihu.com/p/522285892
https://www.cnblogs.com/riddick/p/8486223.html