Detailed explanation of Opencv binocular correction function stereoRectify

The official explanation of the function

function prototype

void cv::stereoRectify 	(   InputArray  cameraMatrix1,
                            InputArray  distCoeffs1,
                            InputArray  cameraMatrix2,
                            InputArray  distCoeffs2,
                            Size        imageSize,
                            InputArray  R,
                            InputArray  T,
                            OutputArray R1,
                            OutputArray R2,
                            OutputArray P1,
                            OutputArray P2,
                            OutputArray Q,
                            int         flags = CALIB_ZERO_DISPARITY,
                            double      alpha = -1,
                            Size        newImageSize = Size(),
                            Rect *      validPixROI1 = 0,
                            Rect *      validPixROI2 = 0 
	) 	

The function of the function is to obtain the transformation matrix and projection matrix required for the rectified binocular image, and then pass it to the initUndistortRectifyMap function to generate a mapping from the rectified image to the pixel coordinates of the original image, and finally use the remap function to obtain the rectified binocular image.
The characteristics of the corrected binocular image mentioned above are: 1. The imaging plane of the binocular camera is the same plane; 2. The left and right images are parallel to the same epipolar line and the y coordinates of all points on the epipolar line are equal.

Parameter Description:

Input parameters:

cameraMatrix1: Left-eye camera intrinsic parameter matrix
distCoeffs1: Left-eye camera distortion parameter
cameraMatrix2: Right-eye camera intrinsic parameter matrix
distCoeffs2: Right-eye camera distortion parameter
imageSize: Image size:
RRotation transformation from left-eye camera coordinate system to right-eye camera coordinate system, namely $R_{rl}$
T: The translation transformation from the coordinate system of the left eye camera to the coordinate system of the right eye camera, that is, $t_{rl}$

flags: If set to CALIB_ZERO_DISPARITY, the function will set the principal points of the two cameras to be the same. Otherwise the image is translated to maximize the useful image area.
alpha: Free scaling parameter. If set to -1 or not set, the function performs default scaling. Otherwise the argument should be 0-1. 0: The rectified image will be zoomed in and translated so that only valid pixels are in the final image; 1: The image will be zoomed out and translated so that all pixels in the original image are visible.
newImageSize: Corrected image resolution. Default (0, 0), set to original image size. Setting it to a high resolution can preserve more details of the original image, especially when the distortion is large.
validPixROI1: A rectangle containing at most valid pixels. (Left eye image)
validPixROI2: A rectangle containing at most valid pixels. (right eye image)

Output parameters:

R1: Correction rotation matrix. Transform the uncorrected point in the first camera coordinate system to the first camera correction coordinate system, that is, R_{left corrected coordinate system}{left uncorrected coordinate system}: corrected rotation matrix
R2. Transform the uncorrected points in the second camera coordinate system to the second camera correction coordinate system, that is, R_{right corrected coordinate system}{right uncorrected coordinate system}: 3x4 left camera projection matrix
P1. Project the points in the left rectified coordinate system to the left rectified coordinate system image plane coordinate system.
P2: 3x4 right camera projection matrix. Project the points in the left rectified coordinate system to the right rectified coordinate system image plane coordinate system.
Q: 4x4 disparity depth map matrix.

For horizontal binocular cameras (most binocular cameras), where P1, P2, Q are defined as follows:

$$\begin{array}{rl}\text{P2}& =\left[\begin{array}{cccc}f& 0& c{x}_2& T_x\cdot f\\ 0& f& cy& 0\\ 0& 0& 1& 0\end{array}\right],\\ \text{P2}& =\left[\begin{array}{cccc}f& 0& c{x}_2& T_x\cdot f\\ 0& f& cy& 0\\ 0& 0& 1& 0\end{array}\right],\\ \text{Q}& =\left[\begin{array}{cccc}1& 0& 0& -c{x}_1\\ 0& 1& 0& -cy\\ 0& 0& 0& f\\ 0& 0& -\frac{1}{T_x}& \frac{c{x}_1-c{x}_2}{T_x}\end{array}\right]\end{array}$$

function code test

We test this function using the binocular camera parameters of the EuRoC MAV dataset:

#include <opencv2/core.hpp>
#include <opencv2/calib3d.hpp>
#include <opencv2/imgproc.hpp>
#include <opencv2/highgui.hpp>
#include <iostream>
#include <Eigen/Core>
#include <Eigen/Geometry>

using namespace cv;
using namespace std;

int main(int argc, char **argv)
{
    
    
    if(argc != 3){
    
    
        cerr << "Error: please input 2 images." << endl;
        return -1;
    }

    // Euroc 数据集双目相机内参
    Mat Kl = ( Mat_<float>(3,3) << 458.654, 0., 367.215, 
                                    0., 457.296, 248.375, 
                                    0., 0., 1.);
    Mat Kr = ( Mat_<float>(3,3) << 457.587, 0., 379.999, 
                                    0., 456.134, 255.238, 
                                    0., 0., 1.);

    Mat Dl = ( Mat_<float>(1,4) << -0.28340811, 0.07395907, 0.00019359, 1.76187114e-05 );       // 畸变参数
    Mat Dr = ( Mat_<float>(1,4) << -0.28368365,  0.07451284, -0.00010473, -3.55590700e-05 );

    // 双目相机相对位姿
    Mat R_rl = ( Mat_<double>(3,3) <<    9.99997256e-01,   2.31206719e-03,   3.76008102e-04, 
	                                    -2.31713572e-03,   9.99898049e-01,   1.40898358e-02,
	                                    -3.43393121e-04,  -1.40906685e-02,   9.99900663e-01 );
    Mat t_rl = ( Mat_<double>(3,1) << -0.11007381,  0.00039912, -0.0008537 );       // 变换的数据类型需要时double的，不然之后执行opencv的函数会报错
    
    Mat img_src_l = imread(argv[1], IMREAD_UNCHANGED);    // 读取左右目图像
    Mat img_src_r = imread(argv[2], IMREAD_UNCHANGED);

    cout << "read images finished. " << endl;

    int width = img_src_l.cols, height = img_src_l.rows;
    Mat Rl, Rr, Pl, Pr, Q;
    Mat undistmap1l, undistmap2l, undistmap1r, undistmap2r;

    stereoRectify( Kl, Dl, Kr, Dr, Size(width, height), R_rl, t_rl, Rl, Rr, Pl, Pr, Q, cv::CALIB_ZERO_DISPARITY, 0 );
    cout << "stereo rectify finished. " << endl;
    initUndistortRectifyMap( Kl, Dl, Rl, Pl, cv::Size(width,height), CV_16SC2, undistmap1l, undistmap2l );
    initUndistortRectifyMap( Kr, Dr, Rr, Pr, cv::Size(width,height), CV_16SC2, undistmap1r, undistmap2r );

    // 将 R_21 和 t_21 转换为两个校正相机坐标系的变换
    cout << "R_rl before rectification: " << endl << R_rl << endl;
    R_rl = Rr * R_rl * Rl.t();
    cout << "R_rl after rectification: " << endl << R_rl << endl;

    cout << "t_rl before rectification: " << endl << t_rl.t() << endl;
    t_rl = Rr * t_rl;
    cout << "t_rl after rectification: " << endl << t_rl.t() << endl;

    // 打印投影矩阵
    cout << "Pl: " << endl << Pl << endl;
    cout << "Pr: " << endl << Pr << endl;

    // 得到校正图像
    Mat img_rtf_l, img_rtf_r;
    remap( img_src_l, img_rtf_l, undistmap1l, undistmap2l, cv::INTER_LINEAR );
    remap( img_src_r, img_rtf_r, undistmap1r, undistmap2r, cv::INTER_LINEAR );

    Mat img_src, img_rtf;
    hconcat(img_src_l, img_src_r, img_src);
    hconcat(img_rtf_l, img_rtf_r, img_rtf);

    cvtColor(img_src, img_src, COLOR_GRAY2BGR);
    cvtColor(img_rtf, img_rtf, COLOR_GRAY2BGR);

    // 绘制平行线
    for(int i = 1, iend = 8; i < iend; i++){
    
    
        int h = height/iend * i;
        line(img_src, Point2i(0, h), Point2i(width*2, h), Scalar(0,0,255));
        line(img_rtf, Point2i(0, h), Point2i(width*2, h), Scalar(0,0,255));
    }

    imshow("image_src", img_src);
    imshow("image_rtf", img_rtf);

    waitKey(0);

    return 0;
}

The rotation matrix from the left-eye camera to the right-eye camera before and after correction:

R_rl before rectification: 
[0.999997256, 0.00231206719, 0.000376008102;
 -0.00231713572, 0.999898049, 0.0140898358;
 -0.000343393121, -0.0140906685, 0.999900663]
R_rl after rectification: 
[0.999999999522207, -2.395107227679009e-12, 6.39099232960692e-12;
 -2.39902704968578e-12, 1.000000000493077, -4.695780200869464e-11;
 6.390414450718838e-12, -4.698293904886045e-11, 1.00000000036201]

The corrected rotation matrix is ​​almost equal to the identity matrix, indicating that the imaging planes of the two cameras are parallel.

The translation vector from the left camera to the right camera before and after correction:

t_rl before rectification: 
[-0.11007381, 0.00039912, -0.0008537]
t_rl after rectification: 
[-0.1100778440394819, -1.899859817072477e-16, 5.956099065901966e-16]

After correction, the positions of the two cameras only shift in the x direction, and because the imaging planes of the two cameras are parallel, the imaging planes of the two cameras are the same plane.

Corrected projection matrix:

Pl: 
[436.2345881250716, 0, 364.4412384033203, 0;
 0, 436.2345881250716, 256.9516830444336, 0;
 0, 0, 1, 0]
Pr: 
[436.2345881250716, 0, 364.4412384033203, -48.01976295625924;
 0, 436.2345881250716, 256.9516830444336, 0;
 0, 0, 1, 0]

The 3x3 matrix on the left is the internal reference matrix.
Because the parameter flagsis set to CALIB_ZERO_DISPARITY, the principal points of the corrected two cameras are equal, and thus the internal parameter matrices are equal.

Draw a line parallel to the x-axis in the binocular image before and after correction:

After correction, the left and right images are parallel to the same epipolar line and the y coordinates of all points on the epipolar line are equal.