Cameras have been around for a long, long time. However, with the advent of cheap pinhole cameras in the late 20th century, pinhole cameras became commonplace in our everyday lives. Unfortunately, this cheapness comes at a price: serious distortions. Fortunately, these are constants, and with scaling and some remapping, we can correct this. In addition, through calibration, the relationship between the camera's natural units (pixels) and real-world units (such as millimeters) can also be determined.

1. Theory

For distortion, OpenCV takes radial and tangential distortion into account.

For 径向畸变, use the following formula:
$x_{ distorted}=x(1+k_1r^2+k_2r^4+k_3r^6) \\ y_{distorted}=y(1+k_1r^2+k_2r^4+k_3r^6)$
Therefore, for an undistorted pixel point (x,y), its position on the distorted image will be ( $x_{distorted}, x_{distorted}$ ). The presence of radial distortion manifests itself as “桶”或“鱼眼”an effect.

切向畸变Occurs because the lens is not perfectly parallel to the imaging plane. It can be represented by the formula:
$x_{distorted} =x + [2p_1xy + p_2(r^2+2x^2) \\ y_{distorted}=y + [2p_2xy + p_2(r^2+2y^2)$

So we have 5 distortion parameters, represented in OpenCV as a row matrix with 5 columns:distortion_coefficients=(k1 k2 p1 p2 k3)

Now, for unit conversion we use the following formula:
$\left[ \begin{matrix} x \\ y \\ w \end{matrix} \right ] = \left[ \begin{matrix} f_x & 0 & c_x\\ 0 & f_y & c_y \\ 0 & 0 & 1\end{matrix} \right]\left[ \begin{matrix} X \\ Y \ \Z \end{matrix} \right]$
here $The existence of w$ is based on the homography coordinate system ( $w = Z$ ) to explain. Unknown parameter is $f_x$ and $f_y$ (camera focal length) and $c_x,c_y)$ , which are the optical centers expressed in pixel coordinates. If both axes have a common focal length and a given aspect ratio (usually 1), then $f_y=f_x∗a$ , in the above formula we will have a single focal length $f$ . The matrix containing these four parameters is called the camera matrix. Although the distortion coefficients are the same regardless of the camera resolution used, these coefficients should be scaled with the current resolution of the target resolution.

The process of determining these two matrices is calibration. The calculation of these parameters is done through basic geometric equations. The equation used depends on the chosen calibration object. Currently OpenCV supports three ( more than three ) types of objects for calibration:

classic black and white chessboard
symmetrical circular pattern
asymmetric circular pattern

Basically, you need to take a snapshot of these calibration patterns with your camera and let OpenCV find them. Each discovered pattern results in a new equation. To solve this equation, at least a predetermined number of model snapshots are required to form a well-posed system of equations. This number is higher in a checkerboard pattern and lower in a circular pattern. For example, theoretically a checkerboard pattern requires at least two snapshots. However, in practice, there is a lot of noise in our input images, so to get good results you probably need at least 10 good snapshots of the input pattern.

2. Goal

The sample application will:

Determining the Distortion Matrix
Determine the camera matrix
Take input from list of camera, video and image files
Read configuration from XML/YAML file
Save the result to an XML/YAML file
Calculate reprojection error

3. Source code

You can also find the source code in the samples/cpp/tutorial_code/calib3d/camera_calibration/ folder of the OpenCV source code repository, or download it from here . For program use, run it with the -h parameter. A program has one basic parameter: the name of its configuration file. If not given, then it will attempt to open a file named "default.xml". Below is an example configuration file in XML format . In the configuration file you can choose to use a camera, a video file or a list of images as input. If you choose the last one, you need to create an enumerated configuration file as shown in the documentation . Something to remember is that you need to specify the image address using an absolute path or a relative path in the application working directory. You can find all of them in the samples directory mentioned above.

Settings are read from the configuration file when the application starts. Although, this is an important part, it has nothing to do with the topic of this tutorial: camera calibration. Therefore, I choose not to post the code for that part here. Technical background on how to do this can be found in the File Input and Output Using XML and YAML Files tutorial .

#include <iostream>
#include <sstream>
#include <string>
#include <ctime>
#include <cstdio>

#include <opencv2/core.hpp>
#include <opencv2/core/utility.hpp>
#include <opencv2/imgproc.hpp>
#include <opencv2/calib3d.hpp>
#include <opencv2/imgcodecs.hpp>
#include <opencv2/videoio.hpp>
#include <opencv2/highgui.hpp>

using namespace cv;
using namespace std;

class Settings
{
    
    
public:
    Settings() : goodInput(false) {
    
    }
    enum Pattern {
    
     NOT_EXISTING, CHESSBOARD, CIRCLES_GRID, ASYMMETRIC_CIRCLES_GRID };
    enum InputType {
    
     INVALID, CAMERA, VIDEO_FILE, IMAGE_LIST };

    void write(FileStorage& fs) const                        //Write serialization for this class
    {
    
    
        fs << "{"
                  << "BoardSize_Width"  << boardSize.width
                  << "BoardSize_Height" << boardSize.height
                  << "Square_Size"         << squareSize
                  << "Calibrate_Pattern" << patternToUse
                  << "Calibrate_NrOfFrameToUse" << nrFrames
                  << "Calibrate_FixAspectRatio" << aspectRatio
                  << "Calibrate_AssumeZeroTangentialDistortion" << calibZeroTangentDist
                  << "Calibrate_FixPrincipalPointAtTheCenter" << calibFixPrincipalPoint

                  << "Write_DetectedFeaturePoints" << writePoints
                  << "Write_extrinsicParameters"   << writeExtrinsics
                  << "Write_gridPoints" << writeGrid
                  << "Write_outputFileName"  << outputFileName

                  << "Show_UndistortedImage" << showUndistorted

                  << "Input_FlipAroundHorizontalAxis" << flipVertical
                  << "Input_Delay" << delay
                  << "Input" << input
           << "}";
    }
    void read(const FileNode& node)                          //Read serialization for this class
    {
    
    
        node["BoardSize_Width" ] >> boardSize.width;
        node["BoardSize_Height"] >> boardSize.height;
        node["Calibrate_Pattern"] >> patternToUse;
        node["Square_Size"]  >> squareSize;
        node["Calibrate_NrOfFrameToUse"] >> nrFrames;
        node["Calibrate_FixAspectRatio"] >> aspectRatio;
        node["Write_DetectedFeaturePoints"] >> writePoints;
        node["Write_extrinsicParameters"] >> writeExtrinsics;
        node["Write_gridPoints"] >> writeGrid;
        node["Write_outputFileName"] >> outputFileName;
        node["Calibrate_AssumeZeroTangentialDistortion"] >> calibZeroTangentDist;
        node["Calibrate_FixPrincipalPointAtTheCenter"] >> calibFixPrincipalPoint;
        node["Calibrate_UseFisheyeModel"] >> useFisheye;
        node["Input_FlipAroundHorizontalAxis"] >> flipVertical;
        node["Show_UndistortedImage"] >> showUndistorted;
        node["Input"] >> input;
        node["Input_Delay"] >> delay;
        node["Fix_K1"] >> fixK1;
        node["Fix_K2"] >> fixK2;
        node["Fix_K3"] >> fixK3;
        node["Fix_K4"] >> fixK4;
        node["Fix_K5"] >> fixK5;

        validate();
    }
    void validate()
    {
    
    
        goodInput = true;
        if (boardSize.width <= 0 || boardSize.height <= 0)
        {
    
    
            cerr << "Invalid Board size: " << boardSize.width << " " << boardSize.height << endl;
            goodInput = false;
        }
        if (squareSize <= 10e-6)
        {
    
    
            cerr << "Invalid square size " << squareSize << endl;
            goodInput = false;
        }
        if (nrFrames <= 0)
        {
    
    
            cerr << "Invalid number of frames " << nrFrames << endl;
            goodInput = false;
        }

        if (input.empty())      // Check for valid input
                inputType = INVALID;
        else
        {
    
    
            if (input[0] >= '0' && input[0] <= '9')
            {
    
    
                stringstream ss(input);
                ss >> cameraID;
                inputType = CAMERA;
            }
            else
            {
    
    
                if (isListOfImages(input) && readStringList(input, imageList))
                {
    
    
                    inputType = IMAGE_LIST;
                    nrFrames = (nrFrames < (int)imageList.size()) ? nrFrames : (int)imageList.size();
                }
                else
                    inputType = VIDEO_FILE;
            }
            if (inputType == CAMERA)
                inputCapture.open(cameraID);
            if (inputType == VIDEO_FILE)
                inputCapture.open(input);
            if (inputType != IMAGE_LIST && !inputCapture.isOpened())
                    inputType = INVALID;
        }
        if (inputType == INVALID)
        {
    
    
            cerr << " Input does not exist: " << input;
            goodInput = false;
        }

        flag = 0;
        if(calibFixPrincipalPoint) flag |= CALIB_FIX_PRINCIPAL_POINT;
        if(calibZeroTangentDist)   flag |= CALIB_ZERO_TANGENT_DIST;
        if(aspectRatio)            flag |= CALIB_FIX_ASPECT_RATIO;
        if(fixK1)                  flag |= CALIB_FIX_K1;
        if(fixK2)                  flag |= CALIB_FIX_K2;
        if(fixK3)                  flag |= CALIB_FIX_K3;
        if(fixK4)                  flag |= CALIB_FIX_K4;
        if(fixK5)                  flag |= CALIB_FIX_K5;

        if (useFisheye) {
    
    
            // the fisheye model has its own enum, so overwrite the flags
            flag = fisheye::CALIB_FIX_SKEW | fisheye::CALIB_RECOMPUTE_EXTRINSIC;
            if(fixK1)                   flag |= fisheye::CALIB_FIX_K1;
            if(fixK2)                   flag |= fisheye::CALIB_FIX_K2;
            if(fixK3)                   flag |= fisheye::CALIB_FIX_K3;
            if(fixK4)                   flag |= fisheye::CALIB_FIX_K4;
            if (calibFixPrincipalPoint) flag |= fisheye::CALIB_FIX_PRINCIPAL_POINT;
        }

        calibrationPattern = NOT_EXISTING;
        if (!patternToUse.compare("CHESSBOARD")) calibrationPattern = CHESSBOARD;
        if (!patternToUse.compare("CIRCLES_GRID")) calibrationPattern = CIRCLES_GRID;
        if (!patternToUse.compare("ASYMMETRIC_CIRCLES_GRID")) calibrationPattern = ASYMMETRIC_CIRCLES_GRID;
        if (calibrationPattern == NOT_EXISTING)
        {
    
    
            cerr << " Camera calibration mode does not exist: " << patternToUse << endl;
            goodInput = false;
        }
        atImageList = 0;

    }
    Mat nextImage()
    {
    
    
        Mat result;
        if( inputCapture.isOpened() )
        {
    
    
            Mat view0;
            inputCapture >> view0;
            view0.copyTo(result);
        }
        else if( atImageList < imageList.size() )
            result = imread(imageList[atImageList++], IMREAD_COLOR);

        return result;
    }

    static bool readStringList( const string& filename, vector<string>& l )
    {
    
    
        l.clear();
        FileStorage fs(filename, FileStorage::READ);
        if( !fs.isOpened() )
            return false;
        FileNode n = fs.getFirstTopLevelNode();
        if( n.type() != FileNode::SEQ )
            return false;
        FileNodeIterator it = n.begin(), it_end = n.end();
        for( ; it != it_end; ++it )
            l.push_back((string)*it);
        return true;
    }

    static bool isListOfImages( const string& filename)
    {
    
    
        string s(filename);
        // Look for file extension
        if( s.find(".xml") == string::npos && s.find(".yaml") == string::npos && s.find(".yml") == string::npos )
            return false;
        else
            return true;
    }
public:
    Size boardSize;              // The size of the board -> Number of items by width and height
    Pattern calibrationPattern;  // One of the Chessboard, circles, or asymmetric circle pattern
    float squareSize;            // The size of a square in your defined unit (point, millimeter,etc).
    int nrFrames;                // The number of frames to use from the input for calibration
    float aspectRatio;           // The aspect ratio
    int delay;                   // In case of a video input
    bool writePoints;            // Write detected feature points
    bool writeExtrinsics;        // Write extrinsic parameters
    bool writeGrid;              // Write refined 3D target grid points
    bool calibZeroTangentDist;   // Assume zero tangential distortion
    bool calibFixPrincipalPoint; // Fix the principal point at the center
    bool flipVertical;           // Flip the captured images around the horizontal axis
    string outputFileName;       // The name of the file where to write
    bool showUndistorted;        // Show undistorted images after calibration
    string input;                // The input ->
    bool useFisheye;             // use fisheye camera model for calibration
    bool fixK1;                  // fix K1 distortion coefficient
    bool fixK2;                  // fix K2 distortion coefficient
    bool fixK3;                  // fix K3 distortion coefficient
    bool fixK4;                  // fix K4 distortion coefficient
    bool fixK5;                  // fix K5 distortion coefficient

    int cameraID;
    vector<string> imageList;
    size_t atImageList;
    VideoCapture inputCapture;
    InputType inputType;
    bool goodInput;
    int flag;

private:
    string patternToUse;


};

static inline void read(const FileNode& node, Settings& x, const Settings& default_value = Settings())
{
    
    
    if(node.empty())
        x = default_value;
    else
        x.read(node);
}

enum {
    
     DETECTION = 0, CAPTURING = 1, CALIBRATED = 2 };

bool runCalibrationAndSave(Settings& s, Size imageSize, Mat&  cameraMatrix, Mat& distCoeffs,
                           vector<vector<Point2f> > imagePoints, float grid_width, bool release_object);

int main(int argc, char* argv[])
{
    
    
    const String keys
        = "{help h usage ? |           | print this message            }"
          "{@settings      |default.xml| input setting file            }"
          "{d              |           | actual distance between top-left and top-right corners of "
          "the calibration grid }"
          "{winSize        | 11        | Half of search window for cornerSubPix }";
    CommandLineParser parser(argc, argv, keys);
    parser.about("This is a camera calibration sample.\n"
                 "Usage: camera_calibration [configuration_file -- default ./default.xml]\n"
                 "Near the sample file you'll find the configuration file, which has detailed help of "
                 "how to edit it. It may be any OpenCV supported file format XML/YAML.");
    if (!parser.check()) {
    
    
        parser.printErrors();
        return 0;
    }

    if (parser.has("help")) {
    
    
        parser.printMessage();
        return 0;
    }

    //! [file_read]
    Settings s;
    const string inputSettingsFile = parser.get<string>(0);
    FileStorage fs(inputSettingsFile, FileStorage::READ); // Read the settings
    if (!fs.isOpened())
    {
    
    
        cout << "Could not open the configuration file: \"" << inputSettingsFile << "\"" << endl;
        parser.printMessage();
        return -1;
    }
    fs["Settings"] >> s;
    fs.release();                                         // close Settings file
    //! [file_read]

    //FileStorage fout("settings.yml", FileStorage::WRITE); // write config as YAML
    //fout << "Settings" << s;

    if (!s.goodInput)
    {
    
    
        cout << "Invalid input detected. Application stopping. " << endl;
        return -1;
    }

    int winSize = parser.get<int>("winSize");

    float grid_width = s.squareSize * (s.boardSize.width - 1);
    bool release_object = false;
    if (parser.has("d")) {
    
    
        grid_width = parser.get<float>("d");
        release_object = true;
    }

    vector<vector<Point2f> > imagePoints;
    Mat cameraMatrix, distCoeffs;
    Size imageSize;
    int mode = s.inputType == Settings::IMAGE_LIST ? CAPTURING : DETECTION;
    clock_t prevTimestamp = 0;
    const Scalar RED(0,0,255), GREEN(0,255,0);
    const char ESC_KEY = 27;

    //! [get_input]
    for(;;)
    {
    
    
        Mat view;
        bool blinkOutput = false;

        view = s.nextImage();

        //-----  If no more image, or got enough, then stop calibration and show result -------------
        if( mode == CAPTURING && imagePoints.size() >= (size_t)s.nrFrames )
        {
    
    
          if(runCalibrationAndSave(s, imageSize,  cameraMatrix, distCoeffs, imagePoints, grid_width,
                                   release_object))
              mode = CALIBRATED;
          else
              mode = DETECTION;
        }
        if(view.empty())          // If there are no more images stop the loop
        {
    
    
            // if calibration threshold was not reached yet, calibrate now
            if( mode != CALIBRATED && !imagePoints.empty() )
                runCalibrationAndSave(s, imageSize,  cameraMatrix, distCoeffs, imagePoints, grid_width,
                                      release_object);
            break;
        }
        //! [get_input]

        imageSize = view.size();  // Format input image.
        if( s.flipVertical )    flip( view, view, 0 );

        //! [find_pattern]
        vector<Point2f> pointBuf;

        bool found;

        int chessBoardFlags = CALIB_CB_ADAPTIVE_THRESH | CALIB_CB_NORMALIZE_IMAGE;

        if(!s.useFisheye) {
    
    
            // fast check erroneously fails with high distortions like fisheye
            chessBoardFlags |= CALIB_CB_FAST_CHECK;
        }

        switch( s.calibrationPattern ) // Find feature points on the input format
        {
    
    
        case Settings::CHESSBOARD:
            found = findChessboardCorners( view, s.boardSize, pointBuf, chessBoardFlags);
            break;
        case Settings::CIRCLES_GRID:
            found = findCirclesGrid( view, s.boardSize, pointBuf );
            break;
        case Settings::ASYMMETRIC_CIRCLES_GRID:
            found = findCirclesGrid( view, s.boardSize, pointBuf, CALIB_CB_ASYMMETRIC_GRID );
            break;
        default:
            found = false;
            break;
        }
        //! [find_pattern]
        //! [pattern_found]
        if ( found)                // If done with success,
        {
    
    
              // improve the found corners' coordinate accuracy for chessboard
                if( s.calibrationPattern == Settings::CHESSBOARD)
                {
    
    
                    Mat viewGray;
                    cvtColor(view, viewGray, COLOR_BGR2GRAY);
                    cornerSubPix( viewGray, pointBuf, Size(winSize,winSize),
                        Size(-1,-1), TermCriteria( TermCriteria::EPS+TermCriteria::COUNT, 30, 0.0001 ));
                }

                if( mode == CAPTURING &&  // For camera only take new samples after delay time
                    (!s.inputCapture.isOpened() || clock() - prevTimestamp > s.delay*1e-3*CLOCKS_PER_SEC) )
                {
    
    
                    imagePoints.push_back(pointBuf);
                    prevTimestamp = clock();
                    blinkOutput = s.inputCapture.isOpened();
                }

                // Draw the corners.
                drawChessboardCorners( view, s.boardSize, Mat(pointBuf), found );
        }
        //! [pattern_found]
        //----------------------------- Output Text ------------------------------------------------
        //! [output_text]
        string msg = (mode == CAPTURING) ? "100/100" :
                      mode == CALIBRATED ? "Calibrated" : "Press 'g' to start";
        int baseLine = 0;
        Size textSize = getTextSize(msg, 1, 1, 1, &baseLine);
        Point textOrigin(view.cols - 2*textSize.width - 10, view.rows - 2*baseLine - 10);

        if( mode == CAPTURING )
        {
    
    
            if(s.showUndistorted)
                msg = cv::format( "%d/%d Undist", (int)imagePoints.size(), s.nrFrames );
            else
                msg = cv::format( "%d/%d", (int)imagePoints.size(), s.nrFrames );
        }

        putText( view, msg, textOrigin, 1, 1, mode == CALIBRATED ?  GREEN : RED);

        if( blinkOutput )
            bitwise_not(view, view);
        //! [output_text]
        //------------------------- Video capture  output  undistorted ------------------------------
        //! [output_undistorted]
        if( mode == CALIBRATED && s.showUndistorted )
        {
    
    
            Mat temp = view.clone();
            if (s.useFisheye)
            {
    
    
                Mat newCamMat;
                fisheye::estimateNewCameraMatrixForUndistortRectify(cameraMatrix, distCoeffs, imageSize,
                                                                    Matx33d::eye(), newCamMat, 1);
                cv::fisheye::undistortImage(temp, view, cameraMatrix, distCoeffs, newCamMat);
            }
            else
              undistort(temp, view, cameraMatrix, distCoeffs);
        }
        //! [output_undistorted]
        //------------------------------ Show image and check for input commands -------------------
        //! [await_input]
        imshow("Image View", view);
        char key = (char)waitKey(s.inputCapture.isOpened() ? 50 : s.delay);

        if( key  == ESC_KEY )
            break;

        if( key == 'u' && mode == CALIBRATED )
           s.showUndistorted = !s.showUndistorted;

        if( s.inputCapture.isOpened() && key == 'g' )
        {
    
    
            mode = CAPTURING;
            imagePoints.clear();
        }
        //! [await_input]
    }

    // -----------------------Show the undistorted image for the image list ------------------------
    //! [show_results]
    if( s.inputType == Settings::IMAGE_LIST && s.showUndistorted && !cameraMatrix.empty())
    {
    
    
        Mat view, rview, map1, map2;

        if (s.useFisheye)
        {
    
    
            Mat newCamMat;
            fisheye::estimateNewCameraMatrixForUndistortRectify(cameraMatrix, distCoeffs, imageSize,
                                                                Matx33d::eye(), newCamMat, 1);
            fisheye::initUndistortRectifyMap(cameraMatrix, distCoeffs, Matx33d::eye(), newCamMat, imageSize,
                                             CV_16SC2, map1, map2);
        }
        else
        {
    
    
            initUndistortRectifyMap(
                cameraMatrix, distCoeffs, Mat(),
                getOptimalNewCameraMatrix(cameraMatrix, distCoeffs, imageSize, 1, imageSize, 0), imageSize,
                CV_16SC2, map1, map2);
        }

        for(size_t i = 0; i < s.imageList.size(); i++ )
        {
    
    
            view = imread(s.imageList[i], IMREAD_COLOR);
            if(view.empty())
                continue;
            remap(view, rview, map1, map2, INTER_LINEAR);
            imshow("Image View", rview);
            char c = (char)waitKey();
            if( c  == ESC_KEY || c == 'q' || c == 'Q' )
                break;
        }
    }
    //! [show_results]

    return 0;
}

//! [compute_errors]
static double computeReprojectionErrors( const vector<vector<Point3f> >& objectPoints,
                                         const vector<vector<Point2f> >& imagePoints,
                                         const vector<Mat>& rvecs, const vector<Mat>& tvecs,
                                         const Mat& cameraMatrix , const Mat& distCoeffs,
                                         vector<float>& perViewErrors, bool fisheye)
{
    
    
    vector<Point2f> imagePoints2;
    size_t totalPoints = 0;
    double totalErr = 0, err;
    perViewErrors.resize(objectPoints.size());

    for(size_t i = 0; i < objectPoints.size(); ++i )
    {
    
    
        if (fisheye)
        {
    
    
            fisheye::projectPoints(objectPoints[i], imagePoints2, rvecs[i], tvecs[i], cameraMatrix,
                                   distCoeffs);
        }
        else
        {
    
    
            projectPoints(objectPoints[i], rvecs[i], tvecs[i], cameraMatrix, distCoeffs, imagePoints2);
        }
        err = norm(imagePoints[i], imagePoints2, NORM_L2);

        size_t n = objectPoints[i].size();
        perViewErrors[i] = (float) std::sqrt(err*err/n);
        totalErr        += err*err;
        totalPoints     += n;
    }

    return std::sqrt(totalErr/totalPoints);
}
//! [compute_errors]
//! [board_corners]
static void calcBoardCornerPositions(Size boardSize, float squareSize, vector<Point3f>& corners,
                                     Settings::Pattern patternType /*= Settings::CHESSBOARD*/)
{
    
    
    corners.clear();

    switch(patternType)
    {
    
    
    case Settings::CHESSBOARD:
    case Settings::CIRCLES_GRID:
        for( int i = 0; i < boardSize.height; ++i )
            for( int j = 0; j < boardSize.width; ++j )
                corners.push_back(Point3f(j*squareSize, i*squareSize, 0));
        break;

    case Settings::ASYMMETRIC_CIRCLES_GRID:
        for( int i = 0; i < boardSize.height; i++ )
            for( int j = 0; j < boardSize.width; j++ )
                corners.push_back(Point3f((2*j + i % 2)*squareSize, i*squareSize, 0));
        break;
    default:
        break;
    }
}
//! [board_corners]
static bool runCalibration( Settings& s, Size& imageSize, Mat& cameraMatrix, Mat& distCoeffs,
                            vector<vector<Point2f> > imagePoints, vector<Mat>& rvecs, vector<Mat>& tvecs,
                            vector<float>& reprojErrs,  double& totalAvgErr, vector<Point3f>& newObjPoints,
                            float grid_width, bool release_object)
{
    
    
    //! [fixed_aspect]
    cameraMatrix = Mat::eye(3, 3, CV_64F);
    if( !s.useFisheye && s.flag & CALIB_FIX_ASPECT_RATIO )
        cameraMatrix.at<double>(0,0) = s.aspectRatio;
    //! [fixed_aspect]
    if (s.useFisheye) {
    
    
        distCoeffs = Mat::zeros(4, 1, CV_64F);
    } else {
    
    
        distCoeffs = Mat::zeros(8, 1, CV_64F);
    }

    vector<vector<Point3f> > objectPoints(1);
    calcBoardCornerPositions(s.boardSize, s.squareSize, objectPoints[0], s.calibrationPattern);
    objectPoints[0][s.boardSize.width - 1].x = objectPoints[0][0].x + grid_width;
    newObjPoints = objectPoints[0];

    objectPoints.resize(imagePoints.size(),objectPoints[0]);

    //Find intrinsic and extrinsic camera parameters
    double rms;

    if (s.useFisheye) {
    
    
        Mat _rvecs, _tvecs;
        rms = fisheye::calibrate(objectPoints, imagePoints, imageSize, cameraMatrix, distCoeffs, _rvecs,
                                 _tvecs, s.flag);

        rvecs.reserve(_rvecs.rows);
        tvecs.reserve(_tvecs.rows);
        for(int i = 0; i < int(objectPoints.size()); i++){
    
    
            rvecs.push_back(_rvecs.row(i));
            tvecs.push_back(_tvecs.row(i));
        }
    } else {
    
    
        int iFixedPoint = -1;
        if (release_object)
            iFixedPoint = s.boardSize.width - 1;
        rms = calibrateCameraRO(objectPoints, imagePoints, imageSize, iFixedPoint,
                                cameraMatrix, distCoeffs, rvecs, tvecs, newObjPoints,
                                s.flag | CALIB_USE_LU);
    }

    if (release_object) {
    
    
        cout << "New board corners: " << endl;
        cout << newObjPoints[0] << endl;
        cout << newObjPoints[s.boardSize.width - 1] << endl;
        cout << newObjPoints[s.boardSize.width * (s.boardSize.height - 1)] << endl;
        cout << newObjPoints.back() << endl;
    }

    cout << "Re-projection error reported by calibrateCamera: "<< rms << endl;

    bool ok = checkRange(cameraMatrix) && checkRange(distCoeffs);

    objectPoints.clear();
    objectPoints.resize(imagePoints.size(), newObjPoints);
    totalAvgErr = computeReprojectionErrors(objectPoints, imagePoints, rvecs, tvecs, cameraMatrix,
                                            distCoeffs, reprojErrs, s.useFisheye);

    return ok;
}

// Print camera parameters to the output file
static void saveCameraParams( Settings& s, Size& imageSize, Mat& cameraMatrix, Mat& distCoeffs,
                              const vector<Mat>& rvecs, const vector<Mat>& tvecs,
                              const vector<float>& reprojErrs, const vector<vector<Point2f> >& imagePoints,
                              double totalAvgErr, const vector<Point3f>& newObjPoints )
{
    
    
    FileStorage fs( s.outputFileName, FileStorage::WRITE );

    time_t tm;
    time( &tm );
    struct tm *t2 = localtime( &tm );
    char buf[1024];
    strftime( buf, sizeof(buf), "%c", t2 );

    fs << "calibration_time" << buf;

    if( !rvecs.empty() || !reprojErrs.empty() )
        fs << "nr_of_frames" << (int)std::max(rvecs.size(), reprojErrs.size());
    fs << "image_width" << imageSize.width;
    fs << "image_height" << imageSize.height;
    fs << "board_width" << s.boardSize.width;
    fs << "board_height" << s.boardSize.height;
    fs << "square_size" << s.squareSize;

    if( !s.useFisheye && s.flag & CALIB_FIX_ASPECT_RATIO )
        fs << "fix_aspect_ratio" << s.aspectRatio;

    if (s.flag)
    {
    
    
        std::stringstream flagsStringStream;
        if (s.useFisheye)
        {
    
    
            flagsStringStream << "flags:"
                << (s.flag & fisheye::CALIB_FIX_SKEW ? " +fix_skew" : "")
                << (s.flag & fisheye::CALIB_FIX_K1 ? " +fix_k1" : "")
                << (s.flag & fisheye::CALIB_FIX_K2 ? " +fix_k2" : "")
                << (s.flag & fisheye::CALIB_FIX_K3 ? " +fix_k3" : "")
                << (s.flag & fisheye::CALIB_FIX_K4 ? " +fix_k4" : "")
                << (s.flag & fisheye::CALIB_RECOMPUTE_EXTRINSIC ? " +recompute_extrinsic" : "");
        }
        else
        {
    
    
            flagsStringStream << "flags:"
                << (s.flag & CALIB_USE_INTRINSIC_GUESS ? " +use_intrinsic_guess" : "")
                << (s.flag & CALIB_FIX_ASPECT_RATIO ? " +fix_aspectRatio" : "")
                << (s.flag & CALIB_FIX_PRINCIPAL_POINT ? " +fix_principal_point" : "")
                << (s.flag & CALIB_ZERO_TANGENT_DIST ? " +zero_tangent_dist" : "")
                << (s.flag & CALIB_FIX_K1 ? " +fix_k1" : "")
                << (s.flag & CALIB_FIX_K2 ? " +fix_k2" : "")
                << (s.flag & CALIB_FIX_K3 ? " +fix_k3" : "")
                << (s.flag & CALIB_FIX_K4 ? " +fix_k4" : "")
                << (s.flag & CALIB_FIX_K5 ? " +fix_k5" : "");
        }
        fs.writeComment(flagsStringStream.str());
    }

    fs << "flags" << s.flag;

    fs << "fisheye_model" << s.useFisheye;

    fs << "camera_matrix" << cameraMatrix;
    fs << "distortion_coefficients" << distCoeffs;

    fs << "avg_reprojection_error" << totalAvgErr;
    if (s.writeExtrinsics && !reprojErrs.empty())
        fs << "per_view_reprojection_errors" << Mat(reprojErrs);

    if(s.writeExtrinsics && !rvecs.empty() && !tvecs.empty() )
    {
    
    
        CV_Assert(rvecs[0].type() == tvecs[0].type());
        Mat bigmat((int)rvecs.size(), 6, CV_MAKETYPE(rvecs[0].type(), 1));
        bool needReshapeR = rvecs[0].depth() != 1 ? true : false;
        bool needReshapeT = tvecs[0].depth() != 1 ? true : false;

        for( size_t i = 0; i < rvecs.size(); i++ )
        {
    
    
            Mat r = bigmat(Range(int(i), int(i+1)), Range(0,3));
            Mat t = bigmat(Range(int(i), int(i+1)), Range(3,6));

            if(needReshapeR)
                rvecs[i].reshape(1, 1).copyTo(r);
            else
            {
    
    
                //*.t() is MatExpr (not Mat) so we can use assignment operator
                CV_Assert(rvecs[i].rows == 3 && rvecs[i].cols == 1);
                r = rvecs[i].t();
            }

            if(needReshapeT)
                tvecs[i].reshape(1, 1).copyTo(t);
            else
            {
    
    
                CV_Assert(tvecs[i].rows == 3 && tvecs[i].cols == 1);
                t = tvecs[i].t();
            }
        }
        fs.writeComment("a set of 6-tuples (rotation vector + translation vector) for each view");
        fs << "extrinsic_parameters" << bigmat;
    }

    if(s.writePoints && !imagePoints.empty() )
    {
    
    
        Mat imagePtMat((int)imagePoints.size(), (int)imagePoints[0].size(), CV_32FC2);
        for( size_t i = 0; i < imagePoints.size(); i++ )
        {
    
    
            Mat r = imagePtMat.row(int(i)).reshape(2, imagePtMat.cols);
            Mat imgpti(imagePoints[i]);
            imgpti.copyTo(r);
        }
        fs << "image_points" << imagePtMat;
    }

    if( s.writeGrid && !newObjPoints.empty() )
    {
    
    
        fs << "grid_points" << newObjPoints;
    }
}

//! [run_and_save]
bool runCalibrationAndSave(Settings& s, Size imageSize, Mat& cameraMatrix, Mat& distCoeffs,
                           vector<vector<Point2f> > imagePoints, float grid_width, bool release_object)
{
    
    
    vector<Mat> rvecs, tvecs;
    vector<float> reprojErrs;
    double totalAvgErr = 0;
    vector<Point3f> newObjPoints;

    bool ok = runCalibration(s, imageSize, cameraMatrix, distCoeffs, imagePoints, rvecs, tvecs, reprojErrs,
                             totalAvgErr, newObjPoints, grid_width, release_object);
    cout << (ok ? "Calibration succeeded" : "Calibration failed")
         << ". avg re projection error = " << totalAvgErr << endl;

    if (ok)
        saveCameraParams(s, imageSize, cameraMatrix, distCoeffs, rvecs, tvecs, reprojErrs, imagePoints,
                         totalAvgErr, newObjPoints);
    return ok;
}

4. Source code explanation

4.1 Read the configuration file

The content of the configuration file in_VID5.xmlis as follows, you can modify it according to your needs

<?xml version="1.0"?>
<opencv_storage>
<Settings>
  <!-- Number of inner corners per a item row and column. (square, circle) -->
  <BoardSize_Width> 9</BoardSize_Width>
  <BoardSize_Height>6</BoardSize_Height>
  
  <!-- The size of a square in some user defined metric system (pixel, millimeter)-->
  <Square_Size>50</Square_Size>
  
  <!-- The type of input used for camera calibration. One of: CHESSBOARD CIRCLES_GRID ASYMMETRIC_CIRCLES_GRID -->
  <Calibrate_Pattern>"CHESSBOARD"</Calibrate_Pattern>
  
  <!-- The input to use for calibration. 
		To use an input camera -> give the ID of the camera, like "1"
		To use an input video  -> give the path of the input video, like "/tmp/x.avi"
		To use an image list   -> give the path to the XML or YAML file containing the list of the images, like "/tmp/circles_list.xml"
		-->
  <Input>"images/CameraCalibration/VID5/VID5.xml"</Input>
  <!--  If true (non-zero) we flip the input images around the horizontal axis.-->
  <Input_FlipAroundHorizontalAxis>0</Input_FlipAroundHorizontalAxis>
  
  <!-- Time delay between frames in case of camera. -->
  <Input_Delay>100</Input_Delay>	
  
  <!-- How many frames to use, for calibration. -->
  <Calibrate_NrOfFrameToUse>25</Calibrate_NrOfFrameToUse>
  <!-- Consider only fy as a free parameter, the ratio fx/fy stays the same as in the input cameraMatrix. 
	   Use or not setting. 0 - False Non-Zero - True-->
  <Calibrate_FixAspectRatio> 1 </Calibrate_FixAspectRatio>
  <!-- If true (non-zero) tangential distortion coefficients  are set to zeros and stay zero.-->
  <Calibrate_AssumeZeroTangentialDistortion>1</Calibrate_AssumeZeroTangentialDistortion>
  <!-- If true (non-zero) the principal point is not changed during the global optimization.-->
  <Calibrate_FixPrincipalPointAtTheCenter> 1 </Calibrate_FixPrincipalPointAtTheCenter>
  
  <!-- The name of the output log file. -->
  <Write_outputFileName>"out_camera_data.xml"</Write_outputFileName>
  <!-- If true (non-zero) we write to the output file the feature points.-->
  <Write_DetectedFeaturePoints>1</Write_DetectedFeaturePoints>
  <!-- If true (non-zero) we write to the output file the extrinsic camera parameters.-->
  <Write_extrinsicParameters>1</Write_extrinsicParameters>
  <!-- If true (non-zero) we write to the output file the refined 3D target grid points.-->
  <Write_gridPoints>1</Write_gridPoints>
  <!-- If true (non-zero) we show after calibration the undistorted images.-->
  <Show_UndistortedImage>1</Show_UndistortedImage>
  <!-- If true (non-zero) will be used fisheye camera model.-->
  <Calibrate_UseFisheyeModel>0</Calibrate_UseFisheyeModel>
  <!-- If true (non-zero) distortion coefficient k1 will be equals to zero.-->
  <Fix_K1>0</Fix_K1>
  <!-- If true (non-zero) distortion coefficient k2 will be equals to zero.-->
  <Fix_K2>0</Fix_K2>
  <!-- If true (non-zero) distortion coefficient k3 will be equals to zero.-->
  <Fix_K3>0</Fix_K3>
  <!-- If true (non-zero) distortion coefficient k4 will be equals to zero.-->
  <Fix_K4>1</Fix_K4>
  <!-- If true (non-zero) distortion coefficient k5 will be equals to zero.-->
  <Fix_K5>1</Fix_K5>
</Settings>
</opencv_storage>

Here is in_VID5.xmlthe code to read the file:

	Settings s;
    const string inputSettingsFile = parser.get<string>(0);
    FileStorage fs(inputSettingsFile, FileStorage::READ); // Read the settings
    if (!fs.isOpened())
    {
    
    
        cout << "Could not open the configuration file: \"" << inputSettingsFile << "\"" << endl;
        parser.printMessage();
        return -1;
    }
    fs["Settings"] >> s;
    fs.release();

For this I use simple OpenCV class input operations. After reading the file, I have an additional postprocessing function that checks the input for validity. Only true if all inputs are good input variables.

4.2 Get the next input, if it fails or we have enough parameters - Calibration

After this we have a big loop where we do the following: Get the next image from the image list, camera or video file. If this fails or we have enough images, then we run the calibration process. In the case of images, we break out of the loop, otherwise the remaining frames are used for dewarping (if that option is set) by changing from detection mode to calibration mode.

for(;;)
    {
    
    
        Mat view;
        bool blinkOutput = false;
        view = s.nextImage();
        //-----  If no more image, or got enough, then stop calibration and show result -------------
        if( mode == CAPTURING && imagePoints.size() >= (size_t)s.nrFrames )
        {
    
    
          if(runCalibrationAndSave(s, imageSize,  cameraMatrix, distCoeffs, imagePoints, grid_width, release_object))
              mode = CALIBRATED;
          else
              mode = DETECTION;
        }
        if(view.empty())          // If there are no more images stop the loop
        {
    
    
            // if calibration threshold was not reached yet, calibrate now
            if( mode != CALIBRATED && !imagePoints.empty() )
                runCalibrationAndSave(s, imageSize,  cameraMatrix, distCoeffs, imagePoints, grid_width,
                                      release_object);
            break;
        }

For some cameras, we may need to flip the input image. We do the same here.

4.3 Finding patterns in the current input

The equations I mentioned above are formed to look for dominant patterns in the input: in a chessboard, these patterns are the corners of the squares, and in the case of a circle, that is the circle itself. These positions will form the result of being written to the pointBuf vector.

		vector<Point2f> pointBuf;
        bool found;
        int chessBoardFlags = CALIB_CB_ADAPTIVE_THRESH | CALIB_CB_NORMALIZE_IMAGE;
        if(!s.useFisheye) {
    
    
            // fast check erroneously fails with high distortions like fisheye
            chessBoardFlags |= CALIB_CB_FAST_CHECK;
        }
        switch( s.calibrationPattern ) // Find feature points on the input format
        {
    
    
        case Settings::CHESSBOARD:
            found = findChessboardCorners( view, s.boardSize, pointBuf, chessBoardFlags);
            break;
        case Settings::CIRCLES_GRID:
            found = findCirclesGrid( view, s.boardSize, pointBuf );
            break;
        case Settings::ASYMMETRIC_CIRCLES_GRID:
            found = findCirclesGrid( view, s.boardSize, pointBuf, CALIB_CB_ASYMMETRIC_GRID );
            break;
        default:
            found = false;
            break;
        }

Depending on the type of input schema, you can use cv::findChessboardCornersthe or cv::findCirclesGridfunction. For these two, you pass the current image and the size of the calibration plate, and you get the position of the pattern. Additionally, they return a boolean variable indicating whether the pattern was found in the input (we only need to consider images for which this is true!)

In the case of the camera, we only take the camera image when an input delay time has elapsed. What this does is allow the user to move the board around, and get a different image. Similar images will generate similar equations, and similar equations on the calibration step will form an ill-posed problem, so the calibration will fail. For square images, the corner positions are only approximate. We can cv::cornerSubPiximprove this by calling functions. winSizeUsed to control the side length of the search window. The default value is 11. winSizeCan be changed via command line arguments --winSize=<number>. It will produce better calibration results. After this, we imagePointsadd a valid input result to the vector to collect all equations into a container. Finally, to visualize the results, we will use cv::findChessboardCornersthe function to plot the found points on the input image.

		if ( found)                // If done with success,
        {
    
    
              // improve the found corners' coordinate accuracy for chessboard
                if( s.calibrationPattern == Settings::CHESSBOARD)
                {
    
    
                    Mat viewGray;
                    cvtColor(view, viewGray, COLOR_BGR2GRAY);
                    cornerSubPix( viewGray, pointBuf, Size(winSize,winSize),
                        Size(-1,-1), TermCriteria( TermCriteria::EPS+TermCriteria::COUNT, 30, 0.0001 ));
                }
                if( mode == CAPTURING &&  // For camera only take new samples after delay time
                    (!s.inputCapture.isOpened() || clock() - prevTimestamp > s.delay*1e-3*CLOCKS_PER_SEC) )
                {
    
    
                    imagePoints.push_back(pointBuf);
                    prevTimestamp = clock();
                    blinkOutput = s.inputCapture.isOpened();
                }
                // Draw the corners.
                drawChessboardCorners( view, s.boardSize, Mat(pointBuf), found );
        }

4.4 Display of status and results to the user, plus command-line control of the application

This part displays text output over the image.

        string msg = (mode == CAPTURING) ? "100/100" :
                      mode == CALIBRATED ? "Calibrated" : "Press 'g' to start";
        int baseLine = 0;
        Size textSize = getTextSize(msg, 1, 1, 1, &baseLine);
        Point textOrigin(view.cols - 2*textSize.width - 10, view.rows - 2*baseLine - 10);
        if( mode == CAPTURING )
        {
    
    
            if(s.showUndistorted)
                msg = cv::format( "%d/%d Undist", (int)imagePoints.size(), s.nrFrames );
            else
                msg = cv::format( "%d/%d", (int)imagePoints.size(), s.nrFrames );
        }
        putText( view, msg, textOrigin, 1, 1, mode == CALIBRATED ?  GREEN : RED);
        if( blinkOutput )
            bitwise_not(view, view);

If we run the calibration and get a camera matrix with distortion coefficients, we probably want to use cv:: undistortionthe function to rectify the image:

        if( mode == CALIBRATED && s.showUndistorted )
        {
    
    
            Mat temp = view.clone();
            if (s.useFisheye)
            {
    
    
                Mat newCamMat;
                fisheye::estimateNewCameraMatrixForUndistortRectify(cameraMatrix, distCoeffs, imageSize,
                                                                    Matx33d::eye(), newCamMat, 1);
                cv::fisheye::undistortImage(temp, view, cameraMatrix, distCoeffs, newCamMat);
            }
            else
              undistort(temp, view, cameraMatrix, distCoeffs);
        }

Then we display the image and wait for the key to be entered, if it is u, we toggle the distortion removal, if it is g, we start the detection process again, and finally for ESCthe key, we exit the application:

        imshow("Image View", view);
        char key = (char)waitKey(s.inputCapture.isOpened() ? 50 : s.delay);
        if( key  == ESC_KEY )
            break;
        if( key == 'u' && mode == CALIBRATED )
           s.showUndistorted = !s.showUndistorted;
        if( s.inputCapture.isOpened() && key == 'g' )
        {
    
    
            mode = CAPTURING;
            imagePoints.clear();
        }

4.5 Display Distortion Removed Image

When you're using a list of images, it's not possible to remove distortion in a loop. So this has to be done after the loop. Using this, I now extend cv::undistortthe function, which actually calls first cv::initUndistortRectifyMapto find the transformation matrix, and then uses cv::remapthe function to perform the transformation. Since, after a successful calibration, the mapping only needs to be calculated once, you can speed up your application by using this extended form:

    if( s.inputType == Settings::IMAGE_LIST && s.showUndistorted && !cameraMatrix.empty())
    {
    
    
        Mat view, rview, map1, map2;
        if (s.useFisheye)
        {
    
    
            Mat newCamMat;
            fisheye::estimateNewCameraMatrixForUndistortRectify(cameraMatrix, distCoeffs, imageSize,
                                                                Matx33d::eye(), newCamMat, 1);
            fisheye::initUndistortRectifyMap(cameraMatrix, distCoeffs, Matx33d::eye(), newCamMat, imageSize,
                                             CV_16SC2, map1, map2);
        }
        else
        {
    
    
            initUndistortRectifyMap(
                cameraMatrix, distCoeffs, Mat(),
                getOptimalNewCameraMatrix(cameraMatrix, distCoeffs, imageSize, 1, imageSize, 0), imageSize,
                CV_16SC2, map1, map2);
        }
        for(size_t i = 0; i < s.imageList.size(); i++ )
        {
    
    
            view = imread(s.imageList[i], IMREAD_COLOR);
            if(view.empty())
                continue;
            remap(view, rview, map1, map2, INTER_LINEAR);
            imshow("Image View", rview);
            char c = (char)waitKey();
            if( c  == ESC_KEY || c == 'q' || c == 'Q' )
                break;
        }
    }

4.6 Calibration and storage

Since each camera only needs to be calibrated once, it makes sense to save it after a successful calibration. This way, you can later load these values into your program. So we first do the calibration, and if the calibration is successful, we save the result to an OpenCV-style XML or YAML file, depending on the extension you gave it in the configuration file.

So in the first function we separate the two processes. Since we want to save a lot of calibration variables, we'll create those here and pass them to the scale and save functions. Again, I won't show the saving part since it has nothing in common with calibration.

bool runCalibrationAndSave(Settings& s, Size imageSize, Mat& cameraMatrix, Mat& distCoeffs,
                           vector<vector<Point2f> > imagePoints, float grid_width, bool release_object)
{
    
    
    vector<Mat> rvecs, tvecs;
    vector<float> reprojErrs;
    double totalAvgErr = 0;
    vector<Point3f> newObjPoints;
    bool ok = runCalibration(s, imageSize, cameraMatrix, distCoeffs, imagePoints, rvecs, tvecs, reprojErrs,
                             totalAvgErr, newObjPoints, grid_width, release_object);
    cout << (ok ? "Calibration succeeded" : "Calibration failed")
         << ". avg re projection error = " << totalAvgErr << endl;
    if (ok)
        saveCameraParams(s, imageSize, cameraMatrix, distCoeffs, rvecs, tvecs, reprojErrs, imagePoints,
                         totalAvgErr, newObjPoints);
    return ok;
}

We cv::calibrateCameraROcalibrate with the help of functions. It has the following parameters:

The object points: vector<Point3f>A vector, for each input image, which describes the appearance of the pattern. If we have a flat pattern (like a checkerboard) then we can simply set all Zcoordinates to zero. Here is a collection of these important points. Since, we use a single mode for all input images, we only need to calculate it once and then multiply it by all other input views. We calcBoardCornerPositionscalculate the corner points with the function:

static void calcBoardCornerPositions(Size boardSize, float squareSize, vector<Point3f>& corners,
                                     Settings::Pattern patternType /*= Settings::CHESSBOARD*/)
{
    
    
    corners.clear();
    switch(patternType)
    {
    
    
    case Settings::CHESSBOARD:
    case Settings::CIRCLES_GRID:
        for( int i = 0; i < boardSize.height; ++i )
            for( int j = 0; j < boardSize.width; ++j )
                corners.push_back(Point3f(j*squareSize, i*squareSize, 0));
        break;
    case Settings::ASYMMETRIC_CIRCLES_GRID:
        for( int i = 0; i < boardSize.height; i++ )
            for( int j = 0; j < boardSize.width; j++ )
                corners.push_back(Point3f((2*j + i % 2)*squareSize, i*squareSize, 0));
        break;
    default:
        break;
    }
}

Then multiply it by:

vector<vector<Point3f> > objectPoints(1);
calcBoardCornerPositions(s.boardSize, s.squareSize, objectPoints[0], s.calibrationPattern);
objectPoints[0][s.boardSize.width - 1].x = objectPoints[0][0].x + grid_width;
newObjPoints = objectPoints[0];
objectPoints.resize(imagePoints.size(),objectPoints[0]);

PLEASE NOTE:
If your calibration board is an inaccurate, unmeasured, roughly planar target (a checkerboard pattern on paper with off-the-shelf printers is the most convenient calibration target, but most are not precise enough), you can utilize a method to significantly improve the accuracy of estimating camera intrinsic parameters. If command line arguments are provided -d=<number>, this new calibration method will be called. In the code snippet above, the value grid_widthis actually -d=<number>set by . It is the measured distance between top left (0,0,0)and top right . (s.squareSize*(s.boardSize.width-1), 0, 0)It should be measured accurately with a ruler or calipers. After calibration, newObjPointsit will be updated to precise 3D object point coordinates.

The image points: vector<Point2f>For each input image contains the coordinates of important points (the corners of the chessboard and the center of the circle pattern). We've gathered this from the cv::findChessboardCornersor function. cv::findCirclesGridWe just have to pass it on.
The size of an image obtained from a camera, video file, or image.
The index point of the fixed object. We set it to -1 to require a standard calibration method. If you want to use the new object release method, set this to the index of the upper right point of the calibration plate grid. See for details cv::calibrateCameraRO.

int iFixedPoint = -1;
if (release_object)
    iFixedPoint = s.boardSize.width - 1;

The camera matrix: If we use the fixed aspect ratio option, we need to set f_x:

    cameraMatrix = Mat::eye(3, 3, CV_64F);
    if( !s.useFisheye && s.flag & CALIB_FIX_ASPECT_RATIO )
        cameraMatrix.at<double>(0,0) = s.aspectRatio;

The distortion coefficient matrix: initialized to zero

distCoeffs = Mat::zeros(8, 1, CV_64F);

For all views, the function will compute rotation and translation vectors that transform target points (given in world coordinate space) to image points (given in model coordinate space). The 7th and 8th parameters are the output vectors of the matrix, where the i-th position contains the rotation and translation vectors from the i-th object point to the i-th image point.
An updated output vector of scaled mode points. Standard calibration methods ignore this parameter.
The last parameter is flag. You need to specify some options here, like fixing the aspect ratio of the focal length, assuming zero tangential distortion, or fixing the principal point. Here we use CALIB_USE_LUto get faster calibration speed.

rms = calibrateCameraRO(objectPoints, imagePoints, imageSize, iFixedPoint,
                        cameraMatrix, distCoeffs, rvecs, tvecs, newObjPoints,
                        s.flag | CALIB_USE_LU);

This function returns the average reprojection error. This number gives a good estimate of the precision of the found parameters. This should be as close to zero as possible. Considering the internal parameters, distortion parameters, rotation and translation matrices, we can cv::projectPointscalculate the error of a view by using , first converting object points to image points. We then compute the absolute norm between the result obtained by the transformation and the corner/circle finding algorithm. To find the average error, we compute the arithmetic mean of the computed errors for all calibration images.

static double computeReprojectionErrors( const vector<vector<Point3f> >& objectPoints,
                                         const vector<vector<Point2f> >& imagePoints,
                                         const vector<Mat>& rvecs, const vector<Mat>& tvecs,
                                         const Mat& cameraMatrix , const Mat& distCoeffs,
                                         vector<float>& perViewErrors, bool fisheye)
{
    
    
    vector<Point2f> imagePoints2;
    size_t totalPoints = 0;
    double totalErr = 0, err;
    perViewErrors.resize(objectPoints.size());
    for(size_t i = 0; i < objectPoints.size(); ++i )
    {
    
    
        if (fisheye)
        {
    
    
            fisheye::projectPoints(objectPoints[i], imagePoints2, rvecs[i], tvecs[i], cameraMatrix,
                                   distCoeffs);
        }
        else
        {
    
    
            projectPoints(objectPoints[i], rvecs[i], tvecs[i], cameraMatrix, distCoeffs, imagePoints2);
        }
        err = norm(imagePoints[i], imagePoints2, NORM_L2);
        size_t n = objectPoints[i].size();
        perViewErrors[i] = (float) std::sqrt(err*err/n);
        totalErr        += err*err;
        totalPoints     += n;
    }
    return std::sqrt(totalErr/totalPoints);
}

4.7 Results

Suppose there is an input checkerboard pattern of size 9 X 6. I created two snapshots of the calibration board using an AXIS IP camera and saved them to the VID5 directory. I put it in images/ camercalibrationa folder in my working directory and created the following VID5.XMLfiles to describe which images to use:

<?xml version="1.0"?>
<opencv_storage>
<images>
images/CameraCalibration/VID5/xx1.jpg
images/CameraCalibration/VID5/xx2.jpg
images/CameraCalibration/VID5/xx3.jpg
images/CameraCalibration/VID5/xx4.jpg
images/CameraCalibration/VID5/xx5.jpg
images/CameraCalibration/VID5/xx6.jpg
images/CameraCalibration/VID5/xx7.jpg
images/CameraCalibration/VID5/xx8.jpg
</images>
</opencv_storage>

How to generate the above xml file?

// genImageXML.cpp
#include <opencv2\opencv.hpp>
#include <opencv2\highgui\highgui.hpp>
#include <opencv2\features2d\features2d.hpp>
#include <opencv2\core\core.hpp>

using namespace std;
using namespace cv;

int main()

{
    
    
	string pattern = "D:/code/PycharmProjects/learn_azureKinect/chessboard_5x7_30mm/*.jpg";
	vector<string> fn;
	glob(pattern, fn, false);
	FileStorage fs("./VID5.xml", cv::FileStorage::WRITE);
	fs << "images" <<"[";

	for (auto name : fn)
	{
    
    
		fs<<name;
	}

	fs << "]";

	fs.release();
	return 0;

}

Then pass images/ camercalibration /VID5/VID5. xmlas input for the configuration file. Here's a checkerboard pattern found when the app runs:
insert image description here
After applying distortion removal, we get:

The same method works for this asymmetrical circular pattern, setting the input width to 4 and height to 11. This time, I'm using a live camera input by giving the input an ID ("1"). Here's what the detected patterns look like:
insert image description here
In both cases, in the specified output XML/YAMLfile, you'll find the camera and distortion coefficient matrices:

<camera_matrix type_id="opencv-matrix">
<rows>3</rows>
<cols>3</cols>
<dt>d</dt>
<data>
 6.5746697944293521e+002 0. 3.1950000000000000e+002 0.
 6.5746697944293521e+002 2.3950000000000000e+002 0. 0. 1.</data></camera_matrix>
<distortion_coefficients type_id="opencv-matrix">
<rows>5</rows>
<cols>1</cols>
<dt>d</dt>
<data>
 -4.1802327176423804e-001 5.0715244063187526e-001 0. 0.
 -5.7843597214487474e-001</data></distortion_coefficients>

Add these values as constants to your program, call cv::initUndistortRectifyMapand cv::remapfunction to remove distortion, and enjoy distortion-free input for cheap and low-quality cameras.

BONUS

The following is based on Python+OpenCV to achieve monocular camera calibration
insert image description here

We will now perform the calibration process using OpenCV. To determine the 9 parameters (4 camera intrinsic coefficients and 5 distortion coefficients), we need some distorted checkerboard images - a dataset of at least 10 images is recommended, taken with the camera to be calibrated.

insert image description here
We start by importing and setting up the data for later use in the calibration process. A termination criterion is required for OpenCV's cornerSubPix()function that performs a high-precision search for corners in a checkerboard image. A set of object points is needed to tell OpenCV that we are using an 8 x 8 chessboard as the calibration target.

import cv2
import numpy as np
import glob
 
# 设置cornerSubPix() 的终止标准
criteria = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 30, 0.001)
 
# 为 8x8 棋盘创建和填充对象点
objp = np.zeros((7 * 7, 3), np.float32)
objp[:,:2] = np.mgrid[0:7, 0:7].T.reshape(-1, 2)
 
# 为对象点和图像点创建数组
objpoints = [] # 现实世界空间中的 3d 点
imgpoints = [] # 图像平面中的 2d点

With the initial variables set, we can iterate through the calibration dataset of distorted checkerboard images and apply OpenCV's findChessboardCorners()function to locate the corners in the checkerboard image.

# 收集文件夹中图像的文件名
images = glob.glob('calibration*.png')
 
# 遍历文件夹中的图像并创建棋盘角
for fname in images:
    print(fname)
    image = cv2.imread(fname)
    gray = cv2.split(image)[0]
    ret, corners = cv2.findChessboardCorners(gray, (7, 7), None)
    if ret == True:
        objpoints.append(objp)
        corners_SubPix = cv2.cornerSubPix(gray, corners, (11, 11), (-1, -1), criteria)
        imgpoints.append(corners_SubPix)
        print("Return value: ", ret)
        img = cv2.drawChessboardCorners(gray, (7, 7), corners_SubPix, ret)
        cv2.imshow("Corners", img)
        cv2.waitKey(500)
cv2.destroyAllWindows()

The figure below shows an example output of findChessboardCorners()the function - OpenCV has successfully detected all the interior corners of the distorted checkerboard image, which can then be used to perform the calibration.
insert image description here
The following code uses OpenCV's calibrateCamera()function to determine the camera intrinsic matrix and distortion coefficients. The file storage API is used to save parameters to an XML file.

# 相机标定: cameraMatrix = 3x3 相机内参; 畸变系数distCoeffs = 5x1 向量
# gray.shape[::-1] 将单通道图像值从 h、w 交换为 w、h（numpy 到 OpenCV 格式）
retval, cameraMatrix, distCoeffs, rvecs, tvecs = cv2.calibrateCamera(objpoints, imgpoints, gray.shape[::-1], None, None)
 
# 保存相机内参和畸变系数
fs = cv2.FileStorage("intrinsics.xml", cv2.FileStorage_WRITE)
fs.write("image_width", gray.shape[1])
fs.write("image_height", gray.shape[0])
fs.write("camera_matrix", cameraMatrix)
fs.write("distortion_coefficients", distCoeffs)
fs.release()

After inspecting the XML file in a text editor, determine the 4 values of the camera intrinsic coefficients and 5 distortion coefficients.
Please add a picture description
After successfully obtaining our parameters, we can import a new distorted image and apply OpenCV's undistort function to straighten the image. The image used was taken from a sample kitchen countertop, which needed to be distorted before applying the visual inspection feature - the barrel distortion was very noticeable in the original image.
Please add a picture description
The code below refines the camera matrix for the image to be processed, then computes and applies the transformation.

# 输入失真图像，保留为 3 通道
image_dist = cv2.imread('./sample.png')
print("Distorted image shape: ", image_dist.shape)
cv2.imshow("Distorted Image", image_dist)
cv2.waitKey(0)
 
# 根据比例因子返回相应的新的相机内参矩阵，并得到有效的ROI
h, w = image_dist.shape[:2]
cameraMatrixNew, roi = cv2.getOptimalNewCameraMatrix(cameraMatrix, distCoeffs, (w, h), 1, (w, h))
 
# 计算原始图像和矫正图像之间的转换关系，将结果以映射的形式表达，映射关系存储在map1和map2中
map1, map2 = cv2.initUndistortRectifyMap(cameraMatrix, distCoeffs, None, cameraMatrixNew, (w, h), cv2.CV_32FC1)
 
# 把原始图像中某位置的像素映射到矫正后的图像指定位置，
# 这里的map1和map2就是上面cv::initUndistortRectifyMap()计算出来的结果。
image_undist = cv2.remap(image_dist, map1, map2, cv2.INTER_LINEAR)
cv2.imshow("Undistorted Image Full", image_undist)
cv2.waitKey(0)

Functions can also be used undistortbecause functions and functions undistortare called inside functions.initUndistortRectifyMapremap

The image below is the transformed undistorted output. The black patches at the edges are a by-product of the remapping process, as pixels are repositioned to achieve straightness.
insert image description here
OpenCV's remapping function identified the black patches at the edge of the image above, providing a valid ROI (Region of Interest), giving the largest possible rectangular image after transformation.

# crop undistorted image to valid ROI
print("Valid ROI: ", roi)
x, y, w, h = roi
image_undist = image_undist[y:y+h, x:x+w]
cv2.imshow("Undistorted Image Valid ROI", image_undist)
cv2.waitKey(0)

The resulting image is a cropped valid ROI that can subsequently be used as input to a vision detection algorithm.
insert image description here

reference list

https://docs.opencv.org/4.x/d4/d94/tutorial_camera_calibration.html

Camera calibration and 3D reconstruction (3) Use OpenCV to calibrate the camera