Code: De-distortion fisheye camera images

Image projection model: pinhole[fx, fy, cx, cy]
Image distortion model: tangential radial distortion[k1, k2, p1, p2]
Description: for memo

1. The first part is the conventional de-distortion operation, which is to de-distort the fisheye camera when the internal parameters are known. The remap mapping is used here.
2. After de-distorting the image, all the pixels on the edge of the image were lost, so we made a meaningless attempt to retain the edge pixels of the image as much as possible. After implementation, we found that the edge pixels were relatively blurry and the center position was removed. Distortion effects have an impact

Image source and reference link:https://github.com/HLearning/fisheye

1. Part 1: Conventional de-distortion operations

Because it is just a simple verification, the internal parameters, image loading path and saving path are all written in the code
Code snippet:

    cv::Mat K = (cv::Mat_<double>(3, 3) << 652.8609862494474, 0.0, 1262.1021584894233,
         0.0, 653.1909758659955, 928.0871455436396,
         0.0, 0.0, 1.0);
    cv::Mat D = (cv::Mat_<double>(4, 1) << -0.024092199861108887, 0.002745976275100771, 0.002545415522352827, -0.0014366825722748522);
    cv::Mat raw_image = cv::imread("../data/pig.jpg");
    cout << raw_image.cols  << " " << raw_image.rows << endl;
    int width = raw_image.cols;
    int height = raw_image.rows;
    cv::Mat map1, map2;
    cv::Mat undistortImg;
    cv::Size imageSize(width, height);
    cv::fisheye::initUndistortRectifyMap(K, D, cv::Mat(), K, imageSize, CV_16SC2, map1, map2);
    cv::remap(raw_image, undistortImg, map1, map2, cv::INTER_LINEAR, cv::BORDER_CONSTANT);
    cv::imwrite("../data/dst.png", undistortImg);

The original image and the result are as follows, becauseremap the function requires that the input and output image sizes are consistent, so this process causes information loss, and the piglets at the edge are missing a lot.

2. Part 2: Preserving image boundary information

The idea is very simple. Since the pixels are lost because the pixel coordinates exceed the boundary after remap, then just expand the original image and then remove the distortion. The height and width of the image are height and width respectively. Expand the upper and lower boundaries of the image$\frac{1}{4}\cdot height$Image element, image element, image element, left and right side of each world$\frac{1}{4}\cdot width$ Picture element

After the image size is changed, the internal projection parameters will also be changed accordingly, which mainly refers to the offset distance of the far point of the pixel coordinatescx, cy. The top, bottom, left and right sides will each increase by 1/4 of the height and width. , so the overall height and width have increased by 1/2, that is, they have become 1.5 times the original size

Use of edge expansion functioncv::copyMakeBorder

The code snippet is as follows:

    // 扩边处理
    cv::Mat K_ = (cv::Mat_<double>(3, 3) << 652.8609862494474, 0.0, 1.5 * 1262.1021584894233,
                  0.0, 653.1909758659955, 1.5 * 928.0871455436396,
                  0.0, 0.0, 1.0);
    cv::Mat D_ = (cv::Mat_<double>(4, 1) << -0.024092199861108887, 0.002745976275100771, 0.002545415522352827, -0.0014366825722748522);
    int width_ = raw_image.cols + raw_image.cols/2;
    int height_ = raw_image.rows + raw_image.rows/2;

    cv::Mat temp(height_, width_, raw_image.type());
    cv::copyMakeBorder(raw_image, temp, raw_image.rows/4, raw_image.rows/4, raw_image.cols/4, raw_image.cols/4, cv::BORDER_ISOLATED);
    cout << temp.cols  << " " << temp.rows << endl;
    cv::imwrite("../data/src2.png", temp);

    cv::Mat map3, map4;
    cv::Mat undistortImg_;
    cv::Size imageSize_(width_, height_);
    cv::fisheye::initUndistortRectifyMap(K_, D_, cv::Mat(), K_, imageSize_, CV_16SC2, map3, map4);
    cv::remap(temp, undistortImg_, map3, map4, cv::INTER_LINEAR, cv::BORDER_CONSTANT);
    cv::imwrite("../data/dst2.png", undistortImg_);

The results are as follows. You can see that more pixels at the boundary of the image are retained:

3 Discussion

However, this idea is wrong, because after the edge expansion of the image, the input image for remap has changed. The remap mapping needs to process the entire image after edge expansion, which will affect the dedistortion effect of the central area of ​​the image. If Continue to increase the size of the expansion, for example, change the added border from 1/4 of the original height and width to 1/2 of the height and width. It will be obvious that the dedistortion effect in the central area is affected (the two railings become (more curved), which is not what we want.

The correct approach is to make a fuss in remap, create a large blank image in the function, map the dedistorted pixels to the corresponding position on the image, and return this image after remap is completed.

4. Complete code

Source code filesfisheye.cpp

#include <iostream>
#include <string>
#include <opencv2/opencv.hpp>
#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/calib3d/calib3d.hpp>
#include <opencv2/imgproc/imgproc.hpp>

using namespace std;
// using namespace cv;

int main(int argc, char **argv)
{
    cv::Mat K = (cv::Mat_<double>(3, 3) << 652.8609862494474, 0.0, 1262.1021584894233,
         0.0, 653.1909758659955, 928.0871455436396,
         0.0, 0.0, 1.0);
    cv::Mat D = (cv::Mat_<double>(4, 1) << -0.024092199861108887, 0.002745976275100771, 0.002545415522352827, -0.0014366825722748522);
    cv::Mat raw_image = cv::imread("../data/pig.jpg");
    cout << raw_image.cols  << " " << raw_image.rows << endl;
    int width = raw_image.cols;
    int height = raw_image.rows;
    cv::Mat map1, map2;
    cv::Mat undistortImg;
    cv::Size imageSize(width, height);
    cv::fisheye::initUndistortRectifyMap(K, D, cv::Mat(), K, imageSize, CV_16SC2, map1, map2);
    cv::remap(raw_image, undistortImg, map1, map2, cv::INTER_LINEAR, cv::BORDER_CONSTANT);
    cv::imwrite("../data/dst.png", undistortImg);


    // 扩边处理
    cv::Mat K_ = (cv::Mat_<double>(3, 3) << 652.8609862494474, 0.0, 1.5 * 1262.1021584894233,
                  0.0, 653.1909758659955, 1.5 * 928.0871455436396,
                  0.0, 0.0, 1.0);
    cv::Mat D_ = (cv::Mat_<double>(4, 1) << -0.024092199861108887, 0.002745976275100771, 0.002545415522352827, -0.0014366825722748522);
    int width_ = raw_image.cols + raw_image.cols/2;
    int height_ = raw_image.rows + raw_image.rows/2;

    cv::Mat temp(height_, width_, raw_image.type());
    cv::copyMakeBorder(raw_image, temp, raw_image.rows/4, raw_image.rows/4, raw_image.cols/4, raw_image.cols/4, cv::BORDER_ISOLATED);
    cout << temp.cols  << " " << temp.rows << endl;
    cv::imwrite("../data/src2.png", temp);

    cv::Mat map3, map4;
    cv::Mat undistortImg_;
    cv::Size imageSize_(width_, height_);
    cv::fisheye::initUndistortRectifyMap(K_, D_, cv::Mat(), K_, imageSize_, CV_16SC2, map3, map4);
    cv::remap(temp, undistortImg_, map3, map4, cv::INTER_LINEAR, cv::BORDER_CONSTANT);
    cv::imwrite("../data/dst2.png", undistortImg_);

    return 0;
}

CMakeLists.txt file

cmake_minimum_required(VERSION 2.8)

project(fisheye_cali)

find_package(OpenCV REQUIRED)

include_directories(${OpenCV_INCLUDE_DIRS})

add_executable(${PROJECT_NAME} fisheye.cpp)
target_link_libraries(${PROJECT_NAME} ${OpenCV_LIBS})