Detailed explanation of Opencv's RANSAC algorithm for straight line fitting and feature point set matching

Detailed explanation of Opencv's RANSAC algorithm for straight line fitting and feature point set matching

• 1. Describe the advantages and disadvantages of Ransac fitting and least squares in curve fitting
• 2. Describe the differences between nearest neighbor matching and Ransac matching when matching feature points
• 3. In addition, Ransac is also used for ellipse fitting, transformation matrix solving, etc.

1. Straight line fitting

1.1 Principle

• The RANSAC (RANdom SAmple Consensus, Random Sampling Consensus) algorithm is an iterative algorithm that correctly estimates the parameters of a mathematical model from a set of data containing "outliers". "Outliers" generally refer to noise in the data, such as mismatches in matching and outliers in the estimation curve. Therefore, RANSAC is also an "outer point" detection algorithm. At the same time, RANSAC is a non-deterministic algorithm, in the sense that it will produce a reasonable result under a certain probability, which allows the use of more iterations to increase its probability.

• The RANSAC algorithm was first proposed by Fischler and Bolles at SRI to solve the LDP (Location Determination Problem) problem.

• A basic assumption for the RANSAC algorithm is that the data is composed of "inner points" and "outer points". "Internal points" are the data that make up the model parameters, and "outer points" are the data that do not fit the model. At the same time, RANSAC assumes: given a set of data containing a small number of "interior points", there is a program that can estimate a model that conforms to the "interior points"

• The main idea of ​​the algorithm:

• Given a data setS, select the minimum number of samples required to build a model (a straight line in space can be determined by at least two points , so the minimum number of samples is 2. The space plane can be determined based on three non-collinear points, so the minimum number of samples is 3. When drawing a circle, the minimum number of samples is 3), and the selected data set is S1
  Use the selected data setS1 to calculate a mathematical model M1
• Use the calculated modelM1 to test the remaining points in the data set. If the tested data points are within the allowable error range, then This data point is judged as an inlier, otherwise it is judged as an outlier. The data set composed of all inliers is recorded as S1*, S1* is called the consistency set of S1
• Compare the number of "inner points" between the current model and the best previously introduced model, and record the model parameters and the number of "inner points" when the maximum number of "inner points" is reached
• Repeat steps 1-4 until the iteration ends or the current model is good enough ("the number of interior points is greater than the set threshold"); the model generated each time is either discarded because there are too few interior points, or it is better than the existing model. The model is better and is chosen

• The process is as shown in the figure below:
  Take two points from the point set to determine a straight line, then select and filter the inner hall by setting rules, and fit it with the most interior points Model as final usable model
  

1.2 Derivation of number of iterations

• According to the introduction of the basic principles of RANSAC above, there are two important parameters that need to be set in this algorithm process, the number of iterations (number of samples) and the distance threshold.
  How large should we choose the number of iterations? Is it possible to know in advance what this value should be? Or can it only be decided based on experience? This value can actually be estimated. Let’s calculate it below.
  

The probability t of an interior point is usually a priori value. Then P is the probability that we want RANSAC to get the correct model. If the value of t is not known in advance, the adaptive iteration number method can be used. That is to say, an infinite number of iterations is set at the beginning, and then each time the model parameter estimate is updated, the current interior point ratio is used as t to estimate the number of iterations.

1.3 Difference from least squares

• The least squares method tries to adapt to all points including external points. Therefore, the least squares method is only suitable for situations where the error is small. If you need to extract a model from a noisy data set (for example, only 20% of the data conforms to the model), the least squares method will be insufficient.
  
• RANSAC is equivalent to a probabilistic model. It calculates the probability of occurrence of internal points and finds the optimal model fitted by a point set outside the noise points, which is usually better able to represent system properties. It is equivalent to iteratively using least squares + sampling testing.

1.4 Code implementation

• C++ implementation:

//====================================================================//
//Program:RANSAC直线拟合，并与最小二乘法结果进行对比
//====================================================================//
#include <iostream>
#include <opencv2/opencv.hpp>


//RANSAC 拟合2D 直线
//输入参数：points--输入点集
//        iterations--迭代次数
//        sigma--数据和模型之间可接受的差值,车道线像素宽带一般为10左右
//              （Parameter use to compute the fitting score）
//        k_min/k_max--拟合的直线斜率的取值范围.
//                     考虑到左右车道线在图像中的斜率位于一定范围内，
//                      添加此参数，同时可以避免检测垂线和水平线
//输出参数:line--拟合的直线参数,It is a vector of 4 floats
//              (vx, vy, x0, y0) where (vx, vy) is a normalized
//              vector collinear to the line and (x0, y0) is some
//              point on the line.
//返回值：无
void fitLineRansac(const std::vector<cv::Point2f>& points,
                   cv::Vec4f &line,
                   int iterations = 1000,
                   double sigma = 1.,
                   double k_min = -7.,
                   double k_max = 7.)
{
    
    
    unsigned int n = points.size();

    if(n<2)
    {
    
    
        return;
    }

    cv::RNG rng;
    double bestScore = -1.;
    for(int k=0; k<iterations; k++)
    {
    
    
        int i1=0, i2=0;
        while(i1==i2)
        {
    
    
            i1 = rng(n);
            i2 = rng(n);
        }
        const cv::Point2f& p1 = points[i1];
        const cv::Point2f& p2 = points[i2];

        cv::Point2f dp = p2-p1;//直线的方向向量
        dp *= 1./norm(dp);
        double score = 0;

        if(dp.y/dp.x<=k_max && dp.y/dp.x>=k_min )
        {
    
    
            for(int i=0; i<n; i++)
            {
    
    
                cv::Point2f v = points[i]-p1;
                double d = v.y*dp.x - v.x*dp.y;//向量a与b叉乘/向量b的摸.||b||=1./norm(dp)
                //score += exp(-0.5*d*d/(sigma*sigma));//误差定义方式的一种
                if( fabs(d)<sigma )
                    score += 1;
            }
        }
        if(score > bestScore)
        {
    
    
            line = cv::Vec4f(dp.x, dp.y, p1.x, p1.y);
            bestScore = score;
        }
    }
}

int main()
{
    
    
    cv::Mat image(720,1280,CV_8UC3,cv::Scalar(125,125,125));

    //以车道线参数为(0.7657,-0.6432,534,548)生成一系列点
    double k = -0.6432/0.7657;
    double b = 548 - k*534;

    std::vector<cv::Point2f> points;

    for (int i = 360; i < 720; i+=10)
    {
    
    
        cv::Point2f point(int((i-b)/k),i);
        points.emplace_back(point);
    }

    //加入直线的随机噪声
    cv::RNG rng((unsigned)time(NULL));
    for (int i = 360; i < 720; i+=10)
    {
    
    
        int x = int((i-b)/k);
        x = rng.uniform(x-10,x+10);
        int y = i;
        y = rng.uniform(y-30,y+30);
        cv::Point2f point(x,y);
        points.emplace_back(point);
    }

    //加入噪声
    for (int i = 0; i < 720; i+=20)
    {
    
    
        int x = rng.uniform(1,640);
        int y = rng.uniform(1,360);

        cv::Point2f point(x,y);
        points.emplace_back(point);
    }





    int n = points.size();
    for (int j = 0; j < n; ++j)
    {
    
    
        cv::circle(image,points[j],5,cv::Scalar(0,0,0),-1);
    }


    //RANSAC 拟合
    if(1)
    {
    
    
        cv::Vec4f lineParam;
        fitLineRansac(points,lineParam,1000,10);
        double k = lineParam[1] / lineParam[0];
        double b = lineParam[3] - k*lineParam[2];

        cv::Point p1,p2;
        p1.y = 720;
        p1.x = ( p1.y - b) / k;

        p2.y = 360;
        p2.x = (p2.y-b) / k;

        cv::line(image,p1,p2,cv::Scalar(0,255,0),2);
    }


    //最小二乘法拟合
    if(1)
    {
    
    
        cv::Vec4f lineParam;
        cv::fitLine(points,lineParam,cv::DIST_L2,0,0.01,0.01);
        double k = lineParam[1] / lineParam[0];
        double b = lineParam[3] - k*lineParam[2];

        cv::Point p1,p2;
        p1.y = 720;
        p1.x = ( p1.y - b) / k;

        p2.y = 360;
        p2.x = (p2.y-b) / k;

        cv::line(image,p1,p2,cv::Scalar(0,0,255),2);
    }




    cv::imshow("image",image);
    cv::waitKey(0);

    return 0;
}

• Python implementation:

#!/usr/bin/env python3
#coding=utf-8

#============================#
#Program:RANSAC_Line.py
===========#

import numpy as np
import random
import math

import cv2

def fitLineRansac(points,iterations=1000,sigma=1.0,k_min=-7,k_max=7):
    """
    RANSAC 拟合2D 直线
    :param points:输入点集,numpy [points_num,1,2],np.float32
    :param iterations:迭代次数
    :param sigma:数据和模型之间可接受的差值,车道线像素宽带一般为10左右
                （Parameter use to compute the fitting score）
    :param k_min:
    :param k_max:k_min/k_max--拟合的直线斜率的取值范围.
                考虑到左右车道线在图像中的斜率位于一定范围内，
                添加此参数，同时可以避免检测垂线和水平线
    :return:拟合的直线参数,It is a vector of 4 floats
                (vx, vy, x0, y0) where (vx, vy) is a normalized
                vector collinear to the line and (x0, y0) is some
                point on the line.
    """
    line = [0,0,0,0]
    points_num = points.shape[0]

    if points_num<2:
        return line

    bestScore = -1
    for k in range(iterations):
        i1,i2 = random.sample(range(points_num), 2)
        p1 = points[i1][0]
        p2 = points[i2][0]

        dp = p1 - p2 #直线的方向向量
        dp *= 1./np.linalg.norm(dp) # 除以模长，进行归一化

        score = 0
        a = dp[1]/dp[0]
        if a <= k_max and a>=k_min:
            for i in range(points_num):
                v = points[i][0] - p1
                dis = v[1]*dp[0] - v[0]*dp[1]#向量a与b叉乘/向量b的摸.||b||=1./norm(dp)
                # score += math.exp(-0.5*dis*dis/(sigma*sigma))误差定义方式的一种
                if math.fabs(dis)<sigma:
                    score += 1
        if score > bestScore:
            line = [dp[0],dp[1],p1[0],p1[1]]
            bestScore = score

    return line



if __name__ == '__main__':
    image = np.ones([720,1280,3],dtype=np.ubyte)*125

    # 以车道线参数为(0.7657, -0.6432, 534, 548)生成一系列点
    k = -0.6432 / 0.7657
    b = 548 - k * 534

    points = []
    for i in range(360,720,10):
        point = (int((i-b)/k),i)
        points.append(point)

    # 加入直线的随机噪声
    for i in range(360,720,10):
        x = int((i-b)/k)
        x = random.sample(range(x-10,x+10),1)
        y = i
        y = random.sample(range(y - 30, y + 30),1)

        point = (x[0],y[0])
        points.append(point)

    # 加入噪声
    for i in range(0,720,20):
        x = random.sample(range(1, 640), 1)
        y = random.sample(range(1, 360), 1)
        point = (x[0], y[0])
        points.append(point)

    for point in points:
        cv2.circle(image,point,5,(0,0,0),-1)


    points = np.array(points).astype(np.float32)
    points = points[:,np.newaxis,:]

    # RANSAC 拟合
    if 1:
        [vx, vy, x, y] = fitLineRansac(points,1000,10)
        k = float(vy) / float(vx)  # 直线斜率
        b = -k * x + y

        p1_y = 720
        p1_x = (p1_y-b) / k
        p2_y = 360
        p2_x = (p2_y-b) / k

        p1 = (int(p1_x),int(p1_y))
        p2 = (int(p2_x), int(p2_y))

        cv2.line(image,p1,p2,(0,255,0),2)

    # 最小二乘法拟合
    if 1:
        [vx, vy, x, y] = cv2.fitLine(points, cv2.DIST_L2, 0, 0.1, 0.01)
        k = float(vy) / float(vx)  # 直线斜率
        b = -k * x + y

        p1_y = 720
        p1_x = (p1_y - b) / k
        p2_y = 360
        p2_x = (p2_y - b) / k

        p1 = (int(p1_x), int(p1_y))
        p2 = (int(p2_x), int(p2_y))

        cv2.line(image, p1, p2, (0, 0, 255), 2)


    cv2.imshow('image',image)
    cv2.waitKey(0)

2. Feature matching

• There will be false matching pairs in feature-based image matching. Therefore, in order to improve the matching rate and achieve precise matching based on rough matching, the following two methods can be used:
  
• Use the RANSAC algorithm to find the optimal homography matrix H. Here, first extract the SIFT feature points for nearest neighbor rough matching, then use Ransac for fine matching, and finally solve the transformation matrix.
• The code is implemented as follows:

//RANSAC算法
int main()
{
    
    
    Mat img_object = imread("./data/101.png", IMREAD_GRAYSCALE);
    Mat img_scene = imread("./data/100.png", IMREAD_GRAYSCALE);

    if (img_object.empty() || img_scene.empty())
    {
    
    
        cout << "Could not open or find the image!\n" << endl;
        return -1;
    }
    //-- Step 1: Detect the keypoints using SURF Detector, compute the descriptors
    int minHessian = 800; // default: 400
    Ptr<SURF> surf = SURF::create(800);
    std::vector<KeyPoint> keypoints_object, keypoints_scene;
    Mat descriptors_object, descriptors_scene;
    surf->detectAndCompute(img_object, noArray(), keypoints_object, descriptors_object);
    surf->detectAndCompute(img_scene, noArray(), keypoints_scene, descriptors_scene);

    //-- Step 2: Matching descriptor vectors with a FLANN based matcher
    // Since SURF is a floating-point descriptor NORM_L2 is used
    Ptr<DescriptorMatcher> matcher = DescriptorMatcher::create(DescriptorMatcher::FLANNBASED);
    std::vector< std::vector<DMatch> > knn_matches;
    matcher->knnMatch(descriptors_object, descriptors_scene, knn_matches, 2);

    //-- Filter matches using the Lowe's ratio test
    const float ratio_thresh = 0.75f;
    std::vector<DMatch> good_matches;
    for (size_t i = 0; i < knn_matches.size(); i++)
    {
    
    
        if (knn_matches[i][0].distance < ratio_thresh * knn_matches[i][1].distance)
        {
    
    
            good_matches.push_back(knn_matches[i][0]);
        }
    }

    //-- Draw matches
    Mat img_matches;
    drawMatches(img_object, keypoints_object, img_scene, keypoints_scene, good_matches, img_matches, Scalar::all(-1),
        Scalar::all(-1), std::vector<char>(), DrawMatchesFlags::NOT_DRAW_SINGLE_POINTS);

    //-- Localize the object
    std::vector<Point2f> obj;
    std::vector<Point2f> scene;

    for (size_t i = 0; i < good_matches.size(); i++)
    {
    
    
        //-- Get the keypoints from the good matches
        obj.push_back(keypoints_object[good_matches[i].queryIdx].pt);
        scene.push_back(keypoints_scene[good_matches[i].trainIdx].pt);
    }
    vector<uchar>inliers;
    Mat H = findHomography(obj, scene, inliers, RANSAC);


    //-- Draw matches with RANSAC
    Mat img_matches_ransac;
    std::vector<DMatch> good_matches_ransac;
    for (size_t i = 0; i < inliers.size(); i++)
    {
    
    
        if (inliers[i])
        {
    
    
            good_matches_ransac.push_back(good_matches[i]);
        }
    }
    drawMatches(img_object, keypoints_object, img_scene, keypoints_scene, good_matches_ransac, img_matches_ransac, Scalar::all(-1),
        Scalar::all(-1), std::vector<char>(), DrawMatchesFlags::NOT_DRAW_SINGLE_POINTS);
    namedWindow("img_matches", WINDOW_NORMAL);
    imshow("img_matches", img_matches);
    imwrite("img_matches.jpg", img_matches);

    namedWindow("img_matches_ransac", WINDOW_NORMAL);
    imshow("img_matches_ransac", img_matches_ransac);
    imwrite("img_matches_ransac.jpg", img_matches_ransac);
	waitKey();

	return 0;
}

• The comparison of the effects of knn matching only and Ransac matching is as follows:
  

reference:

1.https://blog.csdn.net/leonardohaig/article/details/104570965?spm=1001.2014.3001.5506
2.https://blog.csdn.net/H19981118/article/details/122014318?spm=1001.2014.3001.5506