The main job is to sift feature matching based on RANSAC and then use the panoramic image stitching
Following is a brief description of the RANSAC algorithm, self-reference: https://blog.csdn.net/robinhjwy/article/details/79174914
A, RANSAC Overview
Input RANSAC algorithm is a set of observed data can be explained by a parametric model or adapted to the observed data, a number of trusted parameters.
RANSAC to achieve the target by repeatedly selecting a set of random subset of data. The selected subset point is assumed to be the bureau and verified by the following methods:
1. First we assume random point as an initial value within a group bureau. Within this game point and then fitting a model that is adapted to the assumed point Council, all the unknown parameters can be calculated from the assumed point bureau.
2. Model 1 was to test all of the other data, if a point suitable for the estimated model, it is also considered within the bureau point, the point of expansion in the Bureau.
3. If there is more than enough points to be classified as a hypothetical point in the office, it is estimated that the model is reasonable enough.
4. Then, assuming all the points to re-estimate the model Board, as this model only in the initial assumptions game point estimation, there is the subsequent expansion, needs to be updated.
5. Finally, the model was assessed by Bureau estimated error rate in the points with the model.
This whole process is an iterative one, this process is repeated a fixed number of times, each time a model has two outcomes:
1, either because too few points within the office, not as a model, and were discarded,
2, or because better than the existing model is selected.
Two, RANSAC example
A simple example is to find a suitable two-dimensional straight line from a set of observed data. Suppose observed data points contained within the Bureau and outliers, approximately the point where the straight line through the inning, and a straight line away from the outliers. Simple linear least squares method can not be found within a point to adapt to office, because the least squares method and tried to include all the points, including the outliers. Instead, only the RANSAC can come a point calculated by the Bureau of the model, and the probability is high enough yet. However, RANSAC is no guarantee that the result will correct algorithm in order to ensure a high enough reasonable probability, we must be careful choice of parameters of the algorithm.
Three, RANSAC principle
OpenCV to filter erroneous matching using RANSAC algorithm to find an optimal homography matrix H, a matrix size of 3 × 3. RANSAC goal is to find the optimal parameters of the matrix so as to satisfy the maximum number of data points in the matrix, usually so h33 = 1h33 = 1 to the normalized matrix. Since the homography matrix has eight unknown parameters, at least eight linear equation solver, the point corresponding to the location information, a set of two equations can list points, the set of matching points comprises at least four pairs.
Where (x, y) represents the position of the target image corners, (x ', y') angular position of the image scene, s is the scale parameter.
RANSAC algorithm from the matched data set is randomly selected to ensure that four samples are not collinear and between the four samples, the calculated homography matrix, and then use the model to test all the data, and calculates the model satisfies the number of data points and the projection error (i.e., the cost function), if this model is the best model, corresponding to the minimum cost function.
RANSAC algorithm steps:
1. Concentrated Random randomly four sample data (this is not collinear between the four samples) is calculated transformation matrix H, denoted as M from the model data;
2. The error is calculated for all projection data set and the data model M, if the error is less than the threshold value, added to the set point I;
3. If the number of elements in the current I is greater than the set point set point within an optimum I_best, update I_best = I, and update iteration number K;
4. If the number of iterations is greater than k, then exit; otherwise, the number of iterations by 1 and repeat the above steps;
Note: the case where the number of iterations k is not greater than the maximum number of iterations, is constantly updated, rather than fixed, see updated over k; a
wherein, p is the confidence, and generally 0.995; w is the ratio of "the point"; m is the minimum number of samples required for the calculation model = 4;
after determined homography easier, leaving the interior point, the point is easy to use point after screening, discard outer point, an outer point it is possible to It is a false match point.
Four, RANSAC advantages and disadvantages
RANSAC advantage is to estimate the model parameters it robust. For example, you can focus accurately estimated parameters from the data contains a large number of outliers. RANSAC drawback is that it calculates the number of iterations parameter is no upper limit; if you set the upper limit of the number of iterations, the results obtained may not be optimal results, and may even get the wrong result. RANSAC only a certain probability to obtain credible model, with probability proportional to the number of iterations. Another disadvantage of RANSAC is that it requires setting thresholds with issues related.
RANSAC can only estimate from a specific focus on a data model, if there are two (or more) model, RANSAC can not find another model.
The main experiment using sift feature matching algorithm and RANSAC, on sift feature extraction and matching match on my one feature point related articles have detailed introduction to sift feature matching the description here will no longer do.
Here is the main function of the code in this experiment:
from pylab import *
from numpy import *
from PIL import Image
# If you have PCV installed, these imports should work
from PCV.geometry import homography, warp
from PCV.localdescriptors import sift
# 设置数据文件夹的路径
featname = ['D:\pythonxy\image\we'+str(i+1)+'.sift' for i in range(5)]
imname = ['D:\pythonxy\image\we'+str(i+1)+'.jpg' for i in range(5)]
# 使用sift特征自动找到匹配对应
l = {}
d = {}
for i in range(5):
sift.process_image(imname[i],featname[i])
l[i],d[i] = sift.read_features_from_file(featname[i])
matches = {}
for i in range(4):
matches[i] = sift.match(d[i+1],d[i])
# 可视化两张图像
for i in range(4):
im1 = array(Image.open(imname[i]))
im2 = array(Image.open(imname[i+1]))
figure()
sift.plot_matches(im2,im1,l[i+1],l[i],matches[i],show_below=True)
# 将匹配项转化为特征点
def convert_points(j):
ndx = matches[j].nonzero()[0]
fp = homography.make_homog(l[j+1][ndx,:2].T)
ndx2 = [int(matches[j][i]) for i in ndx]
tp = homography.make_homog(l[j][ndx2,:2].T)
# switch x and y - TODO this should move elsewhere
fp = vstack([fp[1],fp[0],fp[2]])
tp = vstack([tp[1],tp[0],tp[2]])
return fp,tp
# 估计单应性矩阵
model = homography.RansacModel()
fp,tp = convert_points(1)
H_12 = homography.H_from_ransac(fp,tp,model)[0] #im 1 to 2
fp,tp = convert_points(0)
H_01 = homography.H_from_ransac(fp,tp,model)[0] #im 0 to 1
tp,fp = convert_points(2) #NB: reverse order
H_32 = homography.H_from_ransac(fp,tp,model)[0] #im 3 to 2
tp,fp = convert_points(3) #NB: reverse order
H_43 = homography.H_from_ransac(fp,tp,model)[0] #im 4 to 3
# 扭曲图像
delta = 2000 # for padding and translation
im1 = array(Image.open(imname[1]), "uint8")
im2 = array(Image.open(imname[2]), "uint8")
im_12 = warp.panorama(H_12,im1,im2,delta,delta)
im1 = array(Image.open(imname[0]), "f")
im_02 = warp.panorama(dot(H_12,H_01),im1,im_12,delta,delta)
im1 = array(Image.open(imname[3]), "f")
im_32 = warp.panorama(H_32,im1,im_02,delta,delta)
im1 = array(Image.open(imname[4]), "f")
im_42 = warp.panorama(dot(H_32,H_43),im1,im_32,delta,2*delta)
figure()
imshow(array(im_42, "uint8"))
axis('off')
show()
Wherein automatically find matching features corresponding to sift to the use of process_image sift.py document and match method and read_features_from_file
Sift.process_image effect process is a process image, and wherein the result is stored in a file, as follows:
def process_image(imagename,resultname,params="--edge-thresh 10 --peak-thresh 5"):
""" process an image and save the results in a file"""
path = os.path.abspath(os.path.join(os.path.dirname("__file__"),os.path.pardir))
path = path+"\\utils\\win32vlfeat\\sift.exe "
if imagename[-3:] != 'pgm':
#create a pgm file
im = Image.open(imagename).convert('L')
im.save('tmp.pgm')
imagename = 'tmp.pgm'
cmmd = str("D:\pythonxy\win64vlfeat\sift.exe "+imagename+" --output="+resultname+
" "+params)
os.system(cmmd)
print 'processed', imagename, 'to', resultnameThe method is characterized in sift.read_features_from_file action attribute value of the read image, is then returned in a matrix form, as follows:
def read_features_from_file(filename):
""" read feature properties and return in matrix form"""
f = loadtxt(filename)
return f[:,:4],f[:,4:] # feature locations, descriptorsSift.match action for each process descriptor of an image, which matches the selected image in the second web, as follows:
def match(desc1,desc2):
""" for each descriptor in the first image,
select its match in the second image.
input: desc1 (descriptors for the first image),
desc2 (same for second image). """
desc1 = array([d/linalg.norm(d) for d in desc1])
desc2 = array([d/linalg.norm(d) for d in desc2])
dist_ratio = 0.6
desc1_size = desc1.shape
matchscores = zeros((desc1_size[0],1))
desc2t = desc2.T #precompute matrix transpose
for i in range(desc1_size[0]):
dotprods = dot(desc1[i,:],desc2t) #vector of dot products
dotprods = 0.9999*dotprods
#inverse cosine and sort, return index for features in second image
indx = argsort(arccos(dotprods))
#check if nearest neighbor has angle less than dist_ratio times 2nd
if arccos(dotprods)[indx[0]] < dist_ratio * arccos(dotprods)[indx[1]]:
matchscores[i] = int(indx[0])
return matchscoresAfter reading feature, drawn by their location on the image can be visualized, this function uses to sift.py plot_matches the method, effect of this method is to display an image with a link between the matching connector, code show as below:
def plot_matches(im1,im2,locs1,locs2,matchscores,show_below=True):
""" show a figure with lines joining the accepted matches
input: im1,im2 (images as arrays), locs1,locs2 (location of features),
matchscores (as output from 'match'), show_below (if images should be shown below). """
im3 = appendimages(im1,im2)
if show_below:
im3 = vstack((im3,im3))
# show image
imshow(im3)
# draw lines for matches
cols1 = im1.shape[1]
for i in range(len(matchscores)):
if matchscores[i] > 0:
plot([locs1[i,0], locs2[int(matchscores[i,0]),0]+cols1], [locs1[i,1], locs2[int(matchscores[i,0]),1]], 'c')
axis('off')Then converted into the matching function homogeneous coordinate points using a method make_homog homography.py file, the effect of this method is a set of points (dim * n array) is converted to homogeneous coordinates. code show as below:
def make_homog(points):
""" Convert a set of points (dim*n array) to
homogeneous coordinates. """
return vstack((points,ones((1,points.shape[1]))))
Next is to estimate a homography matrix used H_from_ransac method homography.py file, the process is the use effect H_from_ransac homography between points corresponding RANSAC robust estimation matrix H, the following code:
def H_from_ransac(fp,tp,model,maxiter=1000,match_theshold=10):
""" Robust estimation of homography H from point
correspondences using RANSAC (ransac.py from
http://www.scipy.org/Cookbook/RANSAC).
input: fp,tp (3*n arrays) points in hom. coordinates. """
from PCV.tools import ransac
# 对应点组
data = vstack((fp,tp))
# 计算H，并返回
H,ransac_data = ransac.ransac(data.T,model,4,maxiter,match_theshold,10,return_all=True)
return H,ransac_data['inliers']The method allows the number of points to provide a desired threshold value and the minimum. The most important parameter is the maximum number of iterations: the program exits too early might get a bad solution; the number of iterations will take up too much time. Method returns a result to the homography and right pairs of points should homography matrix.
We also use the RANSAC algorithm to solve Homography, you first need to file the following for the model class homograpy.py test Homography class, the code is as follows:
class RansacModel(object):
""" Class for testing homography fit with ransac.py from
http://www.scipy.org/Cookbook/RANSAC"""
def __init__(self,debug=False):
self.debug = debug
def fit(self, data):
""" Fit homography to four selected correspondences. """
# transpose to fit H_from_points()
data = data.T
# from points
fp = data[:3,:4]
# target points
tp = data[3:,:4]
# fit homography and return
return H_from_points(fp,tp)
def get_error( self, data, H):
""" Apply homography to all correspondences,
return error for each transformed point. """
data = data.T
# from points
fp = data[:3]
# target points
tp = data[3:]
# transform fp
fp_transformed = dot(H,fp)
# normalize hom. coordinates
fp_transformed = normalize(fp_transformed)
# return error per point
return sqrt( sum((tp-fp_transformed)**2,axis=0) )This class contains Fit () method. This method is only accepted by the ransac corresponding to the selected four points, and then fitting a homography matrix. 4 is the minimum number of points calculated homography matrix desired. Since get_error methods each corresponding point using the homography matrix and then returns the corresponding square sum of distances, so it can be determined RANSAC algorithm which points are correct, which is wrong.
Estimated after a single matrix between the images should be all we need to image distortion on a common image plane. We center of the image left or right region of the zero-filled to make room for distorted images. The method used here to a panorama warp.py file, the role of this method is the use of a homography matrix H coordinating two images, creating a horizontal panoramic image. code show as below:
def panorama(H,fromim,toim,padding=2400,delta=2400):
""" Create horizontal panorama by blending two images
using a homography H (preferably estimated using RANSAC).
The result is an image with the same height as toim. 'padding'
specifies number of fill pixels and 'delta' additional translation. """
# check if images are grayscale or color
is_color = len(fromim.shape) == 3
# homography transformation for geometric_transform()
def transf(p):
p2 = dot(H,[p[0],p[1],1])
return (p2[0]/p2[2],p2[1]/p2[2])
if H[1,2]<0: # fromim is to the right
print 'warp - right'
# transform fromim
if is_color:
# pad the destination image with zeros to the right
toim_t = hstack((toim,zeros((toim.shape[0],padding,3))))
fromim_t = zeros((toim.shape[0],toim.shape[1]+padding,toim.shape[2]))
for col in range(3):
fromim_t[:,:,col] = ndimage.geometric_transform(fromim[:,:,col],
transf,(toim.shape[0],toim.shape[1]+padding))
else:
# pad the destination image with zeros to the right
toim_t = hstack((toim,zeros((toim.shape[0],padding))))
fromim_t = ndimage.geometric_transform(fromim,transf,
(toim.shape[0],toim.shape[1]+padding))
else:
print 'warp - left'
# add translation to compensate for padding to the left
H_delta = array([[1,0,0],[0,1,-delta],[0,0,1]])
H = dot(H,H_delta)
# transform fromim
if is_color:
# pad the destination image with zeros to the left
toim_t = hstack((zeros((toim.shape[0],padding,3)),toim))
fromim_t = zeros((toim.shape[0],toim.shape[1]+padding,toim.shape[2]))
for col in range(3):
fromim_t[:,:,col] = ndimage.geometric_transform(fromim[:,:,col],
transf,(toim.shape[0],toim.shape[1]+padding))
else:
# pad the destination image with zeros to the left
toim_t = hstack((zeros((toim.shape[0],padding)),toim))
fromim_t = ndimage.geometric_transform(fromim,
transf,(toim.shape[0],toim.shape[1]+padding))
# blend and return (put fromim above toim)
if is_color:
# all non black pixels
alpha = ((fromim_t[:,:,0] * fromim_t[:,:,1] * fromim_t[:,:,2] ) > 0)
for col in range(3):
toim_t[:,:,col] = fromim_t[:,:,col]*alpha + toim_t[:,:,col]*(1-alpha)
else:
alpha = (fromim_t > 0)
toim_t = fromim_t*alpha + toim_t*(1-alpha)
return toim_t
Here are the results of this experiment drawing a long shot:
This is in view of the results for long-term picture mosaic, because the five pictures of 1000 * 1280 pixels notes, when magnified look will be more fuzzy, but you can see the result of rough stitching is good, because when shooting light position is not so much change the location of the transfer line will not be too obvious, the better to restore the panorama shooting.
The following are the results of this experiment have taken hierarchy diagram:
This is the result diagram for hierarchical mosaic picture, because the light change when shooting stitching boundary changes in light and dark, but still more complete view of the original stitching out.
Here is the result of close-range shot chart this experiment:
This may be my most satisfying of this experiment is a result of design, picture is relatively clear and there is no light and shade staggered, but a high degree of reduction of the picture on the left was too much tension, it is not natural.