1. Basic principles
Image mosaic is an important branch of computer vision, it is to mosaic two or more partially overlapping images to obtain higher resolution or wide viewing angle images. This article will combine python+opencv to realize the stitching of two images.
General steps of image stitching:
1. According to the given image/set, realize feature matching
2. Calculate the transformation structure between images by matching features
3. Use the image transformation structure to realize image mapping
4. For the superimposed images, use algorithms such as APAP to align
Feature points
5. Through the graph cut method, automatically select the stitching seam
6. Realize fusion according to the multi-band blending strategy
1.1 SIFT algorithm matching features
SIFT features include interest point detectors and descriptors, which are invariant to scale, rotation and brightness.
Use the SIFT algorithm to obtain the features of the reference image img1 and the source image img2, use the FLANN matching algorithm in opencv to perform feature matching, select high-quality matching points, and visualize the matching results.
code:
# 图像拼接
import cv2 as cv
import numpy as np
import matplotlib.pyplot as plt
# 读取待拼接图像
MIN = 10
img1 = cv.imread('jm1.jpg')
img2 = cv.imread('jm2.jpg')
# 统一图像大小
height1 = int(img1.shape[0])
width1 = int(img1.shape[1])
dim = (width1, height1)
img2 = cv.resize(img2, dim, interpolation=cv.INTER_AREA)
# 创建SIFT特征点检测
sift = cv.SIFT_create()
# 检测兴趣点并计算描述子
kp1, describe1 = sift.detectAndCompute(img1, None)
kp2, describe2 = sift.detectAndCompute(img2, None)
# 使用OpenCV中的FLANN匹配算法进行特征匹配，并返回最近邻和次近邻匹配的结果
FLANN_INDEX_KDTREE = 0
indexParams = dict(algorithm=FLANN_INDEX_KDTREE, trees=5)
searchParams = dict(checks=50)
flann = cv.FlannBasedMatcher(indexParams,searchParams)
matches = flann.knnMatch(describe1, describe2, k=2)
# 储存特征匹配最好的优质匹配点对
'''基于距离阈值选择优质匹配点对，如果最近邻m的距离小于0.65倍的次近邻n的距离，
则认为这个匹配点对是优质的，将它存储在good列表中。'''
good = []
for m,n in matches:
if m.distance < 0.65 * n.distance:
good.append(m)
# 可视化特征匹配结果,并保存
pic3 = cv.drawMatches(img1=img1, keypoints1=kp1, img2=img2, keypoints2=kp2, matches1to2=good, outImg=None)
cv.imwrite(r'/Users/xionglulu/Downloads/project1/m11.jpg', pic3)
result:
Result analysis: The SIFT algorithm is more comprehensive in extracting image feature points, but there are also errors in matching, and the matching effect for points with high similarity is not good. By changing the distance threshold, the number of matching point pairs can be changed. The smaller the threshold, the fewer the number of matches and the higher the accuracy.
1.2 RANSAC algorithm
The RANSAC (Random Sample Consensus) algorithm is an iterative algorithm for estimating model parameters. Its main purpose is to estimate the optimal model parameters from a data set containing noise and outliers.
The basic idea of the RANSAC algorithm is to randomly select a set of data points to fit a model, and then use the model to calculate the fitting error of all data points to the model. A data point is considered an inlier if its error from the model is less than a threshold, otherwise an outlier. The algorithm repeats this process until it finds the optimal model that satisfies a certain confidence level.
The steps of the RANSAC algorithm are as follows:
1. Randomly select a set of data points from the dataset and fit a model based on these data points.
2. For all data points, calculate the error with the model, and regard the data points with an error smaller than a threshold as inliers, otherwise as outliers.
3. If the number of inliers is greater than the specified threshold, re-estimate the model using all inliers and calculate the error for the inliers. If the number of inliers is less than the specified threshold, return to step 1.
4. If the number of inliers in the current model is greater than the previous model, use the current model as the optimal model, and update the threshold and confidence of inliers.
5. Repeat steps 1 to 4 until the specified number of iterations or confidence level is reached.
The image stitching in this paper uses the RANSAC algorithm to calculate the homography transformation matrix between feature points, randomly selects 4 feature point pairs each time, and then maps img2 to img1 through perspective transformation.
1.3 APAP algorithm
In the process of image splicing and fusion, due to the influence of objective factors, the spliced and fused images may have problems such as "ghost phenomenon" and excessive discontinuity between images. To solve this problem, the APAP algorithm can be used.
The APAP algorithm steps are as follows:
1. SIFT obtains matching point pairs of two images
2. Eliminate outliers through RANSAC to obtain N pairs of inner points
3. Use DLT and SVD to calculate global homography
4. Divide the target image into grids and take the grid center point, Calculate the Euclidean distance and weight between each center point and the inner point on the target graph.
5. Put the weight into the A matrix of the DLT algorithm to construct a new W*A matrix, re-SVD decomposition, and naturally get the current network The local homography matrix of grid
6. Traversing each grid, using the local homography matrix to map to the panoramic canvas, the target image after APAP transformation is obtained.
7. Finally, the weighted fusion of splicing lines is carried out
1.4 Find the best stitching seam with the method of maximum flow and minimum cut
After the stitching of two images is completed, there may be a situation: excessive discontinuity between the two images, that is, there is a stitching gap, and the grayscale changes on both sides of the stitching line are more obvious. The max-flow min-cut method can solve this problem.
The maximum flow
is given a specified directed graph, in which there are two special point sources S (Sources) and sinks T (Sinks), and each edge has a specified capacity (Capacity), find the satisfying conditions from S to T Maximum flow (MaxFlow).
Minimal
cut is a division of fixed points in the network, which divides all vertices in the network into two vertex sets S and T, where the source point s∈S and the sink point t∈T. Denoted as CUT(S,T), the minimum cut (Min cut) from S to T that satisfies the conditions.
1.5 multi-band blending algorithm
After finding the stitching seam, due to factors such as image noise, illumination, exposure, model matching error, etc., direct image synthesis will cause more obvious edge traces at the stitching of the image overlapping area. These edge traces need to be eliminated using image fusion algorithms. One of these methods is Multi-Band Blending.
The basic principles of Multi-Band Blending are as follows:
1. Image pyramid
First, for the two images to be fused, they need to be constructed into image pyramids respectively. An image pyramid is a data structure that stores images hierarchically. It can decompose an image into multiple sub-images of different scales, thereby realizing hierarchical image processing.
2. Laplace Pyramid
After constructing the image pyramid, it is necessary to perform Laplace transform on each image pyramid layer. The Laplace transform can decompose each image pyramid layer into a high-frequency component and a low-frequency component. The high-frequency component contains the detailed information of the image, and the low-frequency component contains the overall structure information of the image.
3. Fusion
After obtaining the Laplace transform of each image pyramid layer, they can be fused. The Multi-Band Blending algorithm uses a weighted average method, that is, for each Laplacian pyramid layer, their high-frequency components are weighted and averaged, and their low-frequency components are simply averaged to obtain the final fusion result.
4. Reconstruction
Finally, the fused image pyramid needs to be reconstructed. The process of reconstruction is to inversely transform each Laplacian pyramid layer to obtain the final fused image.
2. Code implementation
# 图像拼接
import cv2 as cv
import numpy as np
import matplotlib.pyplot as plt
# 读取待拼接图像
MIN = 10
img1 = cv.imread('jm1.jpg')
img2 = cv.imread('jm2.jpg')
# 统一图像大小
height1 = int(img1.shape[0])
width1 = int(img1.shape[1])
dim = (width1, height1)
img2 = cv.resize(img2, dim, interpolation=cv.INTER_AREA)
# 创建SIFT特征点检测
sift = cv.SIFT_create()
# 检测兴趣点并计算描述子
kp1, describe1 = sift.detectAndCompute(img1, None)
kp2, describe2 = sift.detectAndCompute(img2, None)
# 使用OpenCV中的FLANN匹配算法进行特征匹配，并返回最近邻和次近邻匹配的结果
FLANN_INDEX_KDTREE = 0
indexParams = dict(algorithm=FLANN_INDEX_KDTREE, trees=5)
searchParams = dict(checks=50)
flann = cv.FlannBasedMatcher(indexParams,searchParams)
matches = flann.knnMatch(describe1, describe2, k=2)
# 储存特征匹配最好的优质匹配点对
'''基于距离阈值选择优质匹配点对，如果最近邻m的距离小于0.65倍的次近邻n的距离，
则认为这个匹配点对是优质的，将它存储在good列表中。'''
good = []
for m,n in matches:
if m.distance < 0.65 * n.distance:
good.append(m)
# 可视化特征匹配结果,并保存
pic3 = cv.drawMatches(img1=img1, keypoints1=kp1, img2=img2, keypoints2=kp2, matches1to2=good, outImg=None)
cv.imwrite(r'/Users/xionglulu/Downloads/project1/m11.jpg', pic3)
# RANSAC算法计算单应性矩阵
if len(good) > MIN:
src_pts = np.float32([kp1[m.queryIdx].pt for m in good]).reshape(-1, 1, 2)
tge_pts = np.float32([kp2[m.trainIdx].pt for m in good]).reshape(-1, 1, 2)
M, mask = cv.findHomography(src_pts, tge_pts, cv.RANSAC, 2)
# 源图像img2图像扭曲（透视变换）
warpimg = cv.warpPerspective(img2, np.linalg.inv(M), (img1.shape[1] + img2.shape[1], img2.shape[0]))
cv.namedWindow("warpimg", cv.WINDOW_NORMAL)
cv.imshow('warpimg',warpimg)
# 拼接图像
direct = warpimg.copy()
direct[0:img1.shape[0], 0:img1.shape[1]] = img1
rows, cols = img1.shape[:2]
left = 0
right = cols
# 找到img1和warpimg重叠的最左边界
for col in range(0, cols):
if img1[:, col].any() and warpimg[:, col].any():
left = col
break
# 找到img1和warpimg重叠的最右边界
for col in range(cols - 1, 0, -1):
if img1[:, col].any() and warpimg[:, col].any():
right = col
break
# 图像融合
res = np.zeros([rows, cols, 3], np.uint8)
for row in range(0, rows):
for col in range(0, cols):
if not img1[row, col].any():
res[row, col] = warpimg[row, col]
elif not warpimg[row, col].any():
res[row, col] = img1[row, col]
else:
# 重叠部分加权平均
srcimgLen = float(abs(col - left))
testimgLen = float(abs(col - right))
alpha = srcimgLen / (srcimgLen + testimgLen)
res[row, col] = np.clip(img1[row, col] * (1 - alpha) + warpimg[row, col] * alpha, 0, 255)
warpimg[0:img1.shape[0], 0:img1.shape[1]] = res
img3 = cv.cvtColor(direct, cv.COLOR_BGR2RGB)
plt.imshow(img3), plt.show()
img4 = cv.cvtColor(warpimg, cv.COLOR_BGR2RGB)
plt.imshow(img4), plt.show()
cv.waitKey()
cv.destroyAllWindows()
else:
print("not enough matches!")
result:
Result analysis:
1. The angle, light, exposure and distance when shooting will have an impact on the image matching results.
2. When the SIFT algorithm performs feature matching, the matching effect on highly similar feature points is not good; when the RANSAC algorithm calculates the homography matrix, the inliers are selected by setting different thresholds, and the matching effect is also different; when the threshold is 2 in the experiment, The matching effect is better, but the image deformation is serious during mapping; in the experiment, when the threshold is 10, the deformation is relieved but the matching effect is not good.
3. When performing image fusion, this experiment uses the weighted average method for the overlapping parts. It can be seen that the color transition on both sides of the seam is relatively natural, but the Multi-Band Blending algorithm can still be used for better fusion. Eliminate seams.
The following introduces another method to achieve image stitching and fusion. cv2.Stitcher_create()An s object is created using the function in the opencv library titcher, and stitcher.stitch((img1, img2))the two images are stitched together using the function.img1img2
cv2.Stitcher_create()The function accepts an optional parameter modethat specifies the stitching mode. modeThe value of can be one of the following two constants:
cv2.STITCHER_PANORAMA: Panorama stitching mode.
cv2.STITCHER_SCANS: scan line splicing mode.
stitcher.stitch()The function returns a tuple containing two elements, where the first element is an integer representing the state of stitching, and the second element is a numpy array representing the stitched image.
code:
import cv2
# 读取待拼接图像
img1 = cv2.imread('/Users/xionglulu/Downloads/project1/jm1.jpg')
img2 = cv2.imread('/Users/xionglulu/Downloads/project1/jm2.jpg')
# 创建stitcher对象并将两张图像拼接在一起
stitcher = cv2.Stitcher_create()
(status, stitched) = stitcher.stitch((img1, img2))
# 显示拼接后的图像
cv2.namedWindow("stitched image", cv2.WINDOW_NORMAL)
cv2.imshow("stitched image", stitched)
cv2.waitKey()
cv2.destroyAllWindows()result:
Result analysis: The panorama obtained by this method is almost perfect, the image is not deformed, there is no obvious stitching seam, and the color transition is uniform, but there is a problem, it seems that it cannot stitch all the specified images, and the difference in shooting angle There will be errors in the operation of small images, and the specific reasons have yet to be verified.