Panoramic image stitching

Manual realization of panoramic image stitching
Environment: python3.6 + opencv3.4.2.16

## Sample picture

The image stitching materials used in this experiment are the following three images:

https://andreame.com/2019/11/12/stitch.html

The goal of this experiment is to project these three images on a cylindrical surface and perform panoramic stitching.

Opencv built-in implementation

First of all, opencv has built-in the stitch class, which packs the methods that can be used for panoramic image stitching.
The basic use is quite simple, and the core code is implemented in two lines

import cv2
from cv2 import Stitcher
def stitcher(images)    
    # images是待拼接图像的list
    Stitcher = cv2.Stitcher.create(cv2.Stitcher_PANORAMA)
    (status, pano) = Stitcher.stitch(images)
    return pano

Python

Copy

And the default stitching effect is pretty good

So just to realize the completion of the image stitching function, the above two lines of code can already be solved.

analysis

Of course, it is not enough to just learn to call. It is best to try to implement it manually. Unfortunately, opencv is the underlying cpp library that is called, not the python implementation. The idea of directly modifying the source code cannot be implemented. The pipeline of the encapsulated stitch class as follows

The opencv documentation points out that this class mainly contains the following modules:

Features Finding and Images Matching
Rotation Estimation
Autocalibration
Images Warping
Seam Estimation
Exposure Compensation
Image Blenders

Some splicing processes are slow, and for ordinary panoramic image splicing, such cumbersome steps are not required, so we can design the module composition by ourselves.

Design and implementation

The code implementation mainly refers to this blog and this tutorial. The
main steps are as follows:

Cylindrical projection transformation of image
Calculate the feature points and description points of the image to be stitched
Calculate the feature description position distance between images
Screen the best feature points
Use RANSAC to calculate the homography matrix
Warping
Seam fusion
Stitching
Image cropping

Cylindrical projection

Schematic can refer to blog 1

Simply put, projecting a certain pixel (x, y) on the original image onto a cylindrical surface with focal length f

\begin{array}{l}{\text { 1. } \theta=\arctan \left(\frac{x}{f}\right)} \\ {\text { 2. } x^{\prime}=f \times \theta} \\ {\text { 3. } \frac{y}{\frac{f}{\cos (\theta)}}=\frac{y^{\prime}}{f}}\end{array} 1. θ=arctan(fx) 2. x′=f×θ 3. cos(θ)fy=fy′

Through the above formula, we can achieve the change from $(x,y)$ to $(x^{\prime},y^{\prime})$, the code is as follows:

def cylindricalProjection(img) :
    rows = img.shape[0]
    cols = img.shape[1]

    #f = cols / (2 * math.tan(np.pi / 8))
    result = np.zeros_like(img)
    center_x = int(cols / 2)
    center_y = int(rows / 2)
    alpha = math.pi / 4
    f = cols / (2 * math.tan(alpha / 2))
    for  y in range(rows):
        for x in range(cols):
            theta = math.atan((x- center_x )/ f)
            point_x = int(f * math.tan( (x-center_x) / f) + center_x)
            point_y = int( (y-center_y) / math.cos(theta) + center_y)

            if point_x >= cols or point_x < 0 or point_y >= rows or point_y < 0:
                pass
            else:
                result[y , x, :] = img[point_y , point_x ,:]
    return result

Python

Copy

The result of the transformation is as follows

The picture on the right is the result of the cylindrical projection of the picture on the left.

Of course, in order to ensure that the black border does not affect the effect of image stitching, we also need to remove the black border in advance. How to remove the black border, we will mention later.

Feature point extraction

Feature point extraction
Feature point extraction can be implemented using opencv built-in functions detectAndCompute().
It should be noted that in versions above opencv-python4, due to copyright issues, this function is no longer open, and the version needs to be returned to opencv3. The environment used by the author is

python3.6
opencv-contrib-python 3.4.2.16
opencv-python 3.4.2.16

The function call is as follows:

def getSURFFeatures(im)
        gray = cv2.cvtColor(im, cv2.COLOR_BGR2GRAY)
        surf = cv2.xfeatures2d.SURF_create()
        kp, des = surf.detectAndCompute(gray, None)
        
        #sift = cv2.xfeatures2d.SIFT_create()
        #kp, des = sift.detectAndCompute(im, None)
        
        cv2.imwrite('imageWithKeypoints.png',cv2.drawKeypoints(im,kp,None))
        return {'kp':kp, 'des':des}

Python

Copy

Here you can use the sift method or the surf method built in opencv. The following figure shows these feature points in Figure 3

Feature point distance calculation

After we get the feature points of the two pictures, the next step we need to do is to find the correspondence between the feature points of the two pictures. For stitching two pictures into a large image, we need to find the coincidence point between them. Using these coincident points, it is possible to determine how to rotate, zoom, and deform the first image so that it can be stitched together with the next image. This is to find a correspondence.
Here you can use the two methods provided by opencv to match the image FLANN or BFMatcher method.

The following is the specific implementation code:

#         matches = self.flann.knnMatch(
#             imageSet2['des'],
#             imageSet1['des'],
#             k=2
#         )
        
        match = cv2.BFMatcher()
        matches = match.knnMatch(
            imageSet2['des'],
            imageSet1['des'],
            k=2
        )

Python

Copy

Where k=2, this parameter means to let matches give the two best matches.

 draw_params = dict(matchColor = (0,255,0), # draw matches in green color
        singlePointColor = None,
        flags = 2)
img3 = cv2.drawMatchesKnn(i1,imageSet1['kp'],i2,imageSet2['kp'],matches,None,**draw_params)
cv2.imwrite("correspondences.png", img3)
print(len(matches))

Python

Copy

Draw the effect, the following figure shows the feature point matching between the two images.

Obviously, there are too many feature points, the number of feature points is 615

Filter the best matching feature points

Usually in an image, feature points exist in multiple locations in the image, so further screening is required.

        good = []
        for i, (m, n) in enumerate(matches):
            if m.distance < 0.3*n.distance:
                good.append((m.trainIdx, m.queryIdx))

Python

Copy

In this case, we get the best match between the two images, and the next step is to calculate the homography matrix.

When this ratio is set to 0.3, the filtering effect is shown in the figure below

At this time, the number of feature points is 39

Calculation of homography matrix

The function of the homography matrix is to estimate the relative direction transformation of the two images based on the best matching point. Here you can use the built-in RANSAC method of cv to solve:

H, s = cv2.findHomography(matchedPointsCurrent, matchedPointsPrev, cv2.RANSAC, 4)

Python

Copy

The obtained $H$ is the homography matrix to be solved.

I_x = H \times I_yIx=H×Iy

The resulting homography matrix is as follows

\begin{bmatrix} h_\text{11} & h_\text{12} & h_\text{13} \\ h_\text{21} & h_\text{22} & h_\text{23} \\ h_\text{31} & h_\text{32} & h_\text{33} \end{bmatrix}⎣⎡h11h21h31h12h22h32h13h23h33⎦⎤

Wrap

In the previous step, we got a homography matrix, and the next step is to stitch the image.

First, we get the homography matrix between the two, we can understand what the second image looks like from the perspective of the first image. We need to change the image and transform it into a new space. Then a process of warping transformation is needed next.

Warpage transformation includes

Plane transformation
Cylindrical transformation
Spherical transformation

We can simply call the following function to complete.

warped_image = cv2.warpPerspective(image, homography_matrix, dimension_of_warped_image)

Python

Copy

In the above function, image represents the image to be transformed, homography_matrix is the homography matrix we obtained, and dimension_of_warped_image is the size of the image after transformation.

Therefore, in order to make the warpage transformation, we also need to obtain the size after the warpage transformation to prepare for the splicing.

H = self.matcher_obj.match(a, b, 'left')  
xh = np.linalg.inv(H) 
ds = np.dot(xh, np.array([a.shape[1], a.shape[0], 1])) 
ds = ds / ds[-1]  #
f1 = np.dot(xh, np.array([0, 0, 1]))
f1 = f1 / f1[-1]
xh[0][-1] += abs(f1[0])  
xh[1][-1] += abs(f1[1])  #
ds = np.dot(xh, np.array([a.shape[1], a.shape[0], 1]))  
offsety = abs(int(f1[1]))  
offsetx = abs(int(f1[0]))  
dsize = (int(ds[0]) + offsetx, int(ds[1]) + offsety)

Python

Copy

The final result can be obtained after warping transformation.

We have a homography matrix $H$. If the starting coordinate of each picture is $(0,0)$ and the ending coordinate is $(r_e,c_e)$, then we can get the image size after warping through this change:

Starting point calculation $Point_{strat} := H \times (0,0)$ until the end point calculation $point_{end} := H \times (re,c_e)$.

Of course, the next step of splicing could simply tmp[offsety:b.shape[0]+offsety, offsetx:b.shape[1]+offsetx] = bbe done, but this will make the seams so obvious that we also need to be adjusted.

The effect is as follows:

You can see very obvious seams. Obviously, such a simple treatment cannot achieve the desired effect.

Seam fusion

So how to eliminate the seam, here is the method of weighted fusion.

The idea is to calculate the overlapped part of the two images. For the overlapped part, use the method of gradual transition of pixels to eliminate the abrupt changes that can be observed by the naked eye.

Then you can consider using weighted fusion to solve the problem, that is, let the pixels of the splicing part be the weighted sum of the pixels of the two images with the same coordinates:

point_{result} = \alpha \times point_{a} + (1 - \alpha) \times point_{b}pointresult=α×pointa+(1−α)×pointb

How to determine the $\alpha$ in the formula, here is the distance to calculate the weight, the image from the coincidence boundary to the image boundary, this weight smoothly from 1 to 0.

    def seamEstimation(self,tmp,leftimage,rightimage,offsetx,offsety):
        alpha = 0.0
        processwidth = leftimage.shape[1] - offsetx
        print("initial_width",processwidth)
        for x in range(offsetx,tmp.shape[1]):
            test_y = int((offsety + leftimage.shape[0])/2)
            if np.array_equal(tmp[test_y,x],np.array([0,0,0])):
                #print(tmp[test_y,x])
                processwidth = x - offsetx
                break
        print("now_width",processwidth)
        x_max = np.minimum(offsetx+rightimage.shape[1],tmp.shape[1])
        y_max = np.minimum(offsety+rightimage.shape[0],tmp.shape[0])
        for x in range(offsetx,x_max):
            for y in range(offsety,y_max):
                rx = x - offsetx
                ry = y - offsety
                if not np.array_equal(tmp[y,x],np.array([0,0,0])) and not np.array_equal(rightimage[ry,rx],np.array([0,0,0])):
                    alpha = np.minimum(rx/processwidth,1.0)
                    #alpha = rx/processwidth
                    tmp[y,x] = alpha*rightimage[ry,rx]+(1-alpha)*tmp[y,x]
                elif np.array_equal(tmp[y,x],np.array([0,0,0])):
                    tmp[y,x] = rightimage[ry,rx]
                elif np.array_equal(rightimage[ry,rx],np.array([0,0,0])):
                    tmp[y,x] = tmp[y,x]
                else:
                    print("Some problems happen!")
        return tmp

Python

Copy

Then multiple images are stitched continuously to get a stitched image:

The overall effect is okay, but the obvious splicing parts have ghost images. Of course, this is an inevitable result caused by pixel fusion, which needs to be improved later.

MultipleImagesStitching

Through the above description, we have completed the basic work of image stitching, then we only need to repeat the above two stitching work continuously.

It should be noted that if a plane change is adopted, a picture in the middle needs to be used as the baseline for left and right stitching to avoid excessive deviation. And when the number of pictures is too large, it needs to be spliced in batches.

However, cylindrical splicing is used here, and sequential splicing will not cause too serious consequences. The splicing code is as follows:

    def shift(self):  
        a = self.images[0]  
        for b in self.images[1:]: 
            H = self.matcher_obj.match(a, b, 'left')  # 特征点匹配
            xh = np.linalg.inv(H) 
            ds = np.dot(xh, np.array([a.shape[1], a.shape[0], 1])) 
            ds = ds / ds[-1]  #
            f1 = np.dot(xh, np.array([0, 0, 1])) 
            f1 = f1 / f1[-1]
            xh[0][-1] += abs(f1[0])  
            xh[1][-1] += abs(f1[1])  #
            ds = np.dot(xh, np.array([a.shape[1], a.shape[0], 1])) 
            offsety = abs(int(f1[1]))  # y偏移量  需要了解单应性矩阵的作用
            offsetx = abs(int(f1[0]))  # x偏移量
            dsize = (int(ds[0]) + offsetx, int(ds[1]) + offsety)  # 图片大小统计
            tmp = cv2.warpPerspective(a, xh, dsize) 
            tmp = self.seamEstimation(tmp,a,b,offsetx,offsety)
            a = tmp  # 为循环做准备
        return a

Python

Copy

Image cropping

Of course, the image we get at this time still has a large number of black frames, and we need to crop it through image manipulation.

Convert image to grayscale

    img = cv2.medianBlur(image,3) #中值滤波
    b=cv2.threshold(img,0,255,cv2.THRESH_BINARY)  
    binary_image=b[1]               #二值图--具有三通道
    binary_image=cv2.cvtColor(binary_image,cv2.COLOR_BGR2GRAY)

Python

Copy

Here simply use image operations to convert the image into a binary image.

In addition, we need to continue to outreach 10 pixels to ensure that all his borders are exposed to serve the next step.

binary_image

Calculate the maximum contour

ret, thresh = cv2.threshold(binary_image, 0, 255, cv2.THRESH_BINARY)    
binary ,cnts, hierarchy = cv2.findContours(thresh.copy(), cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
cnt = max(cnts, key=cv2.contourArea)

Python

Copy

Use opencv's built-in function to calculate the maximum contour of this image.

Draw the largest bounding rectangle

mask = np.zeros(thresh.shape, dtype="uint8")
x, y, w, h = cv2.boundingRect(cnt)

Python

Copy

Continuous corrosion operation

We set the initial bounding box on the largest circumscribed rectangular box, and then use this as a starting point to continuously perform clothing operations to reduce the area of the selected box, knowing that our selected box does not contain the background pixel position, so we can find The bounding box we need. $(x,y,w,h)$

The picture below is the stitched image after cropping.

The following is the code to remove the border:

def ExtractImage(image):
    # 去除边框，提取图像内容
    plt.figure(num='ExtractImage')

    image = cv2.copyMakeBorder(image, 10, 10, 10, 10, cv2.BORDER_CONSTANT, (0, 0, 0))
    binary_image = getBinaryImage(image)
    cv2.imwrite("binary_image.png", binary_image)

    ret, thresh = cv2.threshold(binary_image, 0, 255, cv2.THRESH_BINARY)    
    binary ,cnts, hierarchy = cv2.findContours(thresh.copy(), cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
    cnt = max(cnts, key=cv2.contourArea)  # 获取最大轮廓

    mask = np.zeros(thresh.shape, dtype="uint8")
    x, y, w, h = cv2.boundingRect(cnt)
    # 绘制最大外接矩形框（内部填充）
    cv2.rectangle(mask, (x, y), (x + w, y + h), 255, -1)
    cv2.imwrite("mask.png", mask)

    minRect = mask.copy()
    sub = mask.copy()
    print(sub.shape[0] * sub.shape[1])
    # 连续腐蚀操作，直到sub中不再有前景像素
    while cv2.countNonZero(sub) > 0:
        minRect = cv2.erode(minRect, None)
        sub = cv2.subtract(minRect, thresh)

    binary ,cnts, hierarchy = cv2.findContours(minRect.copy(), cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
    cnt = max(cnts, key=cv2.contourArea)
    x, y, w, h = cv2.boundingRect(cnt)

    image = image[y:y + h, x:x + w]
    return image

Python

Copy

Hole filling

In the previous step, we set the cycle condition of the corrosion operation to "no background pixels in the candidate frame", then how to judge the background pixels? It can be simply considered that the pixel value is 0 as the background pixel. However, if the original image or other operations will cause pixels with a pixel value of 0 to appear in the image content, such as:

After the two images are spliced, the resulting image will be converted into a binary image. There are holes in the image, and the upper half will be subtracted if it is cropped rashly.

Direct cropping of such an image will lead to the following results:

Therefore, advanced hole filling is needed to ensure that there are no holes in the image before the next step of cropping can be performed.
After filling, the cropping result:

Final Results

Before splicing

After stitching

Zoom

Time cost :5.6752543449401855

references

Cylindrical projection: https://blog.csdn.net/zwx1995zwx/article/details/81005454
Personal blog: https://www.andreame.com/2019/11/09/stitch.html
https://kushalvyas.github.io/stitching.html
https://medium.com/pylessons/image-stitching-with-opencv-and-python-1ebd9e0a6d78
"opencv文档"：https://docs.opencv.org/3.4.2/d1/d46/group__stitching.html

Python panoramic image stitching (python3.6 + opencv3.4.2.16) fusion

Panoramic image stitching

Opencv built-in implementation

analysis

Design and implementation

Cylindrical projection

Feature point extraction

Feature point distance calculation

Filter the best matching feature points

Calculation of homography matrix

Wrap

Seam fusion

MultipleImagesStitching

Image cropping

Final Results

references

Guess you like