Image local descriptor: SIFT descriptor and Harris corner detection

introduction

Harris corner detection algorithm is one of the simplest corner detection methods, so there are many drawbacks. With the continuous research of scientific researchers and put forward the SIFT method, it can better describe the image features on the original basis, and can match the descriptors more accurately. However, if you want to achieve image feature matching in the most ideal state, the SIFT algorithm alone is not enough. We still need to optimize and innovate algorithms through continuous scientific research to finally achieve more efficient and accurate image feature point matching.

1. Corner detection

First, let's look at Harris corner detection. The Harris corner detector can give the detected points of interest in the image. The code and results are as follows:

from pylab import *
from PIL import Image
from PCV.localdescriptors import harris

# 读入图像
im = array(Image.open(r'C:\Users\18042\Desktop\img1.jpg').convert('L'))

# 检测harris角点
harrisim = harris.compute_harris_response(im)

# Harris响应函数
harrisim1 = 255 - harrisim

figure()
gray()

#画出Harris响应图
subplot(121)
imshow(im)
axis('off')
axis('equal')

threshold = [0.01, 0.05, 0.1]
for i, thres in enumerate(threshold):
    filtered_coords = harris.get_harris_points(harrisim, 6, thres)

subplot(1, 2, 2)
imshow(im)
print (im.shape)
plot([p[1] for p in filtered_coords], [p[0] for p in filtered_coords], '*')
axis('off')

show()

Original image: img1.jpg

Judging from the inspection results alone, it seems to be pretty good, and most of the corner points are basically recognized. But if we compare the result with the SIFT feature, we can see that Harris corner detection is still slightly inferior to SIFT:

# -*- coding: utf-8 -*-
from PIL import Image
from pylab import *
from PCV.localdescriptors import sift
from PCV.localdescriptors import harris

# 添加中文字体支持
from matplotlib.font_manager import FontProperties
font = FontProperties(fname=r"c:\windows\fonts\SimSun.ttc", size=14)

imname = r'C:\Users\18042\Desktop\img1.jpg'
im = array(Image.open(imname).convert('L'))
sift.process_image(imname, 'empire.sift')
l1, d1 = sift.read_features_from_file('empire.sift')

figure()
gray()
subplot(131)
sift.plot_features(im, l1, circle=False)
title(u'SIFT特征',fontproperties=font)
subplot(132)
sift.plot_features(im, l1, circle=True)
title(u'用圆圈表示SIFT特征尺度',fontproperties=font)

# 检测harris角点
harrisim = harris.compute_harris_response(im)

subplot(133)
filtered_coords = harris.get_harris_points(harrisim, 6, 0.1)
imshow(im)
plot([p[1] for p in filtered_coords], [p[0] for p in filtered_coords], '*')
axis('off')
title(u'Harris角点',fontproperties=font)

show()

Result analysis

From the results, we can clearly see that the SIFT feature recognition is stronger than the Harris corner points. For example, the SIFT in the middle of the building in the figure can be well recognized but the Harris foot points cannot be recognized. This is precisely because the principle of the algorithm is different, which leads to the different results. The Harris foot point detection is mainly based on the eigenvalues ​​of the Harris matrix in the following three cases as shown in the following figure. The pixel difference is not large in a certain area. The foot point detection cannot be performed well in the case of , and the SIFT feature extracts features by looking for the precise positioning and main direction of extreme points (feature points, key points) in different scale spaces, and constructing key point descriptors. The transformation in different scale spaces solves the problem of unclear recognition of small areas .

2. Feature point matching

First, let's look at the results of matching through Harris feet:

from pylab import *
from PIL import Image

from PCV.localdescriptors import harris
from PCV.tools.imtools import imresize

im1 = array(Image.open(r'C:\Users\18042\Desktop\img1.jpg').convert("L"))
im2 = array(Image.open(r'C:\Users\18042\Desktop\img2.jpg').convert("L"))

im1 = imresize(im1, (im1.shape[1], im1.shape[0]))
im2 = imresize(im2, (im2.shape[1], im2.shape[0]))

wid = 5
harrisim = harris.compute_harris_response(im1, 5)
filtered_coords1 = harris.get_harris_points(harrisim, wid+1)
d1 = harris.get_descriptors(im1, filtered_coords1, wid)

harrisim = harris.compute_harris_response(im2, 5)
filtered_coords2 = harris.get_harris_points(harrisim, wid+1)
d2 = harris.get_descriptors(im2, filtered_coords2, wid)

print ('starting matching')
matches = harris.match_twosided(d1, d2)

figure()
gray() 
harris.plot_matches(im1, im2, filtered_coords1, filtered_coords2, matches)
show()

Harris result analysis

Here we give two similar pictures, but continue to match with a picture taken by moving slightly to the right. There are many points in the picture that can be matched, but we will find that there are some mismatches, such as In the figure, the points in the upper cloud layer match the points below. If you change the picture and match two pictures that are not a scene at all, then we will find the seriousness of the matter, as shown below:

Two irrelevant pictures have so many matching values ​​in Harris footpoint matching. It seems that Harris algorithm is still the eighth Taihang. On the contrary, look at SIFT:

from PIL import Image
from pylab import *
import sys
from PCV.localdescriptors import sift


if len(sys.argv) >= 3:
  im1f, im2f = sys.argv[1], sys.argv[2]
else:
  im1f = r'C:\Users\18042\Desktop\img1.jpg'
  im2f = r'C:\Users\18042\Desktop\img2.jpg'
im1 = array(Image.open(im1f))
im2 = array(Image.open(im2f))

sift.process_image(im1f, 'out_sift_1.txt')
l1, d1 = sift.read_features_from_file('out_sift_1.txt')
figure()
gray()
subplot(121)
sift.plot_features(im1, l1, circle=False)

sift.process_image(im2f, 'out_sift_2.txt')
l2, d2 = sift.read_features_from_file('out_sift_2.txt')
subplot(122)
sift.plot_features(im2, l2, circle=False)

#matches = sift.match(d1, d2)
matches = sift.match_twosided(d1, d2)
print ('{} matches'.format(len(matches.nonzero()[0])))

figure()
gray()
sift.plot_matches(im1, im2, l1, l2, matches, show_below=True)
show()

SIFT result analysis

We can see that there are almost no mismatches for similar pictures, and only a few repetitive features on buildings are mismatched, which is inevitable.

And for such irrelevant pictures, SIFT will not have a lot of inexplicable matching points like Harris.

At the same time, due to the characteristics of SIFT, the key point descriptors are constructed to extract features by finding the precise positioning and main directions of extreme points (feature points, key points) in different scale spaces. The key points extracted by SIFT have scale invariance and rotation invariance, and feature matching can be performed well in the case of large brightness differences , which is unmatched by Harris angle detection.


As shown in the figure below : As shown in the figure above, we can directly see the characteristics of SIFT, and have a good sensitivity to the rotation brightness angle and scale of the picture. But at the same time, I also found some problems during the test

As shown in the figure, the results of SIFT feature matching on pictures from different shooting angles are extremely poor . Only a few feature points can be successfully matched, and even no feature point can be successfully matched. Although SIFT can match two-dimensional picture rotations well, pictures taken from different angles cannot be accurately matched after three-dimensional rotations. Therefore, although SIFT is better than Harris corner detection, there are still problems. The algorithm can also be optimized to obtain a more accurate and powerful feature point matching algorithm!

3. Geotagged image matching

# -*- coding: utf-8 -*-
import json
import os
import urllib
from pylab import *
from PIL import Image
from PCV.localdescriptors import sift
from PCV.tools import imtools
import pydot
import os
os.environ["PATH"] += os.pathsep + 'D:/Graphviz/bin'

if __name__ == '__main__':
    download_path = r"C:\Users\18042\Desktop\32Vision\img"
    path = r"C:\Users\18042\Desktop\32Vision\img"

    imlist = imtools.get_imlist(download_path)
    nbr_images = len(imlist)

    featlist = [imname[:-3] + 'sift' for imname in imlist]
    for i, imname in enumerate(imlist):
        sift.process_image(imname, featlist[i])

    matchscores = zeros((nbr_images, nbr_images))

    for i in range(nbr_images):
        for j in range(i, nbr_images):  # only compute upper triangle
            print('comparing ', imlist[i], imlist[j])
            l1, d1 = sift.read_features_from_file(featlist[i])
            l2, d2 = sift.read_features_from_file(featlist[j])
            matches = sift.match_twosided(d1, d2)
            nbr_matches = sum(matches > 0)
            print('number of matches = ', nbr_matches)
            matchscores[i, j] = nbr_matches

    # copy values
    for i in range(nbr_images):
        for j in range(i + 1, nbr_images):  # no need to copy diagonal
            matchscores[j, i] = matchscores[i, j]

    # 可视化
    threshold = 2  # min number of matches needed to create link

    g = pydot.Dot(graph_type='graph')  # don't want the default directed graph

    for i in range(nbr_images):
        for j in range(i + 1, nbr_images):
            if matchscores[i, j] > threshold:
                # first image in pair
                im = Image.open(imlist[i])
                im.thumbnail((100, 100))
                filename = path + str(i) + '.jpg'
                im.save(filename)  # need temporary files of the right size
                g.add_node(pydot.Node(str(i), fontcolor='transparent', shape='rectangle', image=filename))

                # second image in pair
                im = Image.open(imlist[j])
                im.thumbnail((100, 100))
                filename = path + str(j) + '.jpg'
                im.save(filename)  # need temporary files of the right size
                g.add_node(pydot.Node(str(j), fontcolor='transparent', shape='rectangle', image=filename))

                g.add_edge(pydot.Edge(str(i), str(j)))
    g.write_jpg('outimg.jpg')