Template matching with OpenCV

1 Introduction

Today we are going to study an indispensable method of object detection and tracking in the field of traditional image processing - template matching, whose main purpose is to find the patterns we need on the image, which sounds very exciting.

So, without further ado, let's get right to it!

2. Concept

The core of the template matching algorithm is simple: it compares the template to each part in the source image, sliding it pixel by pixel. The result is a similarity map where each pixel value reflects how similar the template is to that location in the source image.

Essentially, it convolves the template over the image, similar to the way convolution kernels are used in convolutional neural networks. Through this process, a new image or matrix is ​​created where each pixel value represents the similarity between the template and the corresponding region in the source image. By analyzing this resulting image, we can identify peaks that indicate the precise location in the source image where the template image exists. It is worth noting that the implementation of template matching may be different, mainly because the similarity-based measurement varies from method to method, so we will not expand it here.

3. Take a chestnut

Balabala talked about a bunch of conceptual texts, and many friends will feel boring. Based on this, let's look at our example. First, we introduce the basic library we need, as follows:

# Import libraries
import numpy as np
import matplotlib.pyplot as plt
from skimage.io import imread, imshow
from skimage.color import rgb2gray
from skimage.feature import match_template
from skimage.feature import peak_local_max

Next, let's observe our use case image, the code is as follows:

original_image = imread('emojis.png')
plt.figure(figsize=(20,20))
plt.imshow(original_image)
plt.title('Original Image', fontsize=20, weight='bold')
plt.axis('off')
plt.show()

The displayed image is as follows:

Assume that our task is arranged to go through the basic image processing process and find the heartbeat template we need from the above image. After figuring out the specific requirements, let's start coding directly!

4. Image grayscale

While template matching works for color images, let's simplify and convert the image to grayscale to reduce computation.
The grayscale code is as follows:

# Convert the image to grayscale
gray_image = rgb2gray(original_image[:,:,:3])
plt.figure(figsize=(20,20))
plt.imshow(gray_image, cmap='gray')
plt.title('Grayscale Image', fontsize=20, weight='bold')
plt.axis('off')
plt.show()

The result is as follows:

5. Load the template

Now, let's capture a heart-beating expression from the grayscale image as our target template, the code is as follows:

template = gray_image[1330:1850,625:1140]
plt.figure(figsize=(10,10))
plt.imshow(template, cmap='gray')
plt.title('Template Image', fontsize=20, weight='bold')
plt.axis('off')
plt.show();

The result is as follows:

6 template matching

By using the functions skimagein the library match_template, we can get a heat map that measures the similarity between the template image and the original image, as follows:

result = match_template(gray_image, template)
plt.figure(figsize=(10,10))
imshow(result, cmap='viridis')
plt.show();

The results are as follows:

the more brightly colored areas in the above image show the areas with higher similarity to our template. Did you notice the shape formed by the brightly colored areas in the image? If we assume that the template is found only once in the source image, then we can find its location by looking for the pixel with the highest value (~1.00). code show as below:

x, y = np.unravel_index(np.argmax(result), result.shape)
imshow(gray_image)
template_width, template_height = template.shape
rect = plt.Rectangle((y, x), template_height, template_width, color='y', 
                     fc='none')
plt.gca().add_patch(rect);

The result is as follows:

7. Set tolerance

In order to locate multiple matches of the template, we can achieve it by setting the tolerance of the peak of the correlation value, the code is as follows:

imshow(gray_image)
template_width, template_height = template.shape
for x, y in peak_local_max(result, threshold_abs=0.99):
    rect = plt.Rectangle((y, x), template_height, template_width, color='red', 
                         fc='none')
    plt.gca().add_patch(rect);

The result is as follows:

Then you can use the following code to draw the result to the original image, as shown below:

plt.figure(figsize=(20, 20))
plt.imshow(original_image)
plt.title('We found our heart eyes emojis!', fontsize=20, weight='bold', color='red')
template_width, template_height = template.shape
for x, y in peak_local_max(result, threshold_abs=0.99):
    rect = plt.Rectangle((y, x), template_height, template_width, color='red', 
                         fc='none')
    plt.gca().add_patch(rect);

The final result is as follows:

8. Problem thinking

• What happens if we change the threshold? Lowering the threshold will give us more matches (but also more false positives), while raising the threshold will give us fewer matches, but potentially more accurate.
• How about zooming in on templates? The larger the template, the fewer matches we get. This is because the size of the match must be almost the same size as the template.
• Flip template horizontally? This may result in no matches, since template matching is sensitive to direction.
• Change image contrast? As long as the template and the original image are changed in the same way, the match should remain valid. However, drastic changes could alter the results.

9. Summary

This article focuses on how to use the template matching method in traditional image processing to find samples of heart-beating emoticon templates from emoticon pack images, and gives the corresponding code implementation. Because it is a traditional scheme, the threshold selection and generalization ability of this method have certain limitations, but learning the principles behind it can help us better understand related theoretical concepts.