Chapter 15: Template Matching

Template matching refers to 图像Amatching 图像Bthe most similar part within the current, generally will 图像Abe called 输入图像, will be 图像Bcalled 模板图像. The template matching method is to slide the template image B on the image A, and traverse all pixels one by one to complete the matching.

For example, in the image below, the large image "lena" is the input image, and the "eye" image is the template image. The search method is to slide the template image from the upper left corner in the input image, and traverse the entire input image pixel by pixel to find the part that best matches it.

1. Template matching basis:

1. cv2.matchTemplate() function:

In OpenCV, template matching is implemented through the function cv2.matchTemplate(). The syntax format is:

result=cv2.matchTemplate(image,templ,method[,mask])
- image: The original image, which must be an 8-bit or 32-bit floating-point image.
- templ: It is the template image. It must be smaller or equal in size and of the same type as the original image.
- method: It is the matching method. This parameter is implemented by TemplateMatchModes, and there are 6 possible values, as shown in the table.
  
  Its specific corresponding calculation formula:
- mask: Mask for the template image. It must be the same type and size as the template image templ. Usually this value can use the default value. Currently, this parameter only supports two values of TM_SQDIFF and TM_CCORR_NORMED.

The return value result of the function cv2.matchTemplate() is a result set. The type is a single-channel 32-bit float. is made up of the comparison results for each position.

2. Matching principle:

If the size of the input image (original image) is W * H and the size of the template is w * h, the size of the returned value is (W-w+1)*(H-h+1).

When doing template matching, templates are traversed within the original image. In the horizontal direction:

The starting coordinate of the traversal is the first pixel value from the left of the original image (the serial number starts from 1).
The last comparison is when the template image is located at the far right of the original image, at this time the position of the pixel in the upper left corner is W-w+1.

Therefore, the size of the return value result in the horizontal direction is W-w+1 (the number of comparisons in the horizontal direction).

In the vertical direction:

The starting coordinates of the traversal start from the first pixel at the top of the original image.
The last comparison is when the template image is at the bottom of the original image, and the pixel at the upper left corner is at H-h+1.

Therefore, the size of the return value result in the vertical direction is H-h+1 (the number of comparisons in the vertical direction).

If the size of the original image is W * H and the size of the template is w * h, the size of the returned value is (W-w+1)*(H-h+1). That is, the template image is compared (W-w+1)*(H-h+1) times within the input image.

For example, in the image above, the small 2×2 square in the upper left is the template image, and the 10×10 image in the lower right is the input image (original image). When doing template matching:

The template image is first placed in the upper left corner of the input image.
When the template image moves to the right, it can only be located at the farthest right border of the input image. At this time, the pixel in the upper left corner of the template image corresponds to the ninth column of the input image (input image width-template image width+1=10 -2+1=9).
When moving down, the template image can only be located at the bottom boundary of the input image. At this time, the pixel in the upper left corner of the template image corresponds to the 9th row of the input image (input image height - template image height + 1 = 10 - 2 + 1 = 9).

According to the above analysis, the size of the comparison result result satisfies (W-w+1)*(H-h+1), which is (10-2+1)×(10-2+1) in the above example, that is, 9 ×9. That is to say, the template image needs to be compared 9×9=81 times in total in the input image, and these comparison results will form a 9×9 two-dimensional array.

It should be noted that the function cv2.matchTemplate() decides to use different search methods through the parameter method. For different search methods, the return value result has different meanings. For example:

When the value of method is cv2.TM_SQDIFF and cv2.TM_SQDIFF_NORMED, the result value is 0, indicating the best matching degree, and the larger the value, the worse matching degree.
When the value of method is cv2.TM_CCORR, cv2.TM_CCORR_NORMED, cv2.TM_CCOEFF, and cv2.TM_CCOEFF_NORMED, the smaller the result value, the worse the matching degree, and the larger the result value, the better the matching degree.

The search methods are different, and the results are judged in different ways. When finding the best match, first determine which method is used, and then determine whether to find the maximum value or the minimum value.

3. Find the maximum value:

To find the maximum value (extreme value) and the position of the maximum value, you can use the cv2.minMaxLoc() function. The syntax format is as follows:

minVal,maxVal,minLoc,maxLoc=cv2.minMaxLoc(src[,mask])
- src: a single-channel array.
- minVal: The minimum value returned, if there is no minimum value, it can be NULL (empty value).
- maxVal: the maximum value to return, or NULL if there is no minimum value.
- minLoc: The location of the maximum value, or NULL if there is no maximum value.
- maxLoc: The location of the maximum value, or NULL if there is no maximum value.
- mask: the subset used to select the mask, optional

The function cv2.minMaxLoc() can find the maximum value and their position in the whole array, and can select the extreme value of a specific subset according to the current mask set. For more descriptions and examples of this function, please refer to Chapter 12.

To sum up, the most valued position in the return value of the function cv2.matchTemplate() is the position of the template match.

For example, when the value of method is cv2.TM_SQDIFF and cv2.TM_SQDIFF_NORMED, 0 means the best match, and the larger the value, the worse the matching effect. When using both methods, the best match is found where the minimum is located. The following statement can find the position of the minimum value in the return value of the cv2.matchTemplate() function:

minVal, maxVal, minLoc, maxLoc=cv2.minMaxLoc (return value of matchTemplate function)

topLeft=minLoc # find the location of the minimum value

Take the topLeft point as the coordinates of the upper left corner of the template matching position, and combine the width w and height h of the template image to determine the coordinates of the lower right corner of the matching position. The code is as follows:

bottomRight=(topLeft[0]+w,topLeft[1]+h) #w and h are the width and height of the template image

When the value of method is cv2.TM_CCORR, cv2.TM_CCORR_NORMED, cv2.TM_CCOEFF and cv2.TM_CCOEFF_NORMED, the smaller the return value of the cv2.matchTemplate() function, the worse the matching degree is, and the larger the return value is, the better the matching degree is. good. At this point, it is necessary to find the position where the maximum value is located as the best match.

Through the above method, we have determined the diagonal coordinate position of the template matching rectangle, and then we can use the function cv2.rectangle() to mark the position with white.

The syntax format of the function cv2.rectangle is:

Img=cv.rectangle(img,pt1,pt2,color[,thickness])
- img: Indicates the target image to be marked.
- pt1: is the vertex of the rectangle.
- pt2: is the diagonal vertex of pt1.
- color: is the color or grayscale (grayscale image) to draw the rectangle with.
- thickness: is the width of the rectangle edge.

Therefore, the markup statement used is: cv2.rectangle(img,topLeft,bottomRight,255,2)

import cv2
import numpy as np
import matplotlib.pyplot as plt

img = cv2.imread('../lena.bmp')
template = cv2.imread('../template.bmp')

th, tw = template.shape[:2]
rv = cv2.matchTemplate(img, template, cv2.TM_SQDIFF)
min_val, max_val, min_loc, max_loc = cv2.minMaxLoc(rv)

top_left = min_loc
bottom_right = (top_left[0] + tw, top_left[1] + th)
new_img = img.copy()
cv2.rectangle(new_img, top_left, bottom_right, 255, 2)

plt.subplot(131)
plt.imshow(template, cmap='gray')
plt.title('template')
plt.axis('off')

plt.subplot(132)
plt.imshow(rv, cmap='gray')
plt.title('matcing result')
plt.axis('off')

plt.subplot(133)
plt.imshow(new_img, cmap='gray')
plt.title('result')
plt.axis('off')

plt.show()

2. Multi-template matching:

In the previous example, we searched for the eye submap in the input image lena, which occurs only once in the entire input image. However, in some cases, the template image to be searched is likely to appear multiple times in the input image, and then it is necessary to find multiple matching results. The function cv2.minMaxLoc() can only find the maximum value, but cannot give the location information of all matching areas. Therefore, if you want to match multiple results, it is impossible to use the function cv2.minMaxLoc(), and you need to use the threshold for processing to obtain all matching sets.

1. Get the set of matching locations:

The function where() in the numpy module can get a collection of template matching positions. For different inputs, the returned value is different.

When the input (parameter) is a one-dimensional array, the return value is a one-dimensional index, and there is only one set of index arrays.
When the input is a two-dimensional array, the return is the position index of the matching value, so there will be two sets of index arrays indicating the position of the return value.

For example:

# 当输入数组是一维时
import numpy as np

a=np.array([3,6,8,1,2,88])
b=np.where(a>5)
print(b)

# 输出结果
(array([1,2,5],dtype=int64),)

# 当输入数组是二维时
import numpy as np

am=np.array([[3,6,8,77,66],[1,2,88,3,98],[11,2,67,5,2]])
b=np.where(am>5)
print(b)

# 输出结果
(array([0,0,0,0,1,1,2,2],dtype=int64),
array([1,2,3,4,2,4,0,2],dtype=int64))

To sum up, the function np.where() can find out which positions in the return value of the function cv2.matchTemplate() are greater than the threshold threshold.

In specific implementation, the statement that can be used is:

loc=np.where(res >=threshold)
- res: is the return value of the function cv2.matchTemplate() after template matching.
- threshold: is the preset threshold
- loc: It is an index set of pixels satisfying "res >=threshold".

2. Loop:

Handling multiple values usually requires the use of loops. Therefore, after obtaining the index set of matching values, you can use the following statement to traverse all matching positions and mark these positions:

for i in matches a collection of positions:

marks the matching position.

When looping through matching positions, you can use the function zip() in the loop:

The function zip() takes an iterable object as a parameter, packs the corresponding elements in the object into tuples, and then returns a list composed of these tuples.
For example:

import numpy as np

am = np.random.randint(0, 50, size=(3, 5))
print(am)
b = np.where(am > 20)
print(b)
for i in zip(*b):
    print(i)

# 输出结果
[[37 39 43 48 49]
 [16 48 37 23 25]
 [44 40  0 44 38]]
(array([0, 0, 0, 0, 0, 1, 1, 1, 1, 2, 2, 2, 2], dtype=int64), array([0, 1, 2, 3, 4, 1, 2, 3, 4, 0, 1, 3, 4], dtype=int64))
(0, 0)
(0, 1)
(0, 2)
(0, 3)
(0, 4)
(1, 1)
(1, 2)
(1, 3)
(1, 4)
(2, 0)
(2, 1)
(2, 3)
(2, 4)

Therefore, if you want to loop through the template matching index set returned by np.where(), you can use the following statement:

for i in zip(*template matching index collection):

Mark processing

3. Adjust coordinates

The function numpy.where() can obtain a set of template matching positions that meet the conditions, and then use the function cv2.rectangle() to draw a rectangle at the above matching positions to mark the matching positions.

Use the function numpy.where() to find the specified value in the output value of the function cv2.matchTemplate(), and the obtained position index is in the form of "(row number, column number)". However, the parameter used to specify the vertex in the function cv2.rectangle() uses a position index of the form "(column number, row number)". ** Therefore, before using the function cv2.rectangle() to draw a rectangle, the position index obtained by the function numpy.where() must be "exchanged by row and column". The following statement can be used to exchange the positions of rows and columns in loc:

loc[::-1]

4. Mark the location of the matching image

The function cv2.rectangle() can mark the specific position of the matching image, and specify the original image to be marked, the diagonal vertices, the color, and the width of the rectangle edge.

With respect to the diagonal vertices of a rectangle:

One of the diagonal vertices A can be obtained from the determined "matching position set" that satisfies the condition through a for loop statement.
Another diagonal vertex can be obtained by calculating the position of vertex A and the width (w) and height (h) of the template.
Thus, the statement that marks each matching location is:
- for i in matches the set of locations:
  
  cv2.rectangle(input image, i, (i[0] + w, i[1] + h ), 255, 2)

import cv2
import numpy as np
import matplotlib.pyplot as plt

img = cv2.imread('../four_lena.bmp')
template = cv2.imread('../template.bmp')

h, w = template.shape[:2]
rst = cv2.matchTemplate(img, template, cv2.TM_CCORR_NORMED)
print(rst)
threshold = 0.99
loc = np.where(rst >= threshold)
new_img = img.copy()
for pt in zip(*loc):
    cv2.rectangle(new_img, pt, (pt[0] + w, pt[1] + h), 255, 1)

plt.subplot(121)
plt.imshow(template, cmap='gray')
plt.title('template')
plt.axis('off')

plt.subplot(122)
plt.imshow(new_img, cmap='gray')
plt.title('rst')
plt.axis('off')

plt.show()