prerequisites

Before you start detecting and recognizing faces, you need to set up a development environment. First, you need to "read" the image through Python before doing any processing on it. We will use the plotting library matplotlib to read and manipulate images. Install the latest version of pip through the installer:

pip3 install matplotlib

To use any implementation of the CNN algorithm, you need to install keras. Use the following command to download and install the latest version:

pip3 install keras

Run the following command to install the package pip in the following way:

pip3 install mtcnn

After extracting the face from the image, for comparison, we will use the VGGFace2 algorithm developed by the Oxford University Visual Geometry Group. The TensorFlow-based Keras implementation of the VGG algorithm can be installed as a software package for you to install:

pip3 install keras_vggface

Although you may need to build and train your own model, you need a huge training data set and powerful processing capabilities. Since this tutorial focuses on the practicality of these models, it uses existing trained models provided by experts in the field.

Now that you have successfully installed the prerequisites, let's go directly to this tutorial!

Step 1: Use MTCNN model for face detection

The goals of this step are as follows:

Retrieve images hosted externally on a local server
Read the image through matplotlib's imread() function
Detect and explore faces through the MTCNN algorithm
Extract the face from the image.

Many people learn python and don't know where to start.
Many people learn python and after mastering the basic grammar, they don't know where to find cases to get started.
Many people who have done case studies do not know how to learn more advanced knowledge.
So for these three types of people, I will provide you with a good learning platform, free to receive video tutorials, e-books, and the source code of the course!
QQ group: 705933274

1.1 Store external images

You may often analyze based on images hosted on external servers. In this example, we will use two images, which are hosted on the BBC and The Detroit News websites by Lee Iacocca, the father of the Mustang.

In order to temporarily store the images locally for our analysis, we will retrieve each image from its URL and write it to a local file. For store_image, we define a function:

import urllib.request
def store_image(url, local_file_name):
  with urllib.request.urlopen(url) as resource:
    with open(local_file_name, 'wb') as f:
      f.write(resource.read())

Now you can simply call the function using the URL and the local file where you want to store the image:

store_image('https://ichef.bbci.co.uk/news/320/cpsprodpb/5944/production/_107725822_55fd57ad-c509-4335-a7d2-bcc86e32be72.jpg',
            'iacocca_1.jpg')
store_image('https://www.gannett-cdn.com/presto/2019/07/03/PDTN/205798e7-9555-4245-99e1-fd300c50ce85-AP_080910055617.jpg?width=540&height=&fit=bounds&auto=webp',
            'iacocca_2.jpg')

After successfully retrieving the image, let us detect the faces in it.

1.2 Detect the face in the image

For this, we will perform two imports-matplotlib reads the image and mtcnn detects the face in the image:

from matplotlib import pyplot as plt
from mtcnn.mtcnn import MTCNN

Use the imread() function to read the image:

image = plt.imread('iacocca_1.jpg')

Next, initialize an MTCNN() object as the detector variable, and then use the .detect_faces() method to detect faces in the image. Let's see what it returns:

detector = MTCNN()
faces = detector.detect_faces(image)
for face in faces:
    print(face)

For each face, a Python dictionary is returned with three keys. The box key contains the boundary of the face within the image. It has four values: the x and y coordinates of the top-left vertex, and the width and height of the rectangle containing the face. The other keys are confidence and keypoints. The keypoints key contains a dictionary containing the detected facial features and their coordinates:

{'box': [160, 40, 35, 44], 'confidence': 0.9999798536300659, 'keypoints': {'left_eye': (172, 57), 'right_eye': (188, 57), 'nose': (182, 64), 'mouth_left': (173, 73), 'mouth_right': (187, 73)}}

1.3 Highlight faces in the image

Now that we have successfully detected a face, let us draw a rectangle on it to highlight the face in the image to verify that the detection is correct.

To draw a rectangle, please import the Rectangle object matplotlib.patches from:

from matplotlib.patches import Rectangle

Let us define a function highlight_faces that first displays the image and then draws a rectangle on the detected face. First, read through the image imread() and draw through imshow(). For each face detected, use the Rectangle() class to draw a rectangle.

Finally, use the .show() method to display the image and rectangle. If you are using a Jupyter notebook, you can use the %matplotlib inlinemagic command to display the plot inline:

def highlight_faces(image_path, faces):
  
    image = plt.imread(image_path)
    plt.imshow(image)
    ax = plt.gca()
    
    for face in faces:
        x, y, width, height = face['box']
        face_border = Rectangle((x, y), width, height,
                          fill=False, color='red')
        ax.add_patch(face_border)
    plt.show()

Now, use the highlight_faces() function to display the image and the detected faces:

highlight_faces('iacocca_1.jpg', faces)

A face was detected in Lee Iacocca's image. Source: BBC

Let's show the second image and the faces detected in it:

image = plt.imread('iacocca_2.jpg')
faces = detector.detect_faces(image)
highlight_faces('iacocca_2.jpg', faces)

In these two pictures, you can see that the MTCNN algorithm correctly detects the face. Now, let us extract the face from the image for further analysis.

At this point, you can know the coordinates of the face from the detector. Using the list index to extract faces is a fairly easy task. However, the VGGFace2 algorithm we use needs to adjust the face size to 224 x 224 pixels. We will use the PIL library to resize the image.

The function extract_face_from_image() extracts all faces from the image:

from numpy import asarray
from PIL import Image
def extract_face_from_image(image_path, required_size=(224, 224)):
  
    image = plt.imread(image_path)
    detector = MTCNN()
    faces = detector.detect_faces(image)
    face_images = []
    for face in faces:
        
        x1, y1, width, height = face['box']
        x2, y2 = x1 + width, y1 + height
        
        face_boundary = image[y1:y2, x1:x2]
        
        face_image = Image.fromarray(face_boundary)
        face_image = face_image.resize(required_size)
        face_array = asarray(face_image)
        face_images.append(face_array)
    return face_images
extracted_face = extract_face_from_image('iacocca_1.jpg')
plt.imshow(extracted_face[0])
plt.show()

This is the appearance of the face extracted from the first image.

Extract and resize the face from the first image

Step 2: Use VGGFace2 model for face recognition

In this section, let us first test the model on the two images of Lee Iacocca that we retrieved. Then, we will continue to compare the faces of the Chelsea football team in the images of the top eleven players in 2018 and 2019. You will then be able to evaluate whether the algorithm recognizes the faces of ordinary players between the images.

2.1 Compare two faces

In this section, you need to import three modules: VGGFace prepares the face to be extracted for use in the face recognition model, and cosine's function from SciPy to calculate the distance between two faces:

from keras_vggface.utils import preprocess_input
from keras_vggface.vggface import VGGFace
from scipy.spatial.distance import cosine

Let us define a function that takes the extracted face as input and returns the calculated model score. The model returns a vector that represents facial features:

def get_model_scores(faces):
    samples = asarray(faces, 'float32')
    
    samples = preprocess_input(samples, version=2)
    
    model = VGGFace(model='resnet50',
      include_top=False,
      input_shape=(224, 224, 3),
      pooling='avg')
    
    return model.predict(samples)
faces = [extract_face_from_image(image_path)
         for image_path in ['iacocca_1.jpg', 'iacocca_2.jpg']]
model_scores = get_model_scores(faces)

Since the model score of each face is a vector, we need to find the similarity between the scores of the two faces. We can usually use Euclidean or cosine functions to calculate similarity.

The vector representation of the face is suitable for cosine similarity. This is a detailed comparison of cosine and Euclidean distance and provides an example.

The cosine() function calculates the cosine distance between two vectors. The smaller the number, the better your face will match. In our example, we put the threshold at a distance of 0.4. This threshold is worth discussing and will vary depending on your use case. You should set this threshold based on case studies on the dataset:

if cosine(model_scores[0], model_scores[1]) <= 0.4:
  print("Faces Matched")

In this case, Lee Iacocca's two faces matched.

Faces Matched

2.2 Compare multiple faces in two images

Let's make the most of the model in this part of the tutorial. We will compare faces in two pictures. These two pictures are the UEFA Europa League match against Slavia Prague in the 2018-19 season, Chelsea Football Club’s eleven-year-old start and the 2019-20 season against Liverpool. The UEFA Super Cup game. Although there are many players in the squad on both match days, let's see if the algorithm can detect all ordinary players.

First, let's retrieve the resource from the URL, detect the faces in each image and highlight them:

store_image('https://cdn.vox-cdn.com/thumbor/Ua2BXGAhneJHLQmLvj-ZzILK-Xs=/0x0:4872x3160/1820x1213/filters:focal(1877x860:2655x1638):format(webp)/cdn.vox-cdn.com/uploads/chorus_image/image/63613936/1143553317.jpg.5.jpg',
            'chelsea_1.jpg')
image = plt.imread('chelsea_1.jpg')
faces_staring_xi = detector.detect_faces(image)
highlight_faces('chelsea_1.jpg', faces_staring_xi)
store_image('https://cdn.vox-cdn.com/thumbor/mT3JHQtZIyInU8_uGxVH-TCbF50=/0x415:5000x2794/1820x1213/filters:focal(1878x1176:2678x1976):format(webp)/cdn.vox-cdn.com/uploads/chorus_image/image/65171515/1161847141.jpg.0.jpg',
            'chelsea_2.jpg')
image = plt.imread('chelsea_2.jpg')
faces = detector.detect_faces(image)
highlight_faces('chelsea_2.jpg', faces)

Before proceeding, here are the top 11 of the two games:

Slavia Prague competition begins XI: Kepa, Azpilicueta, Luiz, Christensen, Emerson, Kanter (Kante), Barkley, Kovacic, Danger, Pedro, Giroud
Liverpool’s eleventh game begins: Kopa, Azpilicuta, Christensen, Zuma, Emerson, Kanter, Jorginho, Kovac, Pedro, Giroud, Puli Sic

We have 8 players, which is common for starting XI, and should ideally be matched by an algorithm.

First calculate the score:

slavia_faces = extract_face_from_image('chelsea_1.jpg')
liverpool_faces = extract_face_from_image('chelsea_2.jpg')
model_scores_starting_xi_slavia = get_model_scores(slavia_faces)
model_scores_starting_xi_liverpool = get_model_scores(liverpool_faces)
``
for idx, face_score_1 in enumerate(model_scores_starting_xi_slavia):
  for idy, face_score_2 in enumerate(model_scores_starting_xi_liverpool):
    score = cosine(face_score_1, face_score_2)
    if score <= 0.4:
      
      print(idx, idy, score)
      
      plt.imshow(slavia_faces[idx])
      plt.show()
      plt.imshow(liverpool_faces[idy])
      plt.show()

This is a list of two faces matched by the algorithm. Please note that it has been able to match all eight pairs of faces.

Eight correctly identified faces (Kepa, Azpilicueta, Emerson, Giroud, Kante, Pedro, Christensen, Kovacic)

Although we were able to successfully match every face in the image, I want to take a step back and discuss the impact of scores. As mentioned earlier, there is no universal threshold for matching two images together. You may need to redefine these thresholds with new data that enters the analysis. For example, when the optimal pair of thresholds cannot be determined programmatically, even Google Photos will accept your input.

Google Photos accepts user input for face matching

The best way is to carefully evaluate the case when matching different types of faces. The facial expression and its angle also determine the accuracy. In our use case, please note how I deliberately used the first eleven-year-old photo when the player stared at the camera! You can try to match the first 11 faces with the faces of the trophy celebration, I'm sure the accuracy will drop.

in conclusion

In this tutorial, we first use the MTCNN model to detect human faces in the image and highlight them in the image to determine whether the model is working properly. Next, we use the VGGFace2 algorithm to extract features from faces in the form of vectors, and match different faces to combine them.

I still want to recommend the Python learning Q group I built by myself : 705933274. The group is all learning Python. If you want to learn or are learning Python, you are welcome to join. Everyone is a software development party and share dry goods from time to time ( Only related to Python software development), including a copy of the latest Python advanced materials and zero-based teaching compiled by myself in 2021. Welcome to the advanced and friends interested in Python to join!

Teach you how to use Keras for face detection and recognition