Are you still deceived by the face recognition accuracy index?

Guide

With the improvement of hardware performance and the increase in the amount of face data, face recognition has become more and more mature, and there are more and more commercial applications. I often see many articles saying, what improvements have been made to the face recognition algorithm, the recognition accuracy rate on LFW reaches more than 99.6%.

In fact, only an accuracy index cannot measure the performance of a model. The accuracy rate cannot reflect the pass rate and rejection rate, which are the most important indicators in face recognition. The pass rate includes the probability of the same person passing in two cases and the passing rate of different people. The rejection rate includes the probability of being rejected by different people in two situations and the probability of being rejected by the same person. Usually we hope that the model has a lower pass rate of different people, the higher the pass rate of the same person, the better, the following evaluation indicators It is mainly based on these two evolutions.

Face recognition

Nowadays, the applications of face recognition are diverse, such as face attendance, real-name verification, identity verification, face payment, and Skynet system. In fact, the underlying implementation technology mainly relies on face verification and face search. Face verification is also called 1:1 face matching, and face search is also called 1:N face matching.

First, we extract the position information and key points of the face from the picture containing the face through face detection, and then align the face by combining the key points of the face through affine transformation, and obtain the aligned face picture. Then the face image is extracted through the face recognition model, and the extracted feature is a high-dimensional vector. The dimension of this vector is usually 128, 256, 512, 1024 or even higher.

There are two main indicators for judging whether two face pictures are similar, Euclidean distance and cosine similarity. First, the face image is converted into a feature vector through the face recognition model. As the name suggests, Euclidean distance is to calculate the Euclidean distance of two vectors, so the smaller the Euclidean distance of two vectors, the more similar they are. The cosine similarity is to calculate the cosine value of the angle between two vectors. The value range of cosθ is [−1,1]. We can normalize it to [0,1] by 0.5+0.5∗cosθ Realize, the greater the cosine similarity, the greater the probability of being the same person.

Whether it is 1:1 face matching or 1:N face matching, we need to determine a threshold (Euclidean distance or similarity) first. Usually the measure of face similarity is the similarity, which we will talk about later. The threshold is actually a similarity threshold. Through the face data set, the threshold is determined according to the specified pass rate or rejection rate. If the threshold is greater than this, it means the same person, otherwise it is not the same person.

Face verification

1:1 face verification (verification), which compares whether two pictures are the same person. Common applications include real-name verification of face gates at train stations and face unlocking on mobile phones. By judging whether the similarity of the compared pictures is greater than the threshold, the commonly used performance evaluation indicators are as follows

• FAR

FAR (False Accept Rate) is the false acceptance rate, which is equivalent to the FPR (False Positive Rate) false positive rate. It refers to the number of times that the same person is not the same person but the same person accounts for all the comparisons that are not the same person. ,Calculated as follows

For a detailed description of the confusion matrix (TP, TN, FP, FN), please refer to my other article Common Evaluation Indexes in Classification Algorithms

• TAR

TAR (True Accept Rate) represents the correct acceptance rate, which is equivalent to the true rate of TPR (True Positive Rate). It refers to the number of times that the same person is correctly considered to be the same person in all comparisons of the same person. The calculation formula is as follows

• FRR

FRR (False Reject Rate), and FNR (False Negative Rate) false negative rate, refers to the same person but is not considered the same person to account for the number of times the same person is compared, the calculation formula is as follows

In some 1:1 face recognition competitions, there will also be some other evaluation indicators, FMR (false match rate) and FNMR (false non-match rate). FMR means that the false match rate is equivalent to FAR, and FNMR means that the false mismatch rate is equivalent to FRR.

Sometimes you will see something similar to FNMR @ FMR = 0.000001. This expression indicates that the threshold value when FMR = 0.000001 is calculated on the data set, and then FNMR is calculated based on this threshold value. Similar to this TAR=0.998@FAR=1e-6, it means that when the pass rate of the same person is not 1e-6, the pass rate of the same person is 99.8%. For 1:1 face verification, when the FAR is lower, the higher the TAR, the better

Face search

1:N face recognition evaluation is divided into two cases: open-set identification and close-set identification.

When evaluating 1:N face recognition, we need three sets of images, galleryG (base library, registered face library), probe (probe, face image to be recognized), probe is divided into Pn and Pg​ , The face photos in Pn​ are not in the gallery and are called imposter, and the face photos in Pg​ are in the gallery and are called genius.

Open set recognition

Open-set identification (open-set identification): The problem to be solved in open-set identification is to determine whether a probePj​ is in the gallery, and if that person is. Pj​can be in the gallery or not.

Suppose galleryG={g1​,g2​.....gn​}, gi​ represents a person in the gallery, and the face Pj​ to be tested is calculated as a similarity with each gi​, denoted by Sji​, Sji​ Indicates the probability that two face pictures are the same person. We calculate the similarity between each person in G and Pj​ to obtain a set S{sj1​,sj2​.....sjn​}, and sort the set S from large to small (the Euclidean distance used is Sort from small to large). Assuming that the person corresponding to Pj​ in the gallery is g∗, the definition rank(Pj​)=n indicates that the similarity between Pj​ and g∗ is ranked nth, and rank1 is also called top match.

For face payment, it is an open-set identification of TOP1, and face authentication is an open-set identification of topk.

Closed set recognition

Close-set identification: The problem to be solved in closed-set identification is to find probePj​ in the gallery, and Pj​ belongs to the gallery. Like open set recognition, closed set recognition is also concerned with whether the correct recognition result is included in topk.

Evaluation index

The following evaluation indicators are applicable to both open set recognition and closed set recognition

• TO YOU

DIR (Detection and Identification Rate): refers to the similarity between Pj​∈Pg​ and the real result s∗ in G is greater than τ and greater than the similarity of all other people who are not the same as Pj​ in Pg​ Proportion, DIR measures the passing performance of personnel in the library. Calculated as follows

Let's take an example to illustrate

Assuming that there are face information of three persons A, B, and C in G, each person has a photo, and Pj is another face photo belonging to C. If the similarity of Pj to A, B, and C is 0.5, 0.6, 0.9, τ is 0.7, then the match is correct at this time. If the similarity between Pj and C is 0.68<τ, it will be regarded as a person outside the library, leading to missing matches. If the similarity between Pj and A, B, and C is 0.6, 0.8, 0.78, then the TOP1 match of Pj should be B at this time, because the similarity between C and Pj ranks second is rank(Pj)=2, this When Pj matches the wrong person.

• FAR

FAR (False Alarm Rate): Refers to the similarity of the most similar person in Pj​∈Pn​ and G is greater than the proportion of τ in Pn​. FAR measures the rejection performance of faces outside the library and is also the security of the face recognition system The guarantee of sex. Calculated as follows

For 1:N face search, when the FAR is lower, and the DIR is higher, the performance of the model is better. Let’s use an example to illustrate how these two evaluation indicators are calculated

The above table shows the test results of Pg​ (photo in G) and Pn​ (photo not in G) in gallery (G), a, b, and c correspond to A, B, C, d, e in G, respectively , F are not in G. When the threshold τ is 0.7, let’s calculate DIR and FIR

1. a has the highest similarity with A, and SaA​>τ, so the match is successful
2. Although the similarity between b and B is SbB​>τ, SbB​
3. c has the highest similarity with C, but ScC​<τ, so the matching error

So DIR=1/3≈33.33%

1. d has the highest similarity with A, and SdA​<τ, because d is not in the library, the match is correct
2. e has the highest similarity with C, and SeC​>τ, because c is not in the library, so the matching error
3. f has the highest similarity with C, and SfC​<τ, because f is not in the library, so the match is correct

So FAR=1/3≈33.33%