This article will be updated from time to time
1 Introduction
Some senior engineers in fingerprint recognition companies are very experienced. They don't write papers, but sometimes write patents. To learn from them, it is necessary to read patents.
But for many reasons, patents are not easy to read.
- In many patents, the inventor provides an idea or a basic description, and the patent attorney writes the application text. According to the writing method of legal documents, the written expression is very cumbersome and the words are peculiar, so the reading experience of patents is far inferior to that of papers. A 10-page patent may be only 1 page if the idea is condensed.
- The patent lacks details. In order to expand coverage and keep confidentiality, the patent does not describe the method in particular. Reproducing the algorithm described in the patent is difficult. Maybe the inventor didn't actually make it either.
- Patents do not speak of relationship to related work. It is impossible to trace the source from the patent and figure out some ambiguous expressions.
- Patents do not talk about experimental results. Whether the technology is feasible or what its performance is cannot be seen from the patent. Perhaps the inventors did not experiment.
- The patent does not talk about performance comparison with similar technologies. The relative level of technology cannot be seen from patents. Perhaps the inventor has not compared it with similar technologies.
- Illustrations of patents are usually not clear enough or aesthetically pleasing.
What's more troublesome is that the number of fingerprint patents is no less than that of papers. Papers have handbooks, reviews of other articles, and Google citations as a reference for quality. And how to judge whether a patent is valuable, and where to quickly find valuable patents, there is no simple standard. Searching for treasures in the vast patent literature requires vision and energy.
The second edition of "Handbook of Fingerprint Recognition" has as many as 1,200 references and almost no patents. I wrote an annotation at the time, and when the third edition of this book is released, I will give the author a suggestion to cover important patents. Unexpectedly, I became the author of the third edition and found it difficult to include patents in the book.
This article classifies and summarizes some good patents I have seen for readers' reference. For each patent, only a brief introduction and comments will be made. At present, the patents are mainly on the algorithm, and the sensor will be added in the future.
2. Feature extraction
2.1 Image enhancement and denoising
NEC_Hara_2008_USPA20080226143_CHARACTER NOISE ELIMINATING APPARATUS
Japan's NEC is one of the earliest companies in the world to make AFIS for police use, and its fingerprint algorithm is often among the best in NIST evaluations. Among AFIS companies, NEC has more patents. Hara is the algorithm lead of the fingerprint department and holds many fingerprint patents.
Strong ridge enhancement algorithms (such as Gabor filtering) may destroy details and produce false features. Sometimes fingerprinting experts need less strong augmentation algorithms to avoid introducing spurious features. In this patent, characters are segmented first, and then a suitable grayscale adjustment method is used in the character coverage area.
NEC_Hara_2008_USPA20080031531_Line Noise Eliminating Apparatus
This patent removes the straight lines covering the fingerprint (eg background pattern of live fingerprints, lines on ink stamped fingerprint cards).
2.2 Direction Field Estimation
adaptive window
Motorola_Lo_2007_US20070292005_Method and Apparatus for Adaptive Hierarchical Processing of Print Images
Printrak in the United States is a very early fingerprint identification company, which has been acquired many times: Printrak -> Motorola -> Morpho -> IDEMIA. Lo has been in charge of the algorithm for a long time. He is very experienced in fingerprint recognition and holds many fingerprint patents.
Orientation field estimation is the most critical step in fingerprint feature extraction. The size of the window used when estimating the local orientation is a critical parameter. The advantage of a small window is that it is more accurate, but it is too sensitive to noise; the advantage of a large window is that it has better anti-noise performance, but it may not be accurate (especially in places with large curvature of the direction field).
This patent flexibly determines the window size according to the image local quality (as well as noise, direction consistency). When the noise is not strong, use the small window first; when the noise is strong, use the large window to calculate the direction.
palm print direction field
NEC_Funada_2002_US7027626_System and method for processing fingerprint, palmprint image
There are so many wrinkles on the palmprint (especially the large thenar area), which is a great interference to the estimation algorithm of the ridge direction field. If the direction field estimation algorithm for fingerprints is applied to palmprints, there will be a large number of errors. This patent detects possible sine waves for each image block. Firstly, the reliable region of the sine wave parameters (the region without wrinkle interference) is identified as the starting point, and the sine wave parameters of the unreliable region are gradually extrapolated. During extrapolation, the parameters of the adjacent sine waves should be relatively consistent.
2.3 Pose Estimation
Finger center detection
NEC_Hara_1991_US5040224_Fingerprint processing system capable of detecting a core of a fingerprint image by statistically processing parameters
Estimating the finger center is very useful in subsequent tasks. The patent proposes to predict the center position from different positions of the fingerprint, and accumulate to obtain the final center estimate.
finger direction estimation
NEC_Hara_1998_US5848176_Fingerprint fingertip orientation detection method and device
This patent utilizes the characteristic that the peripheral direction field of the fingerprint is relatively stable, and defines corresponding templates for different pattern fingerprints. For input fingerprints, match various templates to find the best finger orientation.
2.4 Ridge Enhancement
Partial Differential Equations
Cogent_Wei_2007_US7194393_Numerical model for image feature extraction
The American Cogent company is a company founded by Chinese. It was established later than Printrak, NEC, and Morpho, but it has seized several important opportunities (Los Angeles social security, US immigration, Venezuelan elections), and has grown rapidly. It is the first Nasda A listed biometric identification company. Later, the founder Xie Ming donated to name his alma mater, the Department of Electronic Engineering of the University of Southern California.
The inventor Wei is a mathematics professional background. The patent models the partial image of the fingerprint with binary partial differential equations. Knowing the boundary conditions and numerically solving the equations, an enhanced image can be obtained.
2.5 Tertiary features
NEC
NEC_Hara_2011_US8019132_pore
When fingerprint identification experts compare fingerprints, they sometimes use three-level features, including sweat pores, points (1 ridge unit), and immature ridges (incipient ridge). Although there are some papers on fingerprint matching based on three-level features (Jain et al., 2007), the contribution of relatively high-performance minutiae matching algorithms is not obvious. The main reason is that the imaging of the tertiary features is unstable. This patent studies the tertiary features from another angle. Hara noticed that tertiary features can lead to errors in ridge period measurements. Therefore, he first detected the third-level features, and then erased them to reduce the side effects of the third-level features.
2.6 Quality Estimation
NEC_Hara_2011_US7885437_Quality
Previous fingerprint quality estimation methods, such as NFIQ (Tabassi et al., 2004), did not consider the issue of fingerprint pose plausibility. When the pose is too biased, recognition may fail even if the image quality is good. Hara first divides the model area, and then takes the characteristic quality and quantity of the model area into the quality estimation problem. Hara mentioned at the NIST Fingerprint Quality Symposium that the estimated quality correlates well with the final matching performance and can be used to accurately reject low-quality fingerprints.
Motorola_Lo_2008_US20080013803_Method and Apparatus for Determining Print Image Quality
Lo also considers the importance of the fingerprint central region (pattern region) in fingerprint quality estimation. When estimating the center and angle of the finger, he comprehensively considered the pseudo-ridges above and below the fingerprint pattern area, as well as the location of the singularity point. Comprehensive consideration of multiple features is very necessary for low-quality fingerprints and incomplete fingerprints.
2.7 Four-finger segmentation
NEC_Hara_2006_USPA20060067566_Plain four-finger image determination device
This patent performs four-finger segmentation on the basis of the fingerprint center detection algorithm.
Motorola_Lo_2006_US7072496_Slap Print Segmentation System and Method
The four-finger segmentation method of this patent considers the relationship between fingers more.
2.8 Semi-automatic feature labeling
Direction FieldEdit
NEC_Hara_1996_US5519785_Correcting method for directional data of streaked patterns and information processing apparatus
Orientation fields automatically computed from live fingerprints are prone to errors. This patented technology can provide convenience for manual correction of the direction field. The user draws some curves (false ridges) along the direction field in the area to be repaired, and then the algorithm automatically interpolates the direction field of the entire area. It is more convenient to interact with the touch screen.
RidgeEdit
NEC_Hara_2004_US6806878_Graphic editing apparatus for adding or deleting curve to,from graphics by interactive processing
NEC's fingerprint matching algorithm pays special attention to the role of ridges, and getting high-quality ridges is very important to their algorithm. When the ridges automatically extracted from live fingerprints are inaccurate, manual restoration is required. This patent solves the problem of semi-automatic labeling of ridges.
3. Match
3.1 Minutia point matching
Cogent_Wei_US8379982_Fast match
This patent establishes an attribute table (strategy for memory exchange time) for the details of on-site fingerprints (query fingerprints) to improve the efficiency of large database comparisons. In the era of slow CPU and GPU, Cogent's fingerprint comparison accelerator card used to be very advantageous.
3.2 Multi-feature Fusion
Printrak_Lo_1999_US5960101_Expert matcher fingerprint system
This patent integrates the matching scores of features such as minutiae, ridges, direction fields, patterns, and singular points to achieve high-precision recognition.
3.3 Orientation Field Matching
Motorola_Lo_2008_US20080273769_PRINT MATCHING METHOD AND SYSTEM USING DIRECTION IMAGES
On the basis of minutiae point matching, taking the direction field into account can obviously improve the recognition rate. This is common sense. And the patent goes a step further, considering that after different fingerprints are aligned, even if the direction fields in the overlapping areas are relatively similar, unreasonable patterns may appear in the periphery.
3.4 Ridge Matching
NEC_Hara_2007_US7295688_Method and Appratus for Matching Streaked Pattern Image
NEC pays special attention to the matching of ridges. Because fingerprint identification experts also pay special attention to whether the ridges are consistent when matching fingerprints. Hara's ridge matching method is more considerate for complex situations (skin deformation, noise causing ridge topology inconsistency). I did ridge matching during my Ph. D., and I have a deep understanding of its complexity.
3.5 Phase matching
Arete_Thebaud_1999_US5909501_Systems and methods with identity verification by comparison and interpretation of skin patterns
Thebaud's patent has two highlights: (1) The alignment method based on image correlation considers scaling and rotation; (2) The fingerprint phase representation solves the pixel-level offset field estimation. Arete subsidiary Bioscrypt company FVC2004 win seems to rely on this method.
Bioscrypt_Harkless_2005_US20050254693_Pattern-based interchange format
This patent represents each square of the fingerprint image as a sine wave, and this patent corresponds to an ISO/IEC standard.
3.6 Three-level feature matching
Motorola_Lo_2008_US20080101663_Methods for Gray-level ridge feature extraction and associated print matching
Since the performance of the three-level features in the fingerprint image is not stable enough, the method of explicitly extracting and matching the three-level features (Jain et al., 2007) may not be robust enough. This patent proposes a matching method that implicitly uses three-level features, without explicitly extracting three-level features such as sweat pores.
3.7 Palmprint partition
NEC_Monden_2002_US20020141620_Palmprint region deviding
This patent proposes a method of palmprint partitioning, which can improve the efficiency of palmprint matching.
3.8 Template update
Apple_Boshra_2017_US9576126_Updating a template for a biometric recognition device
Apple_Boshra_2019_US10248776_Background enrollment and authentication of a user
AuthenTec's capacitive fingerprint sensor was once the world's first. In 2012, it was acquired by Apple for 356 million US dollars, and then the Touch ID of iPhone 5S was launched.
Boshra is the head of fingerprint algorithms at AuthenTec and Apple and holds several patents. These two patents are about fingerprint registration and template updating. For example, after the user presses the Home button to light up the screen, enter a PIN to unlock the screen. At this time, the image collected by the fingerprint sensor embedded in the Home button can be considered as the registered fingerprint (even if the user has not registered the fingerprint, or the image does not match the registered fingerprint).
Since the fingerprint sensor area of most mobile phones is very small, in order to improve the user experience (increase the correct acceptance rate while the false acceptance rate remains unchanged), the fingerprint recognition system relies on the template update technology to increase the correct acceptance rate. Due to the high frequency of users unlocking mobile phones, template updates are also conditional. In other fingerprint recognition applications (such as immigration, ID card), the frequency of users using fingerprint recognition is very low, the significance of template update is small, and there are not many opportunities. Frequent recognition is unique to mobile phone fingerprint recognition.
Template updating is not unique to fingerprints, and other biometrics also use updating techniques to improve performance (Rattani et al., 2009; Pisani et al., 2019).
There was once a bug in the template update technique. Someone discovered that after his mobile phone's fingerprint sensor was cracked, other people's fingerprints could also unlock his mobile phone. Some technicians took it a step further and found that orange peels can also crack the fingerprint recognition of mobile phones . The reason is that if the surface of the sensor is not clean (there is a background texture caused by sensor cracking, fingerprint stickers, etc.), when the background texture + personal fingerprint is verified, it will be registered as a template. If the characteristics of the background texture account for a large proportion, and then other fingers, orange peels, or anything press the sensor, the sensor reads the pattern of anything + the background texture, and the verification is likely to pass. This bug was basically resolved later. There is a special direction poisoning attack (Biggio et al., 2013) to study the use of template update problems to crack the recognition system.
4. Interaction
fingerprint navigation
AuthenTec_Russo_2009_US7587072_Generate rotational inputs
The patent measures the 2D rotation of the finger from a fingerprint image for control purposes.
references
- Biggio, B., Didaci, L., Fumera, G., & Roli, F. (2013). Poisoning attacks to compromise face templates. In 2013 international conference on biometrics (ICB) (pp. 1-7). IEEE.
- Jain, A. K., Chen, Y., & Demirkus, M. (2007). Pores and ridges: High-resolution fingerprint matching using level 3 features. IEEE transactions on pattern analysis and machine intelligence, 29(1), 15-27.
- Pisani , PH , Mhenni , A. , Giot , R. , Cherrier , E. , Poh , N. , ... & Amara , NEB (2019). Adaptive biometric systems: A review and perspectives. ACM Computing Surveys (CSUR), 52(5), 1-38.
- Rattani, A., Freni, B., Marcialis, G. L., & Roli, F. (2009). Template update methods in adaptive biometric systems: A critical review. In International Conference on Biometrics (pp. 847-856). Springer, Berlin, Heidelberg.
- Tabassi, E., Wilson, C., & Watson, C. (2004). Fingerprint Image Quality. NIST-IR 7151.