https://mp.weixin.qq.com/s/sRwjpW8cYxd2lw2yZD-Mtw
By 超神经
场景描述：风靡各大直播平台的美妆博主，凭借高超的化妆技术吸金无数。而人工智能也已经开始学习这一本领。利用深度学习与计算机视觉技术，仅仅根据人的眼睛特征，就能给出适合用户的美妆搭配。
关键词：几何变换 triplet 损失函数 迁移学习In recent years, more and more beauty bloggers have emerged on the Internet. They explain beauty techniques and share the effects of cosmetic trials in order to accumulate fans and cooperate with businesses to sell products.
For example, Li Jiaqi, who was on fire some time ago, is a beauty blogger known as the "lipstick devil". He frantically tried 380 lipsticks in a live broadcast and set a record of 14,000 lipsticks sold in one minute.
However, many girls who love makeup should have realized that they have bought lipstick exactly like the blogger, but the effect of the painting is different. Seeing that the color numbers used by the "Li Jiaqi" are very beautiful, immortal and expensive, but when it comes to my own mouth...
"Seller Show" and "Buyer Show" in the lipstick world
Yes, it is precisely because everyone's face shape, skin color, lip shape, etc. are different, that led to the results of the "seller show" and the "buyer show".
So the question is, how can I know which beauty product is most suitable for me? The answer given by a company called Mira is: use deep learning.
Deep learning also loves beauty
Many people think that the terms such as artificial intelligence and deep learning should have nothing to do with beauty, but Mira, a startup based in Los Angeles, USA, thinks otherwise.
This company decided to use artificial intelligence technology to help the majority of women who love beauty, such as getting makeup inspiration and purchasing suitable beauty products.
Before and after makeup, the effect is comparable to a face change
After chatting with dozens of beauty professionals at random, the Mira team learned that the biggest difficulty currently encountered by female consumers when looking for suitable makeup products and beauty methods is that there is no authoritative and credible voice that can target them Provide guidance on personal beauty needs.
In this article, we will talk about how Mira's technical team used deep learning and computer vision technology to find examples that hit the key to this problem: find beauty experts, pictures, and videos that explain the specific eye shapes and facial complexions of humans.
Along this way, the Mira team uses three simple but powerful knowledge-geometric transformation, triplet loss function, and transfer learning to solve all kinds of difficult beauty inference problems with minimal human input data.
AI helps you choose the most suitable eye makeup
Schematic diagram of eye parts
Ladies who love makeup know that it is difficult to find beauty products and methods that suit their eyes-everyone's eye shape and facial complexion are different.
Even the same kind of eye makeup (such as smoky makeup), the makeup methods used are quite different depending on the eye shape.
Although some useful makeup guides like Birchbox and others have launched, the Mira team found through investigations that beauty enthusiasts usually still like to listen to professional and credible advice, especially those from people with similar eye shapes. The importance of these suggestions even exceeds the opinions of beauty experts.
Using artificial intelligence technology, we can now let ourselves know how to make up and what cosmetics to buy based on our own eye characteristics and other unique facial features.
AI beauty first step: find similarities
Let’s formalize the problem: According to a set of facial photos and a small number of artificially labeled photos (marked with eye color, eyelid shape, etc.), find the visual similarity measure between the two eyes ("This "My sister I have seen before" means that). The classifier is then used to capture the attributes of the artificial markers.
This article first focuses on how to determine the similarity between eyes, and then explains in detail how to perform classification tasks.
The original image is not very suitable for computing visual similarity or for classification tasks. Because they contain many similarities on the surface (for example, the makeup is very similar, and the skin color looks different because of the strong light).
These have nothing to do with the character's real eye structure and facial skin tone. Moreover, the original images are generally in a high-dimensional space, which requires a large amount of labeled training data for classification tasks.
As shown in the figure above, if only the image pixels are directly compared, the eyes of the characters are highly similar, but careful attention will find that although the eyeshadow, light and direction of the line of sight of the characters are the same, their eye color and facial skin color are different.
The difficulty of processing the original image: Although the eyes of the two people in the above picture are very different, the initial data are very similar in comparison
Then Mira’s first task is to obtain low-dimensional and dense mathematical expressions of eye photos, which are what we call “embeddings”.
It only captures the image quality required by the task (nesting is a classification feature, represented by continuous value features. Usually, nesting refers to the mapping of high-dimensional vectors to low-dimensional spaces.) In this way, "nested "Should ignore these messages:
- Eye position/direction of sight
- Specific light conditions (of course there are powerful filters)
- No matter what kind of makeup I put on my face
When training the eye embedding with a triple function, the system learns to ignore irrelevant features
AI Beauty Step 2: Projection transformation for image normalization
We can remove the surface similarity of an entire category through a simple preprocessing step-projection change.
Although the cropped eye photos will show many obvious structural differences (such as eyes not in the center of the photo, or rotation due to head tilt, etc.), the projection change allows us to "distort" the photo, so that the same can be guaranteed The eye signs of are at the same coordinates.
With a little bit of linear algebra, we can "distort" an image, so that a set of points will be mapped into a new ideal shape. The process of rotating and stretching an image is as follows:
Using the projection change, the above image can be distorted. The 4 red dots in the above image will form a rectangle, thereby "straightening" the text surrounded by the red dots. The Mira team applied the same method when normalizing the eye photos.
The researcher used dlib to detect facial markers (if you are interested in dlib, you can find out in the following link: http://blog.dlib.net/2014/08/real-time-face-pose-estimation.html ).
Crop the eyes in the photo and "distort" them to make sure they are aligned and consistent. This step allows them to focus on making "nesting" unaffected by the pose and tilt of the character's head.
Then proceed to image normalization: detect facial landmarks, crop the eye image, and then use projection transformation to "distort" the eye image to a standard position.
Image samples in the image preprocessing process
AI beauty third step: use triplet loss function for representation learning
When the "distorted" processed images are directly compared, they will still show some surface similarities, including the direction of the line of sight and similar makeup. Deep learning technology is the prescription to solve this problem.
Researchers have trained a convolutional neural network, which will output vectors after inputting eye photos. Compared with different people, the output vectors of the same person's eye photos are more similar. The neural network will learn to output the stable and continuous representation of each human eye in different environments.
Of course, what we rely on here is the triplet loss function mentioned earlier, and its formula is as follows:
This explains in detail that when the function places the two "nested" positions of specific individuals (anchor points and positive samples) closer than the positions of anchor points and unrelated individuals (negative samples), the loss and optimization goals of the model will decrease.
Model architecture diagram
When the researchers applied eye photos to the model, they found that the resulting "nesting" was a good indicator of two photos with similar eye structure and facial skin color.
Examples of photos with similar nesting of eyes
The method used here is actually very similar to Google's FaceNet, that is, through the "distortion" and consistency processing of the photos, the triplet loss function is applied to generate face-level image nesting.
AI Beauty Step 4: Combine Nests
The researchers simply debugged the generated nesting, making it also suitable for supporting person-level eye representation-extracting all the noise data for each frame.
By using the pre-training weights of the above neural network, the researchers adopted a new loss function, which puts the average value of multiple nested groups in very close positions (relative to unrelated individuals), as shown below:
Using the pre-trained weights of the previous neural network, researchers can enable the network to nest the eyes together by averaging, and see that the model converges quickly. This process is often referred to as transfer learning.
Transfer learning allows nesting to be merged into a more holistic representation of individual eyes. Although the neural network architecture is very complex at this time, the model can quickly converge due to the use of transfer learning.
Finally, the researchers verified the model with a data set and found that the nesting generated by the model can capture the subtle similarities between individuals, as shown below:
The eye nests of each line of characters are very similar
Just look at you and give suggestions for perfect makeup
By obtaining high-quality mathematical representations of human eyes in a single photo, researchers can find out the similarity of a person’s eye structure, which lays the foundation for matching a suitable eye makeup style based on the person’s eyes only.
The Mira technical team stated that the next task is to apply several supervised learning methods (classification of eye shapes, regression of eye color, etc.), as well as some analysis methods, to build an AI model that can provide people with makeup suggestions.
In other words, in the future, girls no longer have to worry about what kind of makeup is best for their eyes and skin tone, and they do not have to mechanically refer to standard makeup guides and beauty bloggers' color test results. AI will recommend you more suitable for yourself. Beauty makeup.
As a result, the beauty bloggers might be robbed of their jobs? However, Li Jiaqi no longer has to work so hard, trying 380 colors in a live broadcast.
Note: The implementation of all the codes and results in this article uses NumPy, SciPy, Matplotlib, Chainer, dlib and SqueezeNet architecture.
HyperNeuropedia
Transfer learning
Transfer learning is a machine learning method that takes the model developed for task A as the initial point and reuses it in the process of developing the model for task B.
In deep learning, in computer vision tasks and natural language processing tasks, it is a common method to use pre-trained models as the starting point for new models. Usually these pre-trained models have consumed a huge amount of time when developing neural networks. Resources and computing resources, transfer learning can transfer the acquired powerful skills to related problems.
The following are two commonly used methods:
- Method of developing the model
- Methods of pre-training models
