Author | Gao Weihua
Produced | AI Technology Base Camp
With the development of deep learning, there are more and more applications of GAN models in image style conversion, many of which have achieved good results.
Previously, a technology blogger on reddit, AtreveteTeTe, based on the GAN model hybrid, cartoonized ordinary portrait photos, and generated animations through the First Order Motion model, which aroused the enthusiasm of many netizens. The generation of cartoon-style portraits like this usually does not require high quality of a single element. For the training of the model, some lack of details will be ignored.
However, a developer recently released a project on GitHub that can be used to create line art portraits, and developed ArtLine based on deep learning. The line portraits created by it are amazing in detail.
Project link:
https://github.com/vijishmadhavan/ArtLine
The easiest way to get started with ArtLine:
https://colab.research.google.com/github/vijishmadhavan/Light-Up/blob/master/ArtLine(Try_it_on_Colab).ipynb
In terms of artistic portrait line drawings, Tsinghua University proposed a hierarchical GAN model APDrawingGAN last year , which can effectively generate high-quality and expressive line drawings based on face photos.
However, APDrawingGAN has limitations in identifying facial features. For example, when identifying a frontal photo similar to an ID photo, the photo needs to have clear facial features and no glasses or obvious wrinkles on the face.
ArtLine can solve this limitation, and can recognize any image of a person's posture. Whether the generated portraits reach the appropriate lines around the face, eyes, lips and nose depends entirely on the data provided to the model.
The APDrawing data set is mainly composed of close-up portraits, and it is difficult for this model to recolor clothes, hands, etc. Therefore, ArtLine combined with the selected images from the Anime sketch coloring data set to help the model better understand the line details.
This is a movie poster created instantly using ArtLine.
Technical realization
Self-attention GNN
ArtLine uses the Self-Attention Generative Adversarial Network (SAGAN) generator, which is a pre-trained UNET with spectrum normalization and self-attention functions, which can provide attention-driven remote dependency modeling for image generation tasks.
The traditional convolutional GAN generates high-resolution details according to the function of spatial local points in the low-resolution feature map. In SAGAN, you can use hints from all element positions to generate details. And, the discriminator can check whether the detailed features in the image part are consistent.
In addition, according to recent research, generator tuning can affect the performance of GAN. For this reason, spectrum normalization is applied to the GAN generator to improve the training dynamics.
SAGAN has achieved the most advanced results. On the challenging ImageNet data set, it increased the published Inception score from 36.8 to 52.52, and reduced the Frechet Inception distance from 27.62 to 18.65. According to the visual display of the attention layer, the generator uses the neighborhood corresponding to the shape of the object instead of the local area of the fixed shape.
GAN gradually improves quality, stability and diversity
ArtLine uses a new training method for generating adversarial networks. The key idea is to add new layers from low resolution, and gradually increase generators and discriminators to model more and more detailed details as the training progresses. This not only speeds up the training speed, but also greatly improves the quality of the generated images.
During training, ArtLine adopted a new indicator for evaluating GAN results such as image quality and changes, and constructed a higher-quality version of the CelebA dataset.
Generator loss: Perceptual loss/feature loss based on VGG16.
When converting high-quality images, the per-pixel loss between the output image and the real image is usually used to train the feedforward convolutional neural network. Studies have shown that high-quality images can also be generated by extracting advanced features from the pre-trained network, defining and optimizing the perceptual loss function.
Combining the advantages of these two methods, ArtLine uses a perceptual loss function to train a feedforward network for image conversion tasks. Compared with optimization-based methods, ArtLine can give similar quality image conversion results, but its generation speed is three orders of magnitude faster. For the single-image super-resolution method, experiments show that replacing the per-pixel loss with the perceptual loss is visually more pleasing.
At present, ArtLine also has some problems in image generation, such as its generator will confuse photo shadows with hair; in addition, for low-quality images less than 500 pixels, ArtLine's image conversion effect is not good.
Paper link:
https://arxiv.org/abs/1805.08318
https://arxiv.org/abs/1710.10196
https://arxiv.org/pdf/1603.08155.pdf
更多精彩推荐
☞谷歌知名前 AI 研究员无辜被裁，CEO：调查！
☞惨烈！上班摸鱼被老板张一鸣抓现场...... | 每日趣闻
☞苏宁回应股权质押给淘宝：正常合作;苹果App Store被越狱商店指控垄断；Docker 20.10.0发布|极客日报☞【官方福利】CSDN内测师限时申请，参与赢年末礼包
☞一文聊“图”，从图数据库到知识图谱
☞区块链赋能物流行业：“圈子文化”下，网络货运的数字化转型之路
点分享点点赞点在看