Click the card below to follow the " CVer " public account
AI/CV heavy dry goods, delivered as soon as possible
Bo Wen from Concave Temple
Reprinted from: Qubit (QbitAI)
I've seen a lot of face-changing, have you ever seen a body-changing person?
Given a face, you can automatically change to a lower body, and there is no PS trace in clothing, body shape, and skin color:
The core technology is of course the familiar GAN, but the difference is that now every part of the body can be PS.
From face, skin color, clothing, hair and other parts of the body, and even body movements , can be designed and combined at will, and finally "stitched" into a 1024 × 1024 resolution full body photo:
And this "stitching monster" has no shadows and borders caused by the stitching behavior:
△ The face above is generated by the new method, with few shadow boundaries
How did you do it? Put together the GANs used to generate different parts of the human body .
This is the latest method proposed by the Adobe team to combine multiple pre-trained GANs for image generation. The paper has now been accepted by CVPR 2022:
Let's take a look at how they do it next.
Using PS to GAN out a human body
As we said at the beginning, this is a method of stitching together multiple GANs, which the research team calls InsetGAN .
There are two types of GANs:
Full-Body GAN (Full-Body GAN), which is trained on medium quality data and generates a human body.
Part of the GAN , which contains multiple GANs trained on specific parts such as the face, hands, and feet.
The cooperation of these two types of GANs is similar to PS: a full-body GAN is a canvas that already has a baseline draft, while some GANs are layers stacked on top of each other.
However, when "layers" with different borders are stacked on the canvas, there must be alignment problems.
For example, when adding a face to a body, there may be distortions and loss of detail, or artifacts in the consistency of skin tones, the naturalness of clothing boundaries, and loose hair:
How can we better coordinate multiple GANs so that they produce consistent pixels?
The research team designed such an architecture:
They first introduced a bounding box detector to detect the position of a specific area generated by a partial GAN in the underlying canvas, that is, the area generated by the full-body GAN, and then embedded the specific area after cropping.
This process is equivalent to finding a random latent code between the two regions , so that the boundaries of the selected region can be matched with the embedded region for seamless synthesis.
At the same time, they also downsample these two regions, again increasing the consistency of the pixel content of the image.
Based on this method, InsetGAN can generate multiple full portraits after training, while skin color, hair and related poses can be adjusted accordingly:
The research team also compared with the previous method of generating full-body portraits, CoModGAN, which replaces the face based on the human body on the left. Obviously, the face generated by InsetGAN is more natural:
△ InsetGAN on the top, CoModGAN on the bottom
about the author
The paper has 6 authors, 5 from Adobe Research and 1 from King Abdullah University of Science and Technology (KAUST).
Among them is Jingwan Lu, chief scientist at Adobe, the main algorithm contributor to filters such as Smart Portrait, Skin Smoothing, Shading, and Neural Stylization in PS 2020, and the developer of the RealBrush brush compositor.
The team she currently leads is primarily focused on leveraging big data and generative AI (such as GAN) for visual content creation.
So, are you ready to change shape without leaving home? (manual dog head)
Paper address:
https://arxiv.org/abs/2203.07293
参考链接：
[1]https://www.youtube.com/watch?v=YKFYEt5hvOo
[2]http://afruehstueck.github.io/insetgan/
ICCV和CVPR 2021论文和代码下载
后台回复：CVPR2021，即可下载CVPR 2021论文和代码开源的论文合集
后台回复：ICCV2021，即可下载ICCV 2021论文和代码开源的论文合集
后台回复：Transformer综述，即可下载最新的3篇Transformer综述PDF
CVer-Transformer交流群成立
扫描下方二维码，或者添加微信：CVer6666，即可添加CVer小助手微信，便可申请加入CVer-Transformer 微信交流群。另外其他垂直方向已涵盖：目标检测、图像分割、目标跟踪、人脸检测&识别、OCR、姿态估计、超分辨率、SLAM、医疗影像、Re-ID、GAN、NAS、深度估计、自动驾驶、强化学习、车道线检测、模型剪枝&压缩、去噪、去雾、去雨、风格迁移、遥感图像、行为识别、视频理解、图像融合、图像检索、论文投稿&交流、PyTorch、TensorFlow和Transformer等。
一定要备注：研究方向+地点+学校/公司+昵称（如Transformer+上海+上交+卡卡），根据格式备注，可更快被通过且邀请进群
▲扫码或加微信: CVer6666，进交流群
CVer学术交流群（知识星球）来了！想要了解最新最快最好的CV/DL/ML论文速递、优质开源项目、学习教程和实战训练等资料，欢迎扫描下方二维码，加入CVer学术交流群，已汇集数千人！
▲扫码进群
▲点击上方卡片，关注CVer公众号
整理不易，请点赞和在看