Eat these data sets and models, learn to dance with AI, and do TensorFlowBoys

https://mp.weixin.qq.com/s/zmpvOFAjTc8r8sxJbidpGQ

By 超神经

场景描述：利用深度学习算法 GAN 可实现动作追踪与迁移，将某人物动作复制到其他人，应用到舞蹈领域，人人皆可成舞王。

关键词：GAN  动作迁移  舞蹈

Recently, "This! The second season of "It's Hip-hop" started broadcasting, once again igniting a wave of national dance.

Not long after it started broadcasting, this full-time high-energy program got a high score of 9.6 on Douban. The wonderful performances of the dancers in the competition caused the crowd eating melons in front of the screen to call out "Too burning!" and "Amazing!", and they even couldn't help shaking with the music.

However, if I really want to jump up on my own, it is estimated that there is a gap between Luo Zhixiang's reality and imagination. I imagined myself like this:

But in fact it is like this:

For dancers, their actions are called Hiphop, Breaking, Locking, etc., while for melon-eating people, they are shaking, rolling, and pointing...

Maybe you missed hip-hop in this life? Let's go to the square dance...

and many more! Don't rush to give up. Several big guys from the University of California, Berkeley, have researched an AI "secret weapon" for you to make your dance skills burst out instantly and become the next generation of dance kings.

Everyone can be the king of dance

In August last year, researchers at the University of California, Berkeley published a paper titled "Everybody dance now", using the deep learning algorithm GAN (Generative Adversarial Networks), which can replicate the movements of professional performers and combine The movement is transferred to anyone, so as to realize "Do as I do".

Let's first look at the result display of the copy dance and feel it:

The upper left corner is the professional dancer, the lower left is the detected pose, the middle and the right are the generated videos copied to the target person

In the past, Deepfake's face-changing technology became a big hit, but now the whole person can "Deepfake"! Let's take a look at how this god operation is achieved.

According to the paper, the migration method is divided into the following steps:

• Given two videos, one is the action source video and the other is the target character video;

• Then use an algorithm to detect the dance posture of professional dancers from the source video and create a stickman frame of the corresponding movement;

• Next, use the two trained deep learning algorithms for Generative Adversarial Networks (GAN) to create all the images of the target person and generate clearer and more realistic video images for them.

The end result is that the system can map the body movements of professional dancers to amateur dancers. In addition to imitating actions, it can also perfectly fictional human voices and facial expressions.

Secret behind the black technology

The specific principle of this black technology is as follows. The movement migration pipeline is divided into three parts:

1. Attitude detection:

The team used the existing pose detection model OpenPose (CMU open source project) to extract the key points of body, face and hand pose from the source video. The essence of this step is to encode the body posture, ignoring information such as body shape.

Perform pose detection on dancers and encode them as stickman graphics

2. Global posture standardization:

Calculate the difference between the body shape and position of the source and target characters in a given frame, and convert the source posture graphic to a posture graphic conforming to the body shape and position of the target character.

3. From the standardized posture graphics, infer the image of the target person:

Using a generative confrontation network model, the training model learns to map from the standardized posture graphics to the target person image.

Schematic diagram of training process (top) and migration process (bottom)

In the process of developing the system, the team used GeForce GTX 1080 Ti GPU in NVIDIA TITAN Xp and cuDNN accelerated by PyTorch for training and inference.

In the image conversion stage, the image translation pix2pixHD architecture developed by NVIDIA for confrontation training is adopted. The face residual is predicted by the global generator of pix2pixHD. They use a single 70x70 PatchGAN discriminator on the face.

During the training process, the source video and target video data collection methods are slightly different. To ensure the quality of the target video, use a mobile phone camera to take a real-time shot of the target subject at a rate of 120 frames per second, and each video is at least 20 minutes long.

For the source video, you only need to get the appropriate posture detection results, so you can use high-quality videos of online dance performances.

System mapping result display

For the results of the system, the researchers said it is not perfect. Although most of the videos it produces are still very lifelike, it occasionally reveals the horse's feet, such as the disappearance of certain parts of the body, like "melting" and other abnormal phenomena.

In addition, because the algorithm does not encode clothes, it cannot produce a video of clothes dancing with the action, and the target must wear tight clothes.

If these shortcomings are ignored for the time being, this technology is indeed exciting.

With this AI tool, even if you are a young dancer in dancing, or your limbs are stiff and uncoordinated, you can become a "dancing master" like Aaron Kwok, Show Luo, or any dancer you like. Even Jackson’s spacewalk is just a piece of cake for you.

However, it is not only the Berkeley team that has a dream of dancing. Google has also put much effort into the combination of AI and dancing.

Google AI creates new dance patterns

At the end of last year, Damien Henry, technical project manager of Google's Arts and Culture Department, worked with British choreographer Wayne McGregor to develop a choreography tool that can automatically generate specific styles.

McGregor, who holds an honorary doctorate of science from Plymouth University, has always been interested in science and technology. When he reviewed his 25-year dance videos, he wondered whether he could use technology to keep the performance fresh. So he went to ask Henry how to use technology to continuously create new dance content?

And Henry got inspiration from a post on a scientific website. This post introduces the use of neural networks to predict the next letter based on the handwriting in the previous letter.

So he proposed a similar algorithm that can predict a given motion. The dancer's pose is captured through video, and then the next most likely dance movement is generated and displayed on the screen in real time.

The video demonstrates the effect of AI choreography and real-time display on the screen

This algorithm also ignores people's clothes, just captures the key points of the actor's specific poses to get the stickman model.

When they entered the dance videos of McGregor and his dancers, AI learned how to dance, and the generated dance style was very similar to McGregor's.

Although in terms of dance creativity, artificial intelligence still has certain limitations. This Google AI tool cannot invent actions it has never "seen". It just predicts the most likely action among the actions it has learned.

In addition, this technology can also provide mixed styles of dance choreography, such as inserting a video of Brazilian samba in McGregor's video, AI may give a brand new mixed dance. Henry didn't worry that it would give a dance of four differences, because the source of learning was still input by people.

AI posture tracking is more than just "dancing dreams"

After seeing so many techniques to help you "dance", are you already eager to try it?

Dance AI allows those who dare not to move, move more freely and easily, and experience the fun of dance and sports. But the technology behind this is more than just blogging.

The posture estimation that supports the dance AI has huge energy hidden behind it. It can help us complete body movements more accurately, such as 3D fitness learning, sports posture correction, patient rehabilitation training, and even virtual fitting, photo posture correction , Will bring new breakthroughs.

Wide range of uses for pose estimation

According to this kind of development, machines will learn more about us, become more and more familiar with our posture characteristics and behavior patterns, thereby helping us to better understand ourselves.

Okay, let's not talk about it, I'm going to learn to dance with AI. Do you want to come together?

Super Neural Dataset

COCO large image data set

The COCO dataset was released by Microsoft in 2014 and has now become a standard test platform for image subtitles. The file size is 83.39 GB.

The COCO data set is a large image data set designed for object detection, segmentation, character key point detection, filling segmentation and subtitle generation in the field of machine vision. The COCO data set aims at scene understanding, which is mainly intercepted from complex daily scenes. The target in the image is calibrated by precise segmentation.

The COCO data set has the following features: target segmentation, perception in the scene, superpixel segmentation, 330,000 images (more than 200,000 labels), 1.5 million target instances, 80 target classes, 91 item classes, 25 People with key points.

Hyper-Neural HyperAI collects and organizes hundreds of public data sets around the world, and provides domestic mirror downloads, and provides free services to scientific research institutions and developers.

For more relevant data sets, please visit https://hyper.ai to download