This article has been included on GitHub https://github.com/Jack-Cherish/PythonPark . There are technical dry goods articles, organized learning materials, and first-line manufacturers' interview experience sharing. Welcome to Star and improve it.
I. Introduction
Hi everyone, this is Jack.
Have you seen the " Talking Head Anime " released last year ?
Recently, the Google engineer upgraded the algorithm, and "Talking Head Anime 2" works better !
Only one PNG two-dimensional character picture is needed to generate a virtual idol and a Vtuber character with facial expression capture .
The correct way to open the second element
Old rules, teach everyone how to play today!
Second, the principle of the algorithm
The core idea of the algorithm is: given a face image of an anime character and six-axis pose information (Pose), generate another face image of the same character.
Through real-time control of the six-axis pose information (Pose), the effect of real-time control of cartoon characters can be achieved.
The whole process is classified into two steps:
- The first step: change the facial expression, that is, control the opening and closing of the eyes and mouth.
- Step 2: Rotate the face.
Each step uses a separate network, and uses the output of the first network as the input of the second network.
The first network is the face deformer, and the second network is the face rotator.
In order to obtain better facial movements, the author increased the posture dimension to 42 dimensions in this upgrade .
At the same time, eyebrows, eyes, and mouth deformers are added to the face deformer .
Eyebrow deformer: You can segment the eyebrows and control the twisted shape of the eyebrows.
The network of eye and mouth deformers operate similarly.
For more specific principles, see the author's paper:
https://pkhungurn.github.io/talking-head-anime-2/
Three, algorithm test
"Talking Head Anime 2" is not open source yet, but it should be available soon.
If you are in a hurry, you can play "Talking Head Anime" first, the effect is also very good, and it is also considered as the preparation of the environment for the second generation.
project address:
https://github.com/pkhungurn/talking-head-anime-demo
The first step: configure the development environment.
There are not many dependencies, just use Anaconda to configure and install it.
Here is a detailed environment configuration tutorial: " Set up a Python development environment once and for all "
Step 2: Download the code and weight file.
There are a total of four weight files, downloaded and placed in the data directory of the project directory.
I packaged the code and weight files and can download them directly (extract code: jack):
https://pan.baidu.com/s/1MlBYx2EhyZgRoD7nXWayBA
The third step: Run the program.
python app/manual_poser.pySuper simple, running effect:
Four, summary
Looking forward to the second optimized version, the effect is definitely better!
Real faces can also be controlled in this way, but it requires more dimensions and more data.
Now a variety of virtual anchors have appeared, and many related products have been launched in the field of live broadcast. This type of technology will continue to do better and better.
The driving of the three-dimensional face model will not be a problem.
This is Jack. See you next time.
The first link of the article:
the correct way to open the second element