Content overview: In the Spring Festival sci-fi movie "The Wandering Earth 2", starring Wu Jing, Andy Lau, Sha Yi and others played the roles, spanning 14 years, the visual effects team used Deepfake de-aging technology to present the actor Faces of different age groups.
Keywords: De-aging Deepfake Wandering Earth
"The Wandering Earth 2" has been released for 8 days, and the total box office has exceeded 2.4 billion, breaking 36 records and achieving 91 milestone achievements. Among them, the 21-year-old Xiao Xianrou Liu Peiqiang (played by Wu Jing) and the young and handsome Tu Hengyu (played by Andy Lau) left a deep impression on the audience.
In a video released by the official Weibo of Wandering Earth, it tells the complicated engineering difficulty behind the face-changing special effects.
Change! Change! Change! 20-year-old Wu Jing turned into a little fresh meat
In Zhihu's question "What are the unknown production problems behind the scenes of "The Wandering Earth 2", Xu Jian, the visual effects director of "The Wandering Earth 2" and the founder of MOREVFX, admitted frankly that the first technical difficulty is the "De-aging" number change age.
Digital age change, 45+ Wu Jing turned into a 21-year-old little fresh meat
De-aging is a visual effects technology applied to film and television works. It refers to the use of digitally editing (digitally editing) or computer graphics (CG for short) technology to achieve the rejuvenation of actors in specific scenes.
In the movie "The Wandering Earth 2", the roles played by actors Wu Jing, Andy Lau, Wang Zhi and many others all involve the rejuvenation of the characters, that is, de-aging.
Use the 2D material of Wu Jing when he was young to train the AI model for de-aging
Complete the face change for the real shot material
A small side note: According to the calculation of the movie plot, Liu Peiqiang was born in 2023.
Take Wu Jing as an example. In reality, Wu Jing was 45+ (according to Baidu Encyclopedia) when he was filming this movie, but Liu Peiqiang was only 21 years old in 2044 in the movie. The 45+ actor challenged to play the 21-year-old "Little Fresh " "Meat" really needs some technical support.
3 major mainstream de-aging methods
In fact, the mainstream de-aging methods in the world at that time were divided into three categories.
The method is divided into two main steps:
* Make digital human and de-aging
Through the facial expression capture device worn by the actor, the actor is scanned and a database is generated to create a digital human of the target actor, and then de-aging is performed on the digital human based on the stills and video materials of the actor's early career.
* Animate a digital human based on a motion capture device worn by an actor
According to Xu Jianzai, the visual effects director of "The Wandering Earth 2", sharing on Zhihu, this method has a high budget and insufficient domestic relevant talents, and the equipment on the body may affect the shooting of fighting scenes and the expression of actors' emotions.
Will Smith wears a device for motion and facial expression capture
In the movie "The Irishman", Industrial Light and Magic developed a de-aging technology that does not require wearing physical equipment or marking the face. The actor is surrounded by three camera equipment, one for the normal shooting position and the other two. Responsible for recording the data required for de-aging, collecting necessary data by shooting infrared lenses, and digitally reproducing the performance.
The data recorded by the camera is combined with the influence of light to de-age the character
In the movie "Captain America: Civil War", the visual effects team used Robert Downey Jr.'s early film and television works and visual materials to train the AI model based on the Deepfake algorithm and complete the face change.
Deepfake young version of Iron Man Tony Stark
According to Xu Jian, the visual effects director of "The Wandering Earth 2", the visual effects team started technical experiments as early as a year and a half before the filming started. After testing 5 or 6 face-changing methods, the visual effects team finally decided to use de- Aging method three, based on the Deepfake algorithm, uses the 2D material of the actor when he was young to train the AI model, and after more than 5 million iterations, replace the real material.
Papers on face rejuvenation and aging
《Production-Ready Face Re-Aging for Visual Effects》
The DisneyResearch team proposed a Face Re-Aging Network, referred to as FRAN, which can automatically reshape the face image in the video without losing the identity, so as to achieve the effect of rejuvenation or aging of the target face.
Enter a portrait age of 35
The upper and lower rows are re-aging at the age of 65 and 18
《High Resolution Face Age Editing》
This paper proposes an encoder-decoder architecture for face age editing, using a simple method to achieve high-resolution face age editing, which can output 1024*1024 resolution images.
According to the input face (yellow box)
Output high-quality face images of different age groups
《Only a Matter of Style: Age Transformation Using a Style-Based Regression Model》
This paper proposes an image-to-image translation method that directly encodes real face images into the latent space of a pre-trained unconditional GAN for aging transformation. It can generate the appearance of all ages with only one photo of the tester, accurate to every wrinkle.
Aging results generated using the SAM architecture proposed in the paper
Pixel-level face-changing technology has a long way to go, and industry standards are still needed
In "The Wandering Earth 2", the presentation of the young Liu Peiqiang and Tu Hengyu is based on the Deepfake algorithm. However, after examining many related papers and combined with the introduction of the visual effects director Xu Jian, we know that in the later stages of film production, there is still a need to Professional post-production personnel carry out complex improvements and repairs on the characters after face-changing.
This is because the face-changing technology still has deficiencies in some details, such as blunt expressions and unnatural eye expressions. These "incongruities" will undoubtedly be magnified on the big screen of the movie, causing discomfort to the audience.
The development of age-reducing and age-increasing algorithms may still have a long way to go to achieve a "perfect fit" with characters in film and television works. How to monitor Deepfake, a technology that is not being abused, is also an important issue in the industry and technology circles. Even a potential risk and challenge that ordinary users have to face.
Reference link:
[1] https://www.zhihu.com/question/579613527
[2] https://onlinelibrary.wiley.com/doi/full/10.1111/oli.12302
[3] https://www.respeecher.com/blog/de-aging-technology-changing-hollywood-future-film-making