Hello everyone, today I will talk about technology and recommend a few open source projects that I have seen recently.
STIT
The STIT algorithm, which I introduced in a previous video at station B.
Let's review the effect again:
The "Hermione" in the original video is smiling and talking , changing her expression to anger , and even changing her age to look like a child 's face.
Look at another set:
In addition to real people, anime expressions can also be changed:
The algorithm is open sourced on February 14, Valentine's Day.
project address:
https://github.com/rotemtzaban/STIT
Most face attribute editing algorithms are now based on StyleGAN.
The idea is to control the properties of the image by controlling the latent code in the latent space.
A simple explanation is that a face picture is composed of multi-dimensional features, such as age, gender, light, skin color, hair quality, etc.
In order to better classify or generate data, the characteristics of the data need to be represented, but the data has many characteristics.
These features are related to each other and have high coupling, which makes it difficult for the model to figure out the relationship between them, which makes learning inefficient. Therefore, it is necessary to find the deep relationships hidden under these surface features, and solve these relationships. Coupling, the hidden features obtained, that is, latent code.
However, this idea is for pictures, and the STIT algorithm is for video processing, considering inter-frame information.
The processing flow of the STIT algorithm is mainly divided into six steps:
The input video is first segmented into frames, and the faces in each frame are cropped and aligned;
Using the pre-trained e4e encoder, invert each cropped face into the latent space of the pre-trained StyleGAN2;
Use PTI (a newly proposed method for video face editing) to fine-tune the generator in all parallel frames, correct errors, and restore global consistency;
All frames are edited accordingly by linearly manipulating their pivot latent codes using a fixed orientation and step size;
Fine-tune the generator again to "stitch" the background and edited face together;
Reverse the alignment steps and paste the modified face back into the video.
For more detailed principles, you can directly read the paper:
https://arxiv.org/pdf/2201.08361.pdf
The operation of the algorithm is very simple. The first step is to install the underlying dependency library:
pip install Pillow imageio imageio-ffmpeg dlib face-alignment opencv-python click wandb tqdm scipy matplotlib lpipsSTIT depends on StyleCLIP and can be installed directly.
pip install git+https://github.com/openai/CLIP.gitThen download the pre-trained model and unzip it to the project root directory .
https://drive.google.com/file/d/1cDvUHPTZQAEWvfiK9C0nDuI9C3Qdgbbp/view
Then run the code with the following command:
python edit_video.py --input_folder /path/to/images_dir \
--output_folder /path/to/experiment_dir \
--run_name RUN_NAME \
--edit_name EDIT_NAME \
--edit_range EDIT_RANGEFor specific parameter descriptions, see the README of the project, which is a bit too much.
ACTUALLY
This algorithm is more practical, a general small target detection.
This is quite useful. In some detection scenarios, objects that are too small are easy to miss detection.
SAHI is a general solution to this problem and proposes an open source framework called slice-assisted hyper-inference (SAHI).
This framework provides a general slice-assisted inference and fine-tuning pipeline for small object detection.
The technique is general and it can be applied on top of any available object detector without any fine-tuning.
Using SAHI can greatly improve the mAP of target detection by as much as 10 points.
The target detection trick is worth a try.
project address:
https://github.com/obss/sahi
This project is very comprehensive, with its own tutorials , Paper , code , etc., and the official one is the most comprehensive.
HowToCook
Finally, let's talk about an easy project. When I saw this project for the first time, I almost burst out laughing.
This is also an eye-opening project for me . I look silly, and I can still play like this!
Hot List #1: A programmer's guide to cooking at home.
The project really lives up to its name: HowToCook.
This item is a little different from a regular recipe though.
1cm * 1cm, very strict, remember to bring a ruler when cooking, haha.
The project author has this to say:
Once the project was released, a bunch of friends went to submit PR, the number of contributors rose to 90+, and the number of stars soared to 11.3k in a few days.
The project does not allow the use of imprecise words such as "amount" and "a small amount".
Haha, so true.
Programmers who love to cook, don't miss it.
project address:
https://github.com/Anduin2017/HowToCook
Summarize
There is a lot of content in this issue, so let's play slowly. See you next time!
·················END·················