Introduction
Take a photo and convert it into the hand-painted style works of Japanese manga masters such as Hayao Miyazaki and Makoto Shinkai. This GAN, which specializes in generating animation images, is very useful in actual measurement.
If there is a model that can transform the real picture into a Japanese-style hand-painted picture, it must be very cool. Recently, the heart of the machine found that there are indeed these models, from CartoonGAN to AnimeGAN can generate very interesting images.
Here is a new TensorFlow project that implements AnimeGAN and provides pre-trained models. That is to say, we can directly try to generate the effect after downloading.
Although many of the best examples given by the original project are street scenes, we found that all kinds of scenes are also ok. The following are the original images and generated effects that we tried. Looking at the generation effect of the first cherry blossom road, suddenly there is a feeling of "Spirited Away".
If it is only for characters, the conversion effect is also very good. Try to input the photo of Yui Aragaki into the AnimeGAN model, and then you will have the following magical style.
In the original GitHub project, the author also gave a lot of examples, the above are just the results of the heart of the machine, you can also use it.
AnimeGAN
The whole project implements the method proposed in the paper "AnimeGAN: a novel lightweight GAN for photo animation". The author compares AnimeGAN with CartoonGAN and ComixGAN in the paper.
As can be seen from the figure, AnimeGAN is better than the above two methods in terms of details, the color is relatively more natural, and the smearing feeling is not so strong. The most obvious is the renderings of the second row, the cartoons generated using AnimeGAN are closer to Hayao Miyazaki's style.
Introduction to the method
For the AnimeGAN of this project, the generator network and discriminator network adopted by AnimeGAN are shown below. It seems that the model as a whole is a more conventional convolutional neural network, but it will use instance normalization and the new LReLU activation function.
In addition to the updates in architectural details, the authors also propose the following three new loss functions:
-
grayscale style loss
-
grayscale adversarial loss
-
color reconstruction loss
These loss functions can make the style of the generated image closer to the real comic style.
The table below compares the model size and inference speed of ACartoonGAN and AnimeGAN. It can be clearly seen that AnimeGAN is a relatively lightweight GAN with fewer parameters and faster inference speed.
Overall, the newly proposed AnimeGAN is a lightweight generative adversarial model that uses fewer model parameters and introduces a Gram matrix to enhance the style of photos. The researchers' method requires training on a series of real pictures and a series of cartoon pictures, and these pictures do not need to be matched in pairs, which shows that the training data is very easy to obtain.
Project measurement
We tested this project under Ubuntu 18.04, and the relevant dependencies are as follows:
-
python 3.6.8
-
tensorflow-gpu 1.8
-
opencv
-
tqdm
-
numpy
-
globe
-
argparse
These dependencies can be said to be commonly used extension libraries in CV, so we don't have to bother to solve the problems of various dependency environment conflicts, here is a good review.
The following is the detailed process of training and testing of this project. We first cloned the AnimeGAN project locally and entered in the Jupyter notebook:
!git clone https://github.com/TachibanaYoshino/AnimeGAN
Change the working directory to AnimeGAN:
import os
os.chdir('AnimeGAN')
print(os.getcwd())Next, download the pre-trained model provided by the project author, use vim download_staffs.sh to create a shell file, and enter the following command:
URL=https://github.com/TachibanaYoshino/AnimeGAN/releases/download/Haoyao-style_V1.0/Haoyao-style.zip
ZIP_FILE=./checkpoint/Haoyao-style.zip
TARGET_DIR=./checkpoint/saved_model
mkdir -p ./checkpoint
wget -N $URL -O $ZIP_FILE
mkdir -p $TARGET_DIR
unzip $ZIP_FILE -d $TARGET_DIR
rm $ZIP_FILE
DatesetURL=https://github.com/TachibanaYoshino/AnimeGAN/releases/download/dataset-1/dataset.zip
ZIP_FILE=./dataset.zip
TARGET_DIR=./dataset
rm -rf dataset
wget -N $DatesetURL -O $ZIP_FILE
unzip $ZIP_FILE -d $TARGET_DIR
rm $ZIP_FILE
VGG_FILE=./vgg19_weight/vgg19.npy
wget --load-cookies /tmp/cookies.txt "https://docs.google.com/uc?export=download&confirm=$(wget --quiet --save-cookies /tmp/cookies.txt --keep-session-cookies --no-check-certificate 'https://docs.google.com/uc?export=download&id=1U5HCRpZWAbDVLipNoF8t0ZHpwCRX7kdF' -O- | sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/\1\n/p')&id=1U5HCRpZWAbDVLipNoF8t0ZHpwCRX7kdF" -O $VGG_FILE && rm -rf /tmp/cookies.txtSave and exit, the above command will download and save the pre-trained model, vgg19 weights and training dataset to its corresponding directory. Run in notebook:
!bash download_staffs.sh
At this point, all preparations are completed, and the model can be trained by running the following code:
!python main.py --phase train --dataset Hayao --epoch 101 --init_epoch 1The training process of AnimeGAN is shown in the following figure:
When testing, we need to save the image for testing to the dataset/test/real directory and run the following code:
!python test.py --checkpoint_dir checkpoint/saved_model --test_dir dataset/test/real --style_name H
When you see the above output, the program has been successfully run, and the generated results are saved in the results folder. As you can see, it takes about 2.3 seconds to generate an image on the P100 GPU.
end
That's all for this issue~ I hope it helps you! If you like it, remember to give the editor a three-stroke before leaving~