vid2vid + training + test code debugging (debug + train + test) (a) test article

## Prerequisites

- Linux or macOS
- Python 3
- NVIDIA CUDA GPU + cuDNN
- PyTorch 0.4
but in general, then we have to protect the environment, usually create a virtual environment for each project, as follows:

""" Linux 下安装虚拟环境 """
# 升级 pip
pip install --upgrade pip

# 安装必要的库
sudo pip install virtualenv
sudo pip install virtualenvwrapper

# 配置环境
    1.创建目录用来存放虚拟环境
        mkdir $HOME/.virtualenvs
    2.在~/.bashrc中添加行：
        export WORKON_HOME=$HOME/.virtualenvs
        source /usr/local/bin/virtualenvwrapper.sh
    3.运行:
        source ~/.bashrc

# 查看虚拟机
virtualenv --version

# 新建虚拟环境
mkvirtualenv <虚拟环境名称>

# 查看已存在的虚拟环境
workon

# 使用指定的虚拟环境
workon <虚拟环境名称>

# 退出虚拟环境
deactivate

# 有时候重启终端会出现：找不到 workon 的情况，
source virtualenvwrapper.sh
# 再输入 workon 就可以了

# 删除虚拟环境
rmvirtualenv <虚拟环境名称>

The Getting Started ##
### Installation
- the Install Python the Libraries [Dominate] (https://github.com/Knio/dominate) and Requests.
- The first step to install the necessary libraries

pip install dominate requests

- If you plan to train with face datasets, please install dlib.

- * If you want to train on the face of the data set, install additional libraries

pip install dlib

- If you plan to train with pose datasets, please install [DensePose]

- Training on OpenPose

(https://github.com/facebookresearch/DensePose) and/or [OpenPose](https://github.com/CMU-Perceptual-Computing-Lab/openpose).
- Clone this repo:

- Download Project

git clone https://github.com/NVIDIA/vid2vid
cd vid2vid

Here we all instructions, except otherwise stated, are performed under the "./vid2vid" path.

### test phase

- Docker Image
If you have difficulty building the repo, a docker image can be found in the `docker` folder.

### Testing

- Please first download example dataset by running `python scripts/download_datasets.py`.

- Download sample data set used for testing

python scripts/download_datasets.py

We might as well look at the code for what to do:

import os
from download_gdrive import *

file_id = '1rPcbnanuApZeo2uc7h55OneBkbcFCnnf'
chpt_path = './datasets/'
# 定义路径
if not os.path.isdir(chpt_path):
	os.makedirs(chpt_path)
# 如果不存在此路径，则新建它
destination = os.path.join(chpt_path, 'datasets.zip')
# 定义目标路径
download_file_from_google_drive(file_id, destination) 
# 下载文件到目标路径
unzip_file(destination, chpt_path)
# 解压缩文件

After the download is complete we in the "/ vid2vid" directory will have a new folder "datasets", of which there are three new folders. 

At Cityscapes, for example, we look at the composition of its training test data set. One of the pair should be like this:

训练数据
______________________________________________
train_A        --实例分割mask，对应每一帧
|
train_B        --对应每个mask的真实视频帧（也就是目标生成）
|
train_inst     --区分视频帧的前景（如车、人等）与背景的mask


测试数据
______________________________________________
test_A         --实例分割mask，对应每一帧
|
test_inst      --区分视频帧的前景（如车、人等）与背景的mask

 Among them, we write a function to look at the format of the original video (train_A / test_A) in the video frame.

We can see, each of the mask is one example of a grayscale image segmentation, the object instance described color blocks, wherein different gray value portion of a different object. The following code calculates different values ​​of a pixel on the mask, as follows: 

# ./vid2vid/data_understanding.py
import os
import numpy as np
from PIL import Image

def understanding_cityscape(pth):
    # we will show the different pixel values
    diff_pixels = []
    file = os.listdir(pth)[0]
    img = Image.open(os.path.join(pth, file)).convert('L')
    img_arr = np.array(img)
    H, W = img_arr.shape
    for h in range(H):
        for w in range(W):
            pix = img_arr[h, w]
            if pix not in diff_pixels:
                diff_pixels.append(pix)
    print("There are totally {}/{} different values!".format(len(diff_pixels), H*W))
    print("They are: ")
    print(diff_pixels)
    
    
if __name__ == '__main__':
    pth_cityscape = os.path.join(os.getcwd(), 'datasets', 'Cityscapes', 'train_A', 'seq0000')
    understanding_cityscape(pth_cityscape)

 

- Next, download and compile a snapshot of [FlowNet2](https://github.com/NVIDIA/flownet2-pytorch) by running `python scripts/download_flownet2.py`.

- Here we intend to install FlowNet2 , this should be one of the biggest challenges of this project.

1) Download FlowNet2 project;

python scripts/download_flownet2.py

As a result, in the vid2vid directory will have a new folder "models", inside there is a "flownet2_pytorch" project folder, as shown below:

2) Download FlowNet2 pre-training model

python scripts/download_models_flownet2.py

结果，在“vid2vid/models/flownet2_pytorch”会新增一个“FlowNet2_checkponts.pth.tar”的压缩包，注意我们不需要解压缩！！！

3）编译安装

# flownet2 中的 Readme 文件
# get flownet2-pytorch source
git clone https://github.com/NVIDIA/flownet2-pytorch.git
cd flownet2-pytorch

# install custom layers
bash install.sh

我们看一下“install.sh”这个文件要做什么：

#!/bin/bash
cd ./networks/correlation_package
chmod u+x make.sh
./make.sh
cd ../resample2d_package 
chmod u+x make.sh
./make.sh
cd ../channelnorm_package 
chmod u+x make.sh
./make.sh
cd ..

原来是打算编译这三个文件夹下的C++ CUDA代码。

知道了目的后，我们为了防止出错，可以一个一个来，

① 

cd ./networks/correlation_package
chmod u+x make.sh
./make.sh

在这里我执行到第三个命令的时候出现了以下的问题：

Compiling correlation kernels by nvcc...
rm: cannot remove '../_ext': No such file or directory
Traceback (most recent call last):
  File "build.py", line 3, in <module>
    import torch.utils.ffi
  File "/usr/local/lib/python2.7/dist-packages/torch/utils/ffi/__init__.py", line 1, in <module>
    raise ImportError("torch.utils.ffi is deprecated. Please use cpp extensions instead.")
ImportError: torch.utils.ffi is deprecated. Please use cpp extensions instead.

查了一些文章，说是要使用低于1.0.0版本的pytorch，果然还是打不过版本问题；

如果使用pytorch 0.4的话，还需要安装 cffi，

pip install cffi

② 

cd ./networks/resample2d_package
chmod u+x make.sh
./make.sh

③ 

cd ./networks/channelnorm_package
chmod u+x make.sh
./make.sh

期间可能会提示类似于：

 cannot remove 'ChannelNorm_kernel.o': No such file or directory
rm: cannot remove '../_ext': No such file or directory

不管它，本来就是要删除掉的。

 

- Cityscapes 在CityScapes下测试
- Please download the pre-trained Cityscapes model by:

- 下载预训练模型

- 下载关于道路（street）转换的与训练模型

我们不妨也先看看下载的代码：

# vid2vid/scripts/street/download_models.py
import os
from download_gdrive import *

file_id = '1MKtImgtnGC28EPU7Nh9DfFpHW6okNVkl'
chpt_path = './checkpoints/'
if not os.path.isdir(chpt_path):
	os.makedirs(chpt_path)
destination = os.path.join(chpt_path, 'models.zip')
download_file_from_google_drive(file_id, destination) 
unzip_file(destination, chpt_path)

这整一个过程需要翻墙到Google Drive，所以需要VPN的支持；下载需要时间，此件可以看看其他的论文。

python scripts/street/download_models.py

完了后，我们发现在“.vid2vid”目录下会有一个新的文件夹“checkpoints”，结构如下：

其中，label2city_2048是在多核训练的结果；label2city_single是在单核训练结果。

每个文件夹下又有三个模型（.pth文件） 。

- To test the model (`bash ./scripts/street/test_2048.sh`):

- 测试 Cityscapes 的模型

我们运行脚本：#!./scripts/street/test_2048.sh，其内容是一个Shell上运行Python文件的指令，如下：

python test.py --name label2city_2048 --label_nc 35 --loadSize 2048 --n_scales_spatial 3 --use_instance --fg --use_single_G

# 其中，
# name指明保存文件夹名；
# label_nc指明类别数；
# loadSize是输入输出的图片长边大小；

其结果会保存在：`./results/label2city_2048/test_latest/`中。

 到这里我们就基本完成了项目的“测试劫”。

- We also provide a smaller model trained with single GPU, which produces slightly worse performance at 1024 x 512 resolution.

- 我们也提供了一个更小的模型，处理图片大小为：1024x512.
- Please download the model by

python scripts/street/download_models_g1.py

# 其代码内容如下：
"""
import os
from download_gdrive import *

file_id = '1QoE1p3QikxNVbbTBWWRDtIspg-RcLE8y'
chpt_path = './checkpoints/'
if not os.path.isdir(chpt_path):
	os.makedirs(chpt_path)
destination = os.path.join(chpt_path, 'models_g1.zip')
download_file_from_google_drive(file_id, destination) 
unzip_file(destination, chpt_path)
"""

- 首先下载对应的模型，同样我们还是在街道转换的任务上去做；这之后我们会下载一个新的模型，在路径“checkpoints”中。

- To test the model (`bash ./scripts/street/test_g1_1024.sh`):

- 完了后我们使用这么模型测试一下。运行脚本文件："./scripts/street/test_g1_1024.sh"

python test.py --name label2city_1024_g1 --label_nc 35 --loadSize 1024 --n_scales_spatial 3 --use_instance --fg --n_downsample_G 2 --use_single_G

运行过程我们会发现很快！毕竟size变成了1/4.

结果如下，还是在results目录下，有一个新的文件夹“label2city_1024_g1”。

- You can find more example scripts in the `scripts/street/` directory.
- 你还可以找到更多的脚本。其实没有了，就这两个测试脚本；其他的是训练脚本。

在进入下一步之前，我们来写一个脚本，把输出的图片合成GIF，比较容易观看。

# ./vid2vid/results/gif.py
from PIL import Image
import os
import numpy as np
import imageio


if __name__ == '__main__':
    pth = os.getcwd()
    DIRS = os.listdir(pth)
    
    for DIR in DIRS:
        if os.path.isfile(DIR):
            continue
        path = os.path.join(pth, DIR, 'test_latest', 'stuttgart_00')# target directory
        files = os.listdir(path)
        fakes = []
        reals = []
        for file in files:
            if file[0:4] == 'real':
                reals.append(file)
            elif file[0:4] == 'fake':
                fakes.append(file)
        a = 28
        b = 33
        reals.sort(key= lambda x:int(x[a:b]))
        fakes.sort(key= lambda x:int(x[a:b]))
        
        '''
        print(len(reals))
        print(len(fakse))
        for real in reals:
            print(real)
        for fake in fakes:
            print(fake)
        '''

        L = len(reals)
        frames = []
        for i in range(L):
            
            real = reals[i]
            fake = fakes[i]# 获取文件名
            real = Image.open(os.path.join(path, real)).convert('RGB')
            real = np.array(real)
            fake = Image.open(os.path.join(path, fake)).convert('RGB')
            fake = np.array(fake)   # 读入图片，转换为numpy数组
            
            H, W = real.shape[0], real.shape[1]
            frame = np.zeros((H, 2*W, 3))
            frame[:, 0:W, :] = real
            frame[:, W:2*W, :] = fake   # 数组合并
            frames.append(frame.astype(np.uint8))# 添加
            

            # real = imageio.imread(os.path.join(path, reals[i]))
            # print(real)
            frames.append(frame)
        imageio.mimsave(DIR+'.gif', frames, 'GIF', duration = 0.1)# baocunwei gif tupian 

结果如下，由于图片比较大，这里只上传截图（30MB）。

- Faces

- 下面我们试着进行脸部生成的测试
- Please download the pre-trained model by:

- 首先是下载预训练的模型

python scripts/face/download_models.py

- To test the model (`bash ./scripts/face/test_512.sh`):

- 现在我们尝试测试这个模型，执行脚本文件："#!./scripts/face/test_512.sh"

python test.py --name edge2face_512 --dataroot datasets/face/ --dataset_mode face --input_nc 15 --loadSize 512 --use_single_G

结果将会存储在"./results/edge2face_512/test_latest/"下

同样的，我们可以用前面的脚本生成GIF图，不过有些路径名需要修改；这里只给出一个截图。

- 至于第三个任务，从Pose热值图向真实帧的转换，作者在对应的“vid2vid/scripts/pose”中并没有放置模型下载脚本，所以此处忽略。

至此，我们完成了VID2VID的测试工作。

后面的工作我们在下一篇文章继续跟进。