Transplantation and testing of target tracking algorithm based on YOLACT
1. Initialize the development environment (completed in the development environment based on x86 architecture CPU)
1.1 Initialize the development environment
1. Download the SOPHON SDK development kit
#下载SOPHON SDK
wget https://sophon-file.sophon.cn/sophon-prod-s3/drive/23/06/15/16/Release_230501-public.zip
unzip Release_<Date>-public.zip
cd Release_<Date>-public
cd tpu-mlir_<Date>_xxxx/
2. Configure the Docker container development environment
#如果是首次使用Docker, 可执行下述命令进行安装和配置(仅首次执行):
sudo apt install docker.io
sudo systemctl start docker
sudo systemctl enable docker
sudo groupadd docker
sudo usermod -aG docker $USER
newgrp docker
#从 DockerHub https://hub.docker.com/r/sophgo/tpuc_dev 下载所需的镜像
docker pull sophgo/tpuc_dev:latest
Make sure the installation package is in the current directory, and then create a container in the current directory as follows:
docker run --privileged --name myname -v $PWD:/workspace -it sophgo/tpuc_dev:latest
# myname只是举个名字的例子, 请指定成自己想要的容器的名字
The following assumes that the user is already in the /workspace directory in docker.
1.2 Configure the docker container development environment
- Load tpu-mlir and configure environment variables
tar zxf tpu-mlir_xxxx.tar.gz
source tpu-mlir_xxxx/envsetup.sh
- Download the Yolact algorithm porting code:
git clone https://github.com/sophon-ai-algo/examples.git
# Yolact示例项目代码位置 /examples/simple/yolact
- Download the required data and models through the script:
# 进入项目
cd examples/simple/yolact/
# 执行脚本下载数据和模型
./scripts/download.sh
- Put the image files into the directory and create a working directory:
cp -rf $TPUC_ROOT/regression/dataset/COCO2017 .
cp -rf $TPUC_ROOT/regression/image .
# 创建工作目录
mkdir workspace && cd workspace
Here $TPUC_ROOTis the environment variable, corresponding to the tpu-mlir_xxxx directory.
2. Model conversion (completed in the development environment based on x86 architecture CPU)
2.1 Convert pytorch model to onnx model
Since the yolact source code includes the training part code and the slicing operation, it is necessary to remove the training part and the slicing operation code, and return to features in advance. If you downloaded the source code of yolact before, you should use the modified code yolact.py in the example to replace the yolact.py in the original source code to convert the model.
Since tpu-mlir currently only supports ONNX, TFLite and Caffe models to directly convert F32 and Int8 models, PyTorch, PaddlePaddle and TensorFlow need to be converted to the intermediate format ONNX to convert F32 and Int8 models through MLIR.
Note: TPU-MLIRv23.05.01 already supports direct conversion of pytorch to mlir. How to directly convert pytorch to mlir, please refer to 4. Compile TORCH model - TPU-MLIR 1.1 documentation (sophgo.com) :
#使用示例项目代码中自带的模型转换脚本，可以将pytorch模型转为onnx模型：
python3 ../scripts/converter/convert.py --input ../data/models/yolact_base_54_800000.pth --mode onnx --cfg yolact_base
#移动onnx模型到当前目录
mv ../scripts/converter/yolact_base_54_800000.onnx .
2.2 ONNX to MLIR
# 创建模型转换命令脚本并执行
vi onnx2mlir.sh
sh onnx2mlir.sh
The model conversion command is as follows:
# onnx2mlir.sh中的内容
model_transform.py \
--model_name Yolact \
--model_def ./yolact_base_54_800000.onnx \
--input_shapes [[1,3,550,550]] \
--mean 0.0,0.0,0.0 \
--scale 0.0039216,0.0039216,0.0039216 \
--keep_aspect_ratio \
--pixel_format rgb \
--test_input ../COCO2017/000000000632.jpg \
--test_result yolact_top_outputs.npz \
--mlir yolact.mlir
model_transform.pyThe main parameters are described as follows (for a complete introduction, please refer to the user interface chapter of the TPU-MLIR Development Reference Manual):
| parameter name | required? | illustrate |
|---|---|---|
| model_name | yes | Specify the model name |
| model_def | yes | Specifies the model definition file, such as .onnxor .tfliteor .prototxtfile |
| input_shapes | no | Specify the shape of the input, such as [[1,3,640,640]]; two-dimensional array, which can support multiple input situations |
| resize_dims | no | The size of the original image needs to be resized; if not specified, it will be resized to the input size of the model |
| keep_aspect_ratio | no | Whether to maintain the aspect ratio during Resize, the default is false; when set, the insufficient part will be filled with 0 |
| mean | no | The mean value of each channel of the image, the default is 0.0,0.0,0.0 |
| scale | no | The ratio of each channel of the image, the default is 1.0,1.0,1.0 |
| pixel_format | no | Image type, can be rgb, bgr, gray, rgbd four cases |
| output_names | no | Specify the name of the output, if not specified, the output of the model is used; after specifying, use the specified name as the output |
| test_input | no | Specify the input file for verification, which can be a picture or npy or npz; if it is not specified, the correctness will not be verified |
| test_result | no | Specify the output file after validation |
| excepts | no | Specifies the name of the network layer that needs to be excluded from verification, and multiples are separated by . |
| mlir | yes | Specify the output mlir file name and path |
| post_handle_type | no | Integrate post-processing into the model and specify the type of post-processing, such as yolo, ssd |
The successful operation screen is as follows
After converting to mlir file, a ${model_name}_in_f32.npzfile will be generated, which is the input file of the model.
2.3 MLIR to F32 model
# 创建模型转换命令脚本并执行
vi mlir2bmodel_f32.sh
sh mlir2bmodel_f32.sh
Convert the mlir file to f32 bmodel, the operation method is as follows:
# mlir2bmodel_f32.sh中的内容
model_deploy.py \
--mlir yolact.mlir \
--quantize F32 \
--chip bm1684x \
--tolerance 0.99,0.99 \
--test_input Yolact_in_f32.npz \
--test_reference yolact_top_outputs.npz \
--model Yolact_1684x_f32.bmodel
model_deploy.pyThe main parameters are described as follows (for a complete introduction, please refer to the user interface chapter of the TPU-MLIR Development Reference Manual):
| parameter name | required? | illustrate |
|---|---|---|
| mlir | yes | specify mlir file |
| quantize | yes | Specify the default quantization type, support F32/F16/BF16/INT8 |
| chip | yes | Specify the platform that the model will use, support bm1684x/bm1684/cv183x/cv182x/cv181x/cv180x |
| calibration_table | no | Specify the path of the calibration table, the calibration table is required when there is INT8 quantization |
| tolerance | no | Indicates the error tolerance of the similarity between the MLIR quantized result and the MLIR fp32 inference result |
| test_input | no | Specify the input file for verification, which can be a picture or npy or npz; if it is not specified, the correctness will not be verified |
| test_reference | no | Reference data (in npz format) used to verify the correctness of the model. It is the calculation result of each operator |
| compare_all | no | Whether to compare all intermediate results when verifying correctness, the default is not to compare intermediate results |
| excepts | no | Specifies the name of the network layer that needs to be excluded from verification, and multiples are separated by . |
| fuse_preprocess | no | Whether to put preprocessing into the model, currently only supports CV18xx series chips, which will be introduced in the following chapters |
| customization_format | no | Specifies the image format input to the model, which is related to preprocessing and generally does not need to be specified |
| aligned_input | no | Whether to align the input data, only supports CV18xx series chips, which will be introduced in the following chapters |
| model | yes | Specify the output model file name and path |
The successful operation screen is as follows
After the compilation is complete, ${model_name}_1684x_f32.bmodela file named will be generated.
2.4 MLIR to INT8 model
2.4.1 Generate Calibration Table
Before converting to INT8 model, you need to run calibration to get the calibration table; the number of input data is prepared according to the situation. About 100~1000 sheets.
Then with the calibration table, a symmetric or asymmetric bmodel is generated. If the symmetry meets the requirements, it is generally not recommended to use the asymmetric model, because the performance of the asymmetric model will be slightly worse than that of the symmetric model.
Here we use the existing 100 pictures from COCO2017 as an example to perform calibration, which may take a long time to wait:
vi run_cali.sh
sh run_cali.sh
The content of run_cali.sh is as follows:
run_calibration.py yolact.mlir \
--dataset ../COCO2017 \
--input_num 100 \
-o yolact_cali_table
The successful operation screen is as follows:
${model_name}_cali_tableAfter the operation is completed , a file named will be generated , which is used as the input file for subsequent compilation of the INT8 model.
2.4.2 Compile to INT8 symmetric quantization model
# 创建模型转换命令脚本并执行
vi mlir2bmodel_int8_sym.sh
sh mlir2bmodel_int8_sym.sh
The command to convert to INT8 symmetric quantization model is as follows:
# mlir2bmodel_int8_sym.sh内容如下
model_deploy.py \
--mlir yolact.mlir \
--quantize INT8 \
--calibration_table yolact_cali_table \
--chip bm1684x \
--test_input Yolact_in_f32.npz \
--test_reference yolact_top_outputs.npz \
--tolerance 0.85,0.45 \
--model Yolact_1684x_int8_sym.bmodel
The successful operation screen is as follows:
After the compilation is complete, ${model_name}_1684x_int8_sym.bmodela file named will be generated.
2.4.3 Compile to INT8 asymmetric quantization model
# 创建模型转换命令脚本并执行
vi mlir2bmodel_int8_asym.sh
sh mlir2bmodel_int8_asym.sh
The command to convert to INT8 symmetric quantization model is as follows:
# mlir2bmodel_int8_asym.sh内容如下
model_deploy.py \
--mlir yolact.mlir \
--quantize INT8 \
--asymmetric \
--calibration_table yolact_cali_table \
--chip bm1684x \
--test_input Yolact_in_f32.npz \
--test_reference yolact_top_outputs.npz \
--tolerance 0.90,0.55 \
--model Yolact_1684x_int8_asym.bmodel
Successful operation screen:
After the compilation is complete, ${model_name}_1684x_int8_asym.bmodela file named will be generated.
3. Yolact model reasoning test
3.1 Prepare sample program
Move the three files Yolact_1684x_f32.bmodel, Yolact_1684x_int8_asym.bmodel, Yolact_1684x_int8_sym.bmodel to /data/models
# 退回至yolact目录
mv Yolact_1684x_f32.bmodel Yolact_1684x_int8_asym.bmodel Yolact_1684x_int8_sym.bmodel ../data/models
cd ../python
3.2 Deployment test
# sail解码 + bmcv预处理 + sail推理 + opencv后处理
# 下面的model参数Yolact_1684x_int8_asym.bmodel、Yolact_1684x_int8_sym.bmodel
python3 yolact_bmcv.py --cfgfile configs/yolact_base.cfg --model ../data/models/Yolact_1684x_f32.bmodel --is_video 0 --input_path ../image/
# 执行完毕后，在当前目录生成result bmcv文件夹，检测结果保存在该文件夹下。
# opencv解码 + opencv预处理 + sail推理 + opencv后处理
# 下面的model参数Yolact_1684x_int8_asym.bmodel、Yolact_1684x_int8_sym.bmodel
python3 yolact_sail.py --cfgfile configs/yolact_base.cfg --model ../data/models/Yolact_1684x_f32.bmodel --is_video 0 --input_path ../image/
# 执行完毕后，在当前目录生成result_cv文件夹，检测结果保存在该文件夹下。
3.3 Effect comparison
3.3.1 Inference results of f32 model
3.3.2 Inference results of int8 symmetric quantization model
3.3.3 inference results of int8 asymmetric quantization model
