openvino 将onnx转为IR并进行int8量化
环境
- Ubuntu 22.04
- python 3.10
安装环境
sudo apt-get update
sudo apt-get upgrade
sudo apt-get install python3-venv build-essential python3-dev git-all -y
sudo apt-get install intel-opencl-icd -y
编译 mo
下载 openvino
git clone https://github.com/openvinotoolkit/openvino
编译 mo
cd openvino/tools
python3 -m pip install mo
编译成功输出如下信息：
Collecting mo
Downloading mo-0.3.0-py2.py3-none-any.whl (12 kB)
Requirement already satisfied: PyYAML in /usr/local/lib/python3.10/dist-packages (from mo) (6.0)
Collecting colorama (from mo)
Downloading colorama-0.4.6-py2.py3-none-any.whl (25 kB)
Requirement already satisfied: toml in /usr/local/lib/python3.10/dist-packages (from mo) (0.10.2)
Installing collected packages: colorama, mo
Successfully installed colorama-0.4.6 mo-0.3.0
安装
pip install openvino-dev
查看帮助
mo -h
onnx 转为 IR
mo --input_model onnx/model.onnx --compress_to_fp16 --output_dir ir_model
这里压缩为fp16
输出信息：
[ INFO ] Generated IR will be compressed to FP16. If you get lower accuracy, please consider disabling compression explicitly by adding argument --compress_to_fp16=False.
Find more information about compression to FP16 at https://docs.openvino.ai/2023.0/openvino_docs_MO_DG_FP16_Compression.html
[ INFO ] The model was converted to IR v11, the latest model format that corresponds to the source DL framework input/output format. While IR v11 is backwards compatible with OpenVINO Inference Engine API v1.0, please use API v2.0 (as of 2022.1) to take advantage of the latest improvements in IR v11.
Find more information about API v2.0 and IR v11 at https://docs.openvino.ai/2023.0/openvino_2_0_transition_guide.html
[ SUCCESS ] Generated IR version 11 model.
[ SUCCESS ] XML file: /workspace/bert/ir_model/model.xml
[ SUCCESS ] BIN file: /workspace/bert/ir_model/model.bin
转换成功了！
输出文件对比：
# ls -lh ir_model/
total 321M
-rw-r--r-- 1 root root 319M Sep 22 11:27 model.bin
-rw-r--r-- 1 root root 1.8M Sep 22 11:27 model.xml
# ls -lh onnx/
total 640M
-rw-r--r-- 1 root root 640M Sep 21 20:23 model.onnx
IR 模型量化为 int8
编译 Post-Training Optimization Tool
cd openvino/tools/pot/
python3 setup.py install
bert模型量化步骤参考：
https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/105-language-quantize-bert
Quantization of Image Classification model参考：
https://github.com/openvinotoolkit/openvino_notebooks/tree/main/notebooks/301-tensorflow-training-openvino
参考
- https://github.com/openvinotoolkit/openvino_notebooks#-installation-guide
- https://docs.openvino.ai/2022.3/notebooks/102-pytorch-onnx-to-openvino-with-output.html
- https://github.com/openvinotoolkit/openvino/tree/master/tools/mo
- https://github.com/openvinotoolkit/openvino
- https://github.com/openvinotoolkit/openvino/tree/master/tools/pot