Jizhi AI | Teach you to use the deep learning model debugger polygraphy

Get into the habit of writing together! This is the 20th day of my participation in the "Nuggets Daily New Plan · June Update Challenge", click to view the details of the event .

欢迎关注我的公众号 [极智视界]，获取我的更多笔记分享

Hello everyone, I am Jizhi Vision. This article explains how to use the deep learning model debugger polygraphy.

For the deployment of deep learning models, it often involves conversion between multiple frameworks, generally training frameworks (PyTorch, TensorFlow...) => inference frameworks (TensorRT, OnnxRuntime...), each framework has a different model So the most important thing in this process is not to lose precision after conversion. However, the ideal is often very full, and the reality is not full. When the accuracy is lost, we always want to have a useful tool to help us locate where the problem is. At this time, polygraphy is ready to come out.

1 Introduction to polygraphy

polygraphy is a deep learning model debugging tool, including python API and command line tools . There are relatively few related introductions about polygraphy. Some of its functions are as follows:

Run inference calculations using multiple backends, including TensorRT, onnxruntime, TensorFlow;
Compare the layer-by-layer calculation results of different backends;
The TensorRT engine is generated by the model Enjian and serialized into .plan;
View the layer-by-layer information of the model network;
Modify the Onnx model, such as extracting subgraphs and simplifying the calculation graph;
Analyze the reasons for the failure of Onnx to TensorRT, and save the subgraphs that can/cannot be transferred to TensorRT in the original calculation graph;
Isolate TensorRT terminal error tactic;

The installation method of polygraphy is relatively simple, you can directly install it with pip:

pip install -i https://pypi.douban.com/simple nvidia-pyindex 
pip install -i https://pypi.douban.com/simple polygraphy

2 Polygraphy usage example

Here is an example of polygraphy used to compare the accuracy of onnxruntime and TensorRT. The process is almost like this:

First generate a .onnx file;
Secondly, use polygraphy to generate an FP16 TRT engine, and compare the calculation results using onnxruntime and TensorRT;
Then use polygraphy to generate an FP32 TRT engine, mark all layers in the network as outputs, and compare the calculation results using onnxruntime and TensorRT (layer-by-layer result comparison);

The relevant code is shown as follows:

# 生成一个 .onnx 模型作为 polygraphy 的输入
# export model.onnx from pytorch
# or
# export model.onnx from tensorflow

# 使用上面生成的 model.onnx 构建 TensorRT 引擎，使用 FP16 精度同时在 TensorRT 和 onnxruntime 中运行
polygraphy run model.onnx \
    --onnxrt --trt \
    --workspace 100000000 \
    --save-engine=model_FP16.plan \
    --atol 1e-3 --rtol 1e-3 \
    --fp16 \
    --verbose \
    --trt-min-shapes 'x:0:[1,1,28,28]' \
    --trt-opt-shapes 'x:0:[4,1,28,28]' \
    --trt-max-shapes 'x:0:[16,1,28,28]' \
    --input-shapes 'x:0:[4,1,28,28]' \
    > result-run-FP16.txt

# 使用上面生成的 model.onnx 构建 TensorRT 引擎，使用 FP32 精度同时在 TensorRT 和 onnxruntime 中运行
# 输出所有层的计算结果作对比
polygraphy run model.onnx \
    --onnxrt --trt \
    --workspace 100000000 \
    --save-engine=model_FP32_MarkAll.plan \
    --atol 1e-3 --rtol 1e-3 \
    --verbose \
    --onnx-outputs mark all \
    --trt-outputs mark all \
    --trt-min-shapes 'x:0:[1,1,28,28]' \
    --trt-opt-shapes 'x:0:[4,1,28,28]' \
    --trt-max-shapes 'x:0:[16,1,28,28]' \
    --input-shapes 'x:0:[4,1,28,28]' \
    > result-run-FP32-MarkAll.txt

Select the FP16 inference comparison results to display, as follows:

Well, the above sharing teaches you to use the deep learning model debugger polygraphy. I hope my sharing can help you a little in your study.

【Public number transmission】

"Extremely Intelligent AI | Teach you to use the deep learning model debugger polygraphy"