foreword

Goal: C++ Deploying Deep Model
Current Level: Able to speak Python, Pytorch, and C

1. Download

TensorRT download address: https://developer.nvidia.com/zh-cn/tensorrt
Official website version corresponding version number (2022.9.7):
TensorRT 8.4 GA Update 2: 8.4.3.1
TensorRT 8.4 GA Update 1: 8.4.2.4
TensorRT 8.4 GA: 8.4.1.5
TensorRT 8.4 EA: 8.4.0.6
Select to download TensorRT 8.4 GA Update 2 for Linux x86_64 and CUDA 11.0, 11.1, 11.2, 11.3, 11.4, 11.5, 11.6 and 11.7 TAR Package

1.1 Relevant Information Inquiry

Graphics card model $\to$ Graphics driver $\to$ CUDA version $\to$ cuDNN version
Existing things will not move if they can’t be moved, and the graphics card driver system can’t get in even after changing the graphics card driver system.

Graphics card driver query and download
CUDA version and graphics card driver correspondence table
CUDA download address
cuDNN download address
Check the graphics card model: lspci | grep -i vga(NVIDIA Corporation GP104 [GeForce GTX 1070 Ti] (rev a1)) Graphics card ID query website
Check the graphics card driver version: nvidia-smi(Driver Version: 470.82.01, according to the table, only CUDA <=11.4 can be installed)
Check the operating system kernel: uname -a(x86_64)
Check the operating system version: cat /proc/version(18.04.1-Ubuntu)
Check the cuda version: nvcc -V(11.4.152)
Check the cudnn version: cat /usr/local/cuda/include/cudnn.h | grep CUDNN_MAJOR -A 2, no output, the reason is that the cudnn version is newer, and the version number is not in cudnn.h, cat /usr/local/cuda/include/cudnn_version.h | grep CUDNN_MAJOR -A 2(8.3.2)

"""
#define CUDNN_MAJOR 8
#define CUDNN_MINOR 3
#define CUDNN_PATCHLEVEL 2
--
#define CUDNN_VERSION (CUDNN_MAJOR * 1000 + CUDNN_MINOR * 100 + CUDNN_PATCHLEVEL)
"""

2. Install

2.1 Common installation methods

Install according to the online documentation , click NVIDIA TensorRT Installation Guide to enter the installation guide , which provides a variety of installation methods:
(1) Debian and RPM installation:

Automatically install dependencies
Requires sudo or root privileges
Cannot customize TensorRT installation location
CUDA toolkit and cuDNN are also installed using Debian or RPM
Cannot install multiple versions of TensorRT

(2) tar file

Multiple versions of TensorRT can be installed at the same time
Install the necessary dependencies yourself
self managementLD_LIBRARY_PATH

(3) zip file

The only way to install Windows
Does not support any other platform than Windows
Install the necessary dependencies yourself

2.2 Linux: tar file installation

(1) Unzip
tar -xzvf TensorRT-8.4.3.1.Linux.x86_64-gnu.cuda-11.6.cudnn8.4.tar.gz

(2) Add environment variables
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:<TensorRT-${version}/lib>. This method is only valid in the current terminal. You can choose to ~/.bashrcadd environment variables in :
Open the terminal input: vim ~/.bashrc
Click to ienter insert mode, insert at the end of the file to
export LD_LIBRARY_PATH=/home/jesswork/app/TensorRT-8.4.3.1/lib:$LD_LIBRARY_PATH
Escexit insert mode, enter to :wqsave and exit

(3) Install various Python files

First create a virtual environment
conda create -n tensorrt python=3.7
conda activate tensorrt
TensorRT wheel (select the corresponding Python version, here is cp37)
cd TensorRT-8.4.3.1/python
python3 -m pip install tensorrt-8.4.3.1-cp37-none-linux_x86_64.whl
UFF wheel (only installed with TensorRT on TensorFlow)
cd TensorRT-8.4.3.1/uff
python3 -m pip install uff-0.6.9-py2.py3-none-any.whl
graphsurgeon wheel
cd TensorRT-8.4.3.1/graphsurgeon
python3 -m pip install graphsurgeon-0.4.6-py2.py3-none-any.whl
onnx-graphsurgeon wheel
cd TensorRT-8.4.3.1/onnx_graphsurgeon
python3 -m pip install onnx_graphsurgeon-0.3.12-py2.py3-none-any.whl

2.3 Windows: zip file installation (will update again)

3. Test

https://github.com/NVIDIA/TensorRT , convenient to view the readme of each example

3.1 sampleMNIST

TensorRT-8.4.3.1/samples/sampleMNISTRun under the path , makeand then TensorRT-8.4.3.1/binrun under the path ./sample_mnist, you can see the running results.

3.2 network_api_pytorch_mnist

This example trains a model in PyTorch, recreates the network in TensorRT, imports weights from the trained model, and finally runs inference using the TensorRT engine.

Here is not according to the configuration environment in the official document python3 -m pip install -r requirements.txt, because it is the installed CPU version of Pytorch. Select Pytorchv1.10.0 with CUDA 11.3 installed, Pytorch installation
conda install pytorch==1.10.0 torchvision==0.11.0 torchaudio==0.10.0 cudatoolkit=11.3 -c pytorch -c conda-forge

TensorRT-8.4.3.1/samples/python/network_api_pytorch_mnistJust run it under the path python3 sample.py.

运行问题：
（1）UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors.
发生在 return torch.from_numpy(parsed.astype(m[2], copy=False)).view(*s)，只需要把 copy=False 去掉即可。

（2）TensorRT was linked against cuDNN 8.4.1 but loaded cuDNN 8.3.2
由于电脑 CUDA 环境是装好的，安装过程中发现了 cuDNN 版本和要求的不一致但没重装，示例代码能运行暂时不管了。

（3）后续碰到了个坑 torch.cuda.is_available() 是 False，最后发现是装的时候直接 yes 没注意是 cpu 版的 pytorch，卸载重装也不行，最后开了个新的虚拟环境先装 pytorch 再装 tensorrt 就好了。

3.3 pytorchx & tensorrtx

一般转模型的时候会通过 pytorch $\to$ onnx $\to$ tensorrt，但可能会碰到一些不支持的操作或者层。这项工作将训练好的 pytorch 模型权重导出到 wts 文件，并直接用 tensorrt 定义网络并加载权重。
https://github.com/wang-xinyu/pytorchx
https://github.com/wang-xinyu/tensorrtx

3.3.1 lenet

按照给出的 lenet5 demo 进行测试，由于是用 tar 安装的 tensorrt，只需修改一下 CMakeLists.txt，其余按指定步骤操作即可。

# tensorrt
include_directories(/usr/include/x86_64-linux-gnu/)
link_directories(/usr/lib/x86_64-linux-gnu/)
'修改为'
# tensorrt
include_directories(安装目录/TensorRT-8.4.3.1/include)
link_directories(安装目录/TensorRT-8.4.3.1/lib)

3.3.2 yolov5

同样按教程操作并修改 CMakeLists.txt 即可。

到这两行报错： ./yolov5: error while loading shared libraries: libinference_engine.so: cannot open shared object file: No such file or directory，解决办法是去掉 sudo，原因可能是一些中间文件的权限问题。

sudo ./yolov5 -s yolov5s.wts yolov5s.engine s
sudo ./yolov5 -d yolov5s.engine ../samples

TensorRT learning (1): download, install, test