NVIDIA official website released tensorRT 7.0. For model server deployment, tensorRT is really good. Put the official website link first.
https://docs.nvidia.com/deeplearning/sdk/tensorrt-developer-guide/index.html
First introduce TensorRT
The core of TensorRT is a deep learning high-performance C++ inference library, which is basically adapted to mainstream training frameworks, such as Tensorflow, Caffe, Pytorch, MXNet.
Linux installation reference: [TensorRT] tensorRT 7.0 installation configuration
Windows installation reference: Installation and testing of TensorRT 7.0 in Win10+VS2017 environment
1. Installation steps
I used TensorRT version 5.0 + Cuda 10.0 + cuDNN 7.3.1, now try TensorRT 7.0
The installation steps are as follows:
1) Download the installation package from the official website: https://developer.nvidia.com/tensorrt
2) Unzip the installation package
3) Add the corresponding installation directory and the corresponding bin and lib directories to the system path macro definition
4) Ensure that the directory where the corresponding cuda version is located and the bin directory of cuda are in the system path. I have done this step before
Two, test
Test the sample_mnist that comes with TensorRT 7.0. It compiles correctly, but exits abnormally during runtime. The handling is as follows:
1) Add debug information (only Release version in the example), locate the fault point, check whether the file cannot be found, etc.-the result is not, it is dead on the call of builder->createBuilderConfig()
2) Check the Cuda version requirements of TensorRT 7.0. The version required for my download package is: Cuda10.0+cuDNN7.6, so upgrade cuDNN 7.3.1 to the corresponding version (7.6)-still not working
3) Update the graphics card driver to the latest version (my graphics card is Geforce GTX 1650, download it from the NVidia website)-still not working
4) Considering whether the TensorRT version is too high and many environments do not support it, install TensorRT 6.0 and test it-still dead in the same place
5) Update Cuda from 10.0 to 10.2, the problem is solved-TensorRT 6.0 & 7.0 are correct
6) When using cuDNN version 7.3.1, there will be a warning when running, but the result is still correct. The warning disappeared after updating cuDNN version to 7.6.4.
7) Although the running result is correct, but observing the DLL loaded during the running of the program, it is found that both cuda10.0 and cuda10.2 DLL libraries are loaded. Open the program link prompt, and indeed only the lib library of cuda10.2 is loaded during the program linking process. It should be that the corresponding TensorRT version is compiled under cuda10.0, so use the corresponding library. After re-downloading the cuda10.2 version of TensorRT, the cuda10.0 library is no longer loaded.
The above is for reference only, the following is the actual operation:
Find the version you want to download from the download address.
Download link: https://developer.nvidia.com/nvidia-tensorrt-7x-download
Development documentation: https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html
After downloading TensorRT-7.0.0.11.Windows10.x86_64.cuda-10.0.cudnn7.6.zip, unzip it, and then use vs2017 to open sample_mnist.sln (because vc141 is used, the default for vs2015 is vc140, and the default for vs2017 is vc141, so use vs2017 open the project, otherwise the compilation will not pass.)
Since I have installed cuda10.0 and cudnn7.6.5 before, they can be used directly. Because I also installed cuda10.2 and cudnn7.6.5, so I checked the dependencies after compilation and cuda10.2. After all, I just started to use vs2017, I don't know where cuda is configured, vs2015 can directly configure which cuda is used, vs2017 did not find where to configure.
>dumpbin /dependents sample_mnist.exe
Microsoft (R) COFF/PE Dumper Version 14.16.27042.0
Copyright (C) Microsoft Corporation. All rights reserved.
Dump of file sample_mnist.exe
File Type: EXECUTABLE IMAGE
Image has the following dependencies:
nvinfer.dll
nvparsers.dll
cudart64_102.dll
MSVCP140.dll
VCRUNTIME140.dll
api-ms-win-crt-runtime-l1-1-0.dll
api-ms-win-crt-heap-l1-1-0.dll
api-ms-win-crt-utility-l1-1-0.dll
api-ms-win-crt-convert-l1-1-0.dll
api-ms-win-crt-stdio-l1-1-0.dll
api-ms-win-crt-filesystem-l1-1-0.dll
api-ms-win-crt-time-l1-1-0.dll
api-ms-win-crt-math-l1-1-0.dll
api-ms-win-crt-environment-l1-1-0.dll
api-ms-win-crt-string-l1-1-0.dll
api-ms-win-crt-locale-l1-1-0.dll
KERNEL32.dll1. Generate sample_mnist.exe in the TensorRT-7.0.0.11\bin directory after compilation.
2. Copy the dynamic library in the TensorRT-7.0.0.11\lib directory to the TensorRT-7.0.0.11\bin directory.
3. Download mnist data: open powershell and execute it in the TensorRT-7.0.0.11\data\mnist directory
python .\download_pgms.py4. Under powershell, return to the root directory\TensorRT-7.0.0.11 folder and execute the program:
> .\bin\sample_mnist.exe
&&&& RUNNING TensorRT.sample_mnist # D:\Program Files\TensorRT-7.0.0.11.Windows10.x86_64.cuda-10.0.cudnn7.6\TensorRT-7.0.0.11\bin\sample_mnist.exe
[09/24/2020-22:13:31] [I] Building and running a GPU inference engine for MNIST
[09/24/2020-22:13:33] [I] [TRT] Detected 1 inputs and 1 output network tensors.
[09/24/2020-22:13:33] [W] [TRT] Current optimization profile is: 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles
[09/24/2020-22:13:33] [I] Input:
@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@@@@@@@@@@@@#=. +*=#@@@@@@@
@@@@@@@@@@@* :. -@@@@@@@
@@@@@@@@@@# :#@@: +@@@@@@@
@@@@@@@@@* :@@@* .@@@@@@@@
@@@@@@@@= =@@@@. *@@@@@@@@
@@@@@@@= -@@@@* =@@@@@@@@@
@@@@@@@ -@@@%: -@@@@@@@@@@
@@@@@@% %%+: *@@@@@@@@@@
@@@@@@@ .. @@@@@@@@@@@
@@@@@@@# .=%%: =@@@@@@@@@@@
@@@@@@@@@@@@@# +@@@@@@@@@@@
@@@@@@@@@@@@@# @@@@@@@@@@@@
@@@@@@@@@@@@@@ @@@@@@@@@@@@
@@@@@@@@@@@@@# @@@@@@@@@@@@
@@@@@@@@@@@@@+ @@@@@@@@@@@@
@@@@@@@@@@@@@% @@@@@@@@@@@@
@@@@@@@@@@@@@@. #@@@@@@@@@@@
@@@@@@@@@@@@@@* :%@@@@@@@@@@
@@@@@@@@@@@@@@@: -@@@@@@@@@@
@@@@@@@@@@@@@@@@= %@@@@@@@@@
@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[09/24/2020-22:13:33] [I] Output:
0:
1:
2:
3:
4:
5:
6:
7:
8:
9: **********
&&&& PASSED TensorRT.sample_mnist # D:\Program Files\TensorRT-7.0.0.11.Windows10.x86_64.cuda-10.0.cudnn7.6\TensorRT-7.0.0.11\bin\sample_mnist.exe