使用NVIDIA产品进行量化后推理必须使用TensorRT才能加速，否则会默认使用CPU处理，比TensorFlow-GPU推理还慢。这篇文章记录如何在Windows上配置TensorFlow+TensorRT。

基本思路

网上查到：先安装CUDA+cuDNN，下载对应版本的TensorRT将lib和include文件夹下的文件移到CUDA文件下覆盖即可。

安装过程

实际操作我是先下载了TensorRT-8.4.0.6.Windows10.x86_64.CUDA-11.6.cuDNN8.3.zip包，对应CUDA 11.6和cuDNN 8.3，但是离谱的是在cuDNN官网上找不到适用CUDA 11.6版本的cuDNN，只好下了一个cuDNN 8.4 for CUDA 11.x，最后发现也能行。

根据NVIDIA官方文档，需要将下载后的TensorRT lib添文件夹加到系统路径（后面把include文件夹也添加扔进去了，不知道实际上需不需要），有两种方法：

去系统路径添加lib文件夹
将lib文件夹复制到 C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\vX.Y\bin目录下
打开python子目录，安装tensorrt的wheel

install_tensorrt

在python内导入tensorrt，出现错误FileNotFoundError: Could not find: nvinfer.dll. Is it on your PATH?

(base) C:\WINDOWS\system32>python
Python 3.9.12 (main, Apr  4 2022, 05:22:27) [MSC v.1916 64 bit (AMD64)] :: Anaconda, Inc. on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensorrt
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\11197\Miniconda3\lib\site-packages\tensorrt\__init__.py", line 127, in <module>
    ctypes.CDLL(find_lib(lib))
  File "C:\Users\11197\Miniconda3\lib\site-packages\tensorrt\__init__.py", line 91, in find_lib
    raise FileNotFoundError(
FileNotFoundError: Could not find: nvinfer.dll. Is it on your PATH?
Note: Paths searched were:
['C:\\Users\\11197\\Miniconda3', 'C:\\Users\\11197\\Miniconda3\\Library\\mingw-w64\\bin', 'C:\\Users\\11197\\Miniconda3\\Library\\usr\\bin', 'C:\\Users\\11197\\Miniconda3\\Library\\bin', 'C:\\Users\\11197\\Miniconda3\\Scripts', 'C:\\Users\\11197\\Miniconda3\\bin', 'C:\\Users\\11197\\miniconda3\\condabin', 'C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v11.6\\bin', 'C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v11.6\\libnvvp', 'C:\\WINDOWS\\system32', 'C:\\WINDOWS', 'C:\\WINDOWS\\System32\\Wbem', 'C:\\WINDOWS\\System32\\WindowsPowerShell\\v1.0', 'C:\\WINDOWS\\System32\\OpenSSH', 'C:\\Program Files (x86)\\NVIDIA Corporation\\PhysX\\Common', 'C:\\Program Files\\Microsoft VS Code\\bin', 'C:\\Program Files\\Git\\cmd', 'C:\\Program Files\\dotnet', 'C:\\Program Files\\nodejs', 'C:\\Program Files\\NVIDIA Corporation\\Nsight Compute 2022.1.0', 'C:\\Program Files\\NVIDIA Corporation\\NVIDIA NvDLISR', 'C:\\Users\\11197\\AppData\\Local\\Programs\\Python\\Python39\\Scripts', 'C:\\Users\\11197\\AppData\\Local\\Programs\\Python\\Python39', 'C:\\Users\\11197\\AppData\\Local\\Microsoft\\WindowsApps', 'C:\\Users\\11197\\.dotnet\\tools', 'C:\\Users\\11197\\AppData\\Roaming\\npm']
>>>

打开TensorRT的__init__.py文件，发现find_lib函数如下：

def find_lib(name):

    paths = os.environ["PATH"].split(os.path.pathsep)

    for path in paths:

        libpath = os.path.join(path, name)

        if os.path.isfile(libpath):

            return libpath



    raise FileNotFoundError(

        "Could not find: {:}. Is it on your PATH?\nNote: Paths searched were:\n{:}".format(name, paths)

    )

所有的lib都是在“Path”名称的变量下搜索的，因此在里面添加nvinfer.dll所在的目录

envsetting

安装完成！

顺便安装一下pycuda，之后会用到。

(base) C:\WINDOWS\system32>pip install pycuda
Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
Collecting pycuda
  Downloading pycuda-2021.1.tar.gz (1.7 MB)
     |████████████████████████████████| 1.7 MB 1.1 MB/s
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Requirement already satisfied: appdirs>=1.4.0 in c:\users\11197\miniconda3\lib\site-packages (from pycuda) (1.4.4)
Requirement already satisfied: pytools>=2011.2 in c:\users\11197\miniconda3\lib\site-packages (from pycuda) (2022.1.9)
Requirement already satisfied: mako in c:\users\11197\miniconda3\lib\site-packages (from pycuda) (1.2.0)
Requirement already satisfied: platformdirs>=2.2.0 in c:\users\11197\miniconda3\lib\site-packages (from pytools>=2011.2->pycuda) (2.5.2)
Requirement already satisfied: typing_extensions>=4.0 in c:\users\11197\miniconda3\lib\site-packages (from pytools>=2011.2->pycuda) (4.1.1)
Requirement already satisfied: MarkupSafe>=0.9.2 in c:\users\11197\miniconda3\lib\site-packages (from mako->pycuda) (2.1.1)
Building wheels for collected packages: pycuda
  Building wheel for pycuda (PEP 517) ... done
  Created wheel for pycuda: filename=pycuda-2021.1-cp39-cp39-win_amd64.whl size=366136 sha256=51aa63305e507b7b54e47e1656f8ac8ee07f7311a55ec3266987f98ec67adb96
  Stored in directory: C:\Users\11197\AppData\Local\Temp\pip-ephem-wheel-cache-ss72konn\wheels\8e\e2\ce\6dbdd116792fcc3b7f5732eba784df2dbd8467a6c10f1592f5
Successfully built pycuda
Installing collected packages: pycuda
Successfully installed pycuda-2021.1

猪老大要进步！

TensorRT--Windows下安装

基本思路

安装过程

参考文献