torch.backends.cudnn.benchmark = true

Setting this flag allows the built-in auto-tuner cuDNN efficient algorithm to automatically find the most suitable for the current configuration, to solve the problem to optimize operating efficiency.

1. If the input data on the type or dimension of the network is not changed, provided torch.backends.cudnn.benchmark = true operating efficiency can be increased;
2. If the input data networks in each iteration are changes, it will lead to cuDNN every time again to find the optimal configuration, but this will reduce the operating efficiency.