【服务器】NVIDIA-SMI has failed because it couldn‘t communicate with the NVIDIA driver.

Recently, when running the model, I found that the code suddenly stopped running, and no exception was thrown or exited. After restarting the server, it was found that nvidia-smithe command would report an error.

NVIDIA-SMI has failed because it couldn‘t communicate with the NVIDIA driver.

solution:

1. ls /usr/src | grep nvidia
   Here my driver version is nvidia-510.73.05
2. sudo apt-get install dkms
3. sudo dkms install -m nvidia -v 510.73.05

It will return to normal after the installation is complete. Acceleration for this command
can also be used .nvidia-smi -pm 1