Problem Description:
The server was reinstalled with Ubuntu 22.04 system and has 10 graphics cards. Using nvidia-smi showed that there were only 9 graphics cards. One graphics card disappeared and reinstalling the driver did not solve the problem. Reference blog: (600 messages) ubuntu 18.04 Two GPU graphics cards, nvidia-smi only displays one_nvidia-smi only displays one graphics card_Jason.su.ai's blog-CSDN blog
1. Use the lspci |grep NVIDIA command to see if there is a problem with the physical connection of the graphics card.
You can see that all 10 graphics cards can be displayed, indicating that there is no problem with the connection.
2. Use the command ls -l /dev/nvidia* to check whether the nvidia driver is normal.
You can see that the drivers for all 10 graphics cards are normal.
3. Use echo "hello" > /dev/nvidia0 to check whether the communication is normal.
You can see that there is an error in reading and writing on the sixth graphics card. If this happens, it should be that the graphics card is broken. If one is missing, then one is missing. The problem is not solved, please update after it is solved!