Ubuntu builds a Pytorch environment (Anaconda, Cuda, cuDNN, Pytorch, Python, Pycharm, Jupyter)

1. Configure the mirror source and install the necessary environment

1. Check the Ubuntu version number: cat /etc/issue, and then add the corresponding mirror source according to the version number
2. Backup the mirror source: sudo cp /etc/apt/sources.list /etc/apt/sources.list.bak
3. Find and copy the Alibaba mirror source corresponding to the version number on the official website of the Alibaba Cloud mirror source :

Or find and copy the Tsinghua image source corresponding to the version number on the Tsinghua image source official website :

4. Edit the software source configuration file: sudo vi /etc/apt/sources.list, replace the content in the file with the Ali mirror source or Tsinghua mirror source
5. Refresh the software source: sudo apt-get update
6. Install the tool library for compiling and building software:sudo apt-get install build-essential

2. Anaconda download, installation, uninstallation and environment configuration

1. Download Anaconda

Enter Anaconda official website , download the linux version of anaconda (file ending in .sh)

2. Install Anaconda

1. Copy the file to Ubuntu, and enter the command in the corresponding location sh 文件名.shto install (it is recommended to install under the root user)

2. Keep pressing the ENTER key until the page for selecting yes and no appears, and enter yes on the keyboard

3. Select the installation location and enter the location you want to install (Note: It is best to install anaconda in a directory that ordinary users can access, such as /usr/local, /opt, /home, otherwise ordinary users may not be able to use it normally )

If you only need to use anaconda under the root account, just press ENTER to select the default location. I installed it under the default path /root/anaconda3

4. Initialize the conda environment: Enter yes on the keyboard to automatically configure the environment; enter no on the keyboard to customize the configuration environment later.

3.Anaconda environment configuration

1. Find the installation location of Anaconda. There will be a prompt for the default installation, which is usually home/username/anaconda3. I installed it under /root/anaconda3 this time. 2. Enter and edit the environment

variables vi ~/.bashrcand add the following content:

export PATH="/自己对应的路径名/anaconda3/bin:$PATH"

Write your corresponding path name here, mine is export PATH="/root/anaconda3/bin:$PATH"

3. Activate the modified content: source ~/.bashrc
4. Test whether the configuration is successful: input conda, if it does not display not fond, it means that the anaconda installation and configuration is successful

4. Give ordinary users/super users permission to use Anaconda

Regardless of whether it is installed under the root user or an ordinary user, it is best to install anaconda in a directory that ordinary users can access, such as /usr/local, /opt, /home, otherwise ordinary users cannot use anaconda.
(1) The anaconda installed under the root user wants to give ordinary users the permission to use anaconda.
For example, the installation location is /opt/anaconda3, execute directly under the ordinary user: /opt/anaconda3/bin/conda init bash, and then execute conda env listthe test

(2) Anaconda installed under the ordinary user , want to give the super user (root) permission to use anaconda
For example, the installation location is /home/xxx/anaconda3, run directly under the root user: /home/xxx/anaconda3/bin/conda init bash, and then execute conda env listthe test

5. Uninstall Anaconda

If you accidentally install anaconda to a directory that cannot be accessed by ordinary users, such as /root, /home/root, you can uninstall anaconda and reinstall it.
(1) Delete the installation directory: rm -rf /root/anaconda3
(2) Edit the environment variable file vi ~/.bashrc, comment or delete the path of anaconda3
(3) Make the modified environment variable take effect immediately:source ~/.bashrc

3. Installation of graphics card driver, Cuda, cuDNN (optional for GPU version)

If the host has a graphics card, you can install Cuda to accelerate model training with GPU

1. Install the driver corresponding to the graphics card hardware

You can refer to several ways to install the graphics card driver on Ubuntu physical machine to install the graphics card driver
Input nvidia-smi, check the cuda version number you need, mine is 12.1

2. Download cuda corresponding to the graphics card driver version

Open the nvidia official website , click the corresponding version link to enter the download interface (mine is 12.1, you can choose any version of 12.1),


copy the command generated above, and execute the first line of command to download cuda:

wget https://developer.download.nvidia.com/compute/cuda/12.1.0/local_installers/cuda_12.1.0_530.30.02_linux.run

3.cuda installation

After the download is complete, execute chmod 777 【下载的sh文件】the execution permission, and then execute the second line of command to install cuda:

sudo sh cuda_12.1.0_530.30.02_linux.run

(1) Encounter the following interface, select continue to continue

(2) Encounter the following interface, enter accept

(3) Encounter the following interface, press Enter to uncheck the driver (the graphics card driver has already been installed), click install to install other components

After the installation is complete, the following path will be displayed.

At the same time, the system will automatically generate a /cuda link pointing to /cuda-version number in the installation directory.

4.cuda environment variable configuration

4.1 Enter sudo vi ~/.bashrcthe command to modify the environment variable.
According to the path name of your own anaconda3, add the following four lines at the end of the file:

export PATH="/anaconda3的路径名/anaconda3/bin:$PATH"
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:/usr/local/cuda/extras/CPUTI/lib64
export CUDA_HOME=/usr/local/cuda/bin
export PATH=$PATH:$LD_LIBRARY_PATH:$CUDA_HOME

4.2 Enter source ~/.bashrcthe command to activate the content just modified
4.3 Enter the following commands in order to test whether cuda is installed successfully

nvcc -V
cd /usr/local/cuda/extras/demo_suite/
./bandwidthTest

Result = PASS is successful

5.cuda uninstall

5.1 Enter the cuda installation directory, find the file related to uninstall, and execute the uninstall command: sudo /usr/local/cuda-12.1/bin/cuda-uninstaller

5.2 Check all the components of the cuda version to be uninstalled

5.3 After the uninstallation is successful, delete the relevant files

5.4 Delete the relevant environment variables
Enter sudo vi ~/.bashrcto delete the bottom cuda-related , execute source ~/.bashrcreactivation environment variable configuration

6. Download cuDNN corresponding to the cuda version

Open the nvidia official website , click the corresponding version link to enter the download interface (my cuda version is 12.1, you can choose any version of cuDNN 12.x, it is recommended to use the tar package ), download needs to register and log in to the nvidia account (by yourself Baidu)


Copy the downloaded file to the Ubuntu system, and execute the following decompression command:

unxz cudnn-linux-x86_64-8.9.0.131_cuda12-archive.tar.xz
tar -vxf cudnn-linux-x86_64-8.9.0.131_cuda12-archive.tar

7. cuDNN installation

Enter the decompressed directory, open the terminal, and enter the following command:

sudo cp include/cudnn*.h /usr/local/cuda/include/
sudo cp lib/libcudnn* /usr/local/cuda/lib64
sudo chmod a+r /usr/local/cuda/include/cudnn*.h /usr/local/cuda/lib64/libcudnn*

After the execution is complete, enter the following command to verify that cuDNN is installed successfully:

cat /usr/local/cuda/include/cudnn_version.h | grep CUDNN_MAJOR -A 2

The prompt message means: the ​constexpr​ keyword cannot be used in this file, because this file is limited to C language.

8. Uninstall of cuDNN

For cudnn installed through the tar package, delete the cudnn-related files copied to the cuda installation directory with the following command:

sudo rm -rf /usr/local/cuda/include/cudnn.h
sudo rm -rf /usr/local/cuda/lib64/libcudnn*

4. Create Anaconda virtual environment

1. Anaconda common commands: view, create, delete environment

(1) View the installed version of anaconda: conda --version
(2) View the installed environment (the one with an asterisk indicates the environment currently in use): conda env list
(3) Activate the corresponding virtual environment (enter the environment): source activate envone(环境名)
(4) Activate the default virtual environment Environment (base environment): source activate
(5) Exit the current conda environment: conda deactivate
(6) Delete the environment: conda remove -n envone(环境名) –all
(7) Delete the conda environment: conda uninstall -n envone(环境名) --all
(8) Install the required packages in the environment: pip install xxx
(9) Check the installed packages:pip list

2. Create ai space for building pytorch deep learning framework

After using anaconda to install the environment, the local default environment becomes the base environment.
(1) Create a space named ai: conda create -n ai
(2) Enter the ai space (virtual environment):conda activate ai

5. Build the Pytorch environment

1. Enter the created ai space through the command source activate ai
2. Enter the pytorch official website to download the corresponding version of pytorch

• Select the pytorch version
• Select the corresponding operating system
• Choose what installation to use, you can use pip to install in conda
• Select the programming language as python
• Select the cuda version: My version is 12.1. The pytorch official website currently only supports 11.8 Cuda, but the community has clearly indicated that it is compatible with higher versions of Cuda.

3. Execute the command generated above in the ai space

pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

WARNING: Running pip as the ‘root’ user can result in broken
permissions and conflicting behaviour with the system package manager.
It is recommended to use a virtual environment instead:
https://pip.pypa.io/warnings/venv WARNING: There was an error checking
the latest version of pip.

The above warning message means that running pip as the "root" user may result in broken permissions and conflicts with the behavior of the system package manager.

6. How to install related dependent libraries

1. Use the official website command to install the required dependent library

(1) Enter the anaconda official website to search for the name of the library to be installed (no need to register and log in)

(2) Select the required version

(3) Copy the installation command on the official website

(4) Execute and source activate aienter the ai virtual environment and execute the installation command copied above conda install -c pytorch-lts pytorch

(5) Log in to python to execute import torchthe verification whether it is successful

2. Mirror acceleration configuration of conda and pip

Modify the conda source and pip source to the domestic source to improve the download and installation speed
2.1 Modify the conda source
Modify the ~/.condarc file: sudo vi ~/.condarc, modify the file content as follows:

channels:
  - defaults
show_channel_urls: true
channel_alias: https://mirrors.tuna.tsinghua.edu.cn/anaconda
default_channels:
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/r
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/pro
  - https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/msys2
custom_channels:
  conda-forge: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
  msys2: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
  bioconda: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
  menpo: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
  pytorch: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud
  simpleitk: https://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud

If Tsinghua's conda source cannot be used, you can use the submitted source instead:

channels:
- https://mirrors.sjtug.sjtu.edu.cn/anaconda/pkgs/main/
- https://mirrors.sjtug.sjtu.edu.cn/anaconda/pkgs/free/
- defaults
show_channel_urls: true

2.2 Modify the pip source
Reference: Set pip mirror source for linux and windows, the most practical environment download acceleration setting - [understand in one article]
input to conda deactivateclose the virtual environment, return to the terminal interface
update pip tool: pip install --upgrade pip
configure pip source:pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple

3. Install common deep learning dependency libraries

Execute the following conda command to automatically search, download and install the applicable version and related dependencies of related dependent libraries.
(1) Install the opencv library (this library is used for image processing) command: conda install opencv
(2) Install the torchvision library (this library is used to download and train advanced pre-training models) command: conda install torchvision
(3) Install the torch library (this library is used for deep learning Build, train and deploy the neural network model in the task) Command: conda install torch
(4) Install the cv2 library (this library provides many functions and tools in computer vision and image processing): (5 conda install -c necla-ml opencv-python
) Install the tensorboardx library (this library provides visualization and Tools for analyzing the training process and results of neural networks):conda install -c conda-forge tensorboardx

4. Common problems when installing dependent libraries

Question 1: "PackagesNotFoundError" error is displayed when installing dependent libraries

If you encounter a "PackagesNotFoundError" error when using Conda or a similar package manager, it may indicate that a desired package or version of a package is not in the currently configured software source channel. You can refer to https://blog.csdn.net/weixin_45552562/article/details/109668589 to solve this problem. Run the following command to update the software source information: conda update --all


Re-execute conda install torch, if the error is still reported, you can follow the error prompt and refer to 6.1 to find the corresponding command on the official website to install.

7. Python environment description and Pycharm, Jupyter tool installation

1.Python environment instructions can refer to the system python under Ubuntu and python under anaconda

2. Pycharm tool installation can refer to Ubuntu 16.04 to install PyCharm
(1) Download the installation package on PyCharm official website

(community version is free) (2) Run tar -zxvf pycharm-community-2022.3.2.tar.gzthe command to decompress the installation package

(3) Enter the bin directory of the software, run the command sh ./pycharm.shto open Pycharm

(4) Ordinary user permissions problem
If the error message is about ordinary user permissions, you can try to change the owner of the relevant file

sudo chown ownername:groupname /path/to/directory

In the above command, "ownername" is the username of the specified new owner, "groupname" is the groupname of the specified new owning group, and "/path/to/directory" is the path to the directory you want to modify.

3. Anaconda has its own Jupyter tool, which can be opened with the following command: jupyter notebook, you can refer to the jupyter tutorial