[Tutorial] Guide to Building Miniconda+Pytorch+Pycharm Development Environment under Windows

Guide for building Miniconda+Pytorch+Pycharm development environment under Windows

The novice tutorial written for undergraduates has encountered many common problems for novices, which can be used for your reference. Before you start reading, please pay attention to two key points: 1. Why use Conda instead of installing Python directly locally, and experience the differences and advantages brought by Conda. 2. Clarify the compatibility between the CUDA version and the torch-gpu version.

1. Introduction and installation of Conda

1.1 Introduction to Conda

Conda is an open source package management system and environment management system for installing multiple versions of packages and their dependencies, and switching between them easily. is the most popular Python environment management tool.

1.2 Why use Conda?

Consider the scenario in the figure below, three projects, using three different Python and dependent packages (especially in the scenario of doing deep learning related research, it is necessary to reproduce the code work of other papers for comparative experiments, different versions of the library are very There may be many conflicts), if you install Python directly locally, there will be conflicts. Do we have to create the environment of project A to run project A first, then uninstall the environment of A and then create the environment of project B to run project B...

But if you use Conda, you can use Conda to create multiple Python environments locally at the same time and easily switch between them. Each environment is independent and will not affect other environments .

1.3 Anaconda vs. Miniconda

There are two distributions of Conda. Anaconda is more complicated and cumbersome , and miniconda is cleaner and lighter .

Anaconda is a packaged collection, which is pre-installed with conda, a certain version of python, many packages, scientific computing tools, etc., that is, many commonly used libraries that are not commonly used are installed for you.

Miniconda, as the name suggests, only includes the most basic content—python and conda, as well as related dependencies. For users with strict space requirements, Miniconda is an option. It only contains the most basic things, and other libraries have to be installed by yourself.

It is recommended to use Miniconda, students who have already installed Anaconda can ignore it.

1.4 Miniconda installation and configuration under Windows

Students who have installed Python directly locally, if they want to change to Miniconda, uninstall the local Python directly. If uninstalling, remember to delete the originally configured Python environment variables , otherwise the system will default to the original path, and some commands will cause errors.

process

1. Download the installation package from the official website of Miniconda , as shown in the figure, this option will install a corresponding Python version as the base environment by default;

2. Next and Agree all the way until you are asked to Select Installation Type ,

3. At this time, it is recommended to choose Just Me . For students with multiple hard disks, if you choose All User, the later creation environment will be created on the C drive by default.

​ That is, the following situation occurs (the pytorch environment is installed on the C drive, and most of us do not want to install it on the system drive);

4. Select the target path you want to install (my demo computer has only one C drive), and then continue to Next;

   

5. It is recommended to only check the first and third (create shortcuts; we add environment variables ourselves to avoid a series of unknown conflicts; set the python of the base environment to the default Python of the machine;

   

6. You don’t need to tick;

   

7. Configure environment variables. In the PATH variable of the system environment variable, add the following three environment variables according to the path where you installed Miniconda locally:
   

8. Theoretically, it has been installed successfully.

verify

1. Open the cmd terminal command prompt, directly enter the command: conda, and the following information appears, indicating that the installation is successful;

2. Directly enter the command: python to enter the python interpreter, and the information about the Python version of the default base environment will be displayed.

   

1.5 Basic use of Conda

• Initialization, students who install Conda locally for the first time may encounter the following problems:

  You may need to close and restart your shell after running ‘conda init’.

At this point, enter the following command to execute, then close the terminal command line window and reopen it .

conda init cmd.exe

• Create environment: py39test is the name of the environment, please define it yourself, python=3.9 means to install Python3.9 in this environment, and change it according to your own needs.

  conda create -n py39test python=3.9
  

• Activate the environment (that is, enter an environment), and only after entering the environment can the environment be operated.

  conda activate py39test
  

• exit environment

  conda deactivate
  

• To delete an environment, specify the environment name to delete.

  conda remove -n py39test --all
  

• Check the environment

  conda env list
  

  It can be seen that there are two environments of base and py39test in the current conda

  

• After entering a certain environment, installing other libraries in the current environment is similar to using pip, such as:

  conda install pandas
  

  Of course, you can also use pip to install directly in the current environment, and the two are generally considered to be almost the same . There are more than 1500 packages available in the conda repository, but it is still small compared to the more than 150,000 packages available on PyPI, so many times the packages that need to be installed can only be installed through pip .

• Change download source

  This step can be done or not . Because conda uses the official source to download by default, the speed will be relatively slow, especially for large packages like Pytorch, you can replace it with a domestic source to speed up the download, such as the Tsinghua mirror source.

  • Windows users cannot directly create .condarca file named , they can first execute conda config --set show_channel_urls yesto generate the file and then modify it. The file is automatically generated under the C disk-user-user name.

  

  • After opening the .condarc file, reset the content to (note the format):

    show_channel_urls: true
    channels:
      - http://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main/
      - http://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/
      - http://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge/
      - http://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/pytorch/
    ssl_verify: true
    

  • Then run conda clean -iclear index cache to ensure that the index provided by the mirror site is used.

  • If you want to use Tsinghua source to download later, you need to remove the '-c pytorch' parameter at the end of the command .

• For more conda commands, please use Baidu.

1.6 Pycharm integrates Conda

How to use the Python interpreter of the Conda environment in Pycharm?

1. Open the project, click File->Settings, search for Python Interpreter, and click Add Local Interpreter on the right...as shown in the figure:

2. Select Conda Environment, set the path of Conda executable to the path of the conda.exe file under MinicondaScripts, and then click Load Environments on the right to load the list of Conda environments. ( Be careful not to choose Virtualenv Environment, you will create the virtual environment that comes with Pycharm yourself )

3. Note, click Use existing environment, select an existing Python environment as the running environment of the current project , Miniconda3 is the base environment, but it is not recommended to use the base environment directly, here I choose py39test. Click OK, success, you can run the code to test it yourself.

• If you don't find it here, please find the path of the virtual environment you installed in the interpreter line, there are three dots on the right, and add the python interpreter in this environment.

The interpreter of the default base environment that comes with Conda is in the root directory where Conda is installed:

The rest of the environment is under the envs folder;

Enter an environment file, select the path of the interpreter, and you can use the virtual environment in Pycharm.

as the picture shows:

2. Pytorch installation

2.1 Introduction to Pytorch

A simple, fast, efficient, easy-to-use, and ecologically rich open source deep learning framework that is currently the most popular in academia, based on Python, supports GPU accelerated computing.

2.2 Conda install Pytorch

• Note, before installation, please clarify in which environment you want to install Pytorch , first activate (conda activate xxx) to enter the environment , and then install the library in this environment , otherwise , it will be installed in the base environment by default . (For example, each environment is like an independent room, and it will not have any impact on other rooms. You must enter the room before you can perform any operations on the room, and your operations will not affect other rooms. )
• Activate to enter the py38test environment, and install pytorch in this environment without affecting other environments.
• After activation, (current environment name) will appear on the left side of the terminal command line , as shown in the figure below, I have entered the py38test environment of the local conda:

Open the Pytorch official website , and select instructions according to the PyTorch version, your operating system, Conda/Pip (directly use pip without Conda), language, and computing platform. The official website page only provides the latest version installation instructions.

Click Install previous versions of PyTorch to view the installation instructions of Pytorch versions that have been released in the past ( it is recommended to find them here, ).

• Note that there is a small error in the installation instructions of the latest pytorch1.13 version provided by the official website:

conda install pytorch==1.13.1 torchvision==0.14.1 torchaudio==0.13.1 pytorch-cuda=11.6 -c pytorch -nvidia

The last '-nvidia is missing a -c' in front of it, change it to

conda install pytorch==1.13.1 torchvision==0.14.1 torchaudio==0.13.1 pytorch-cuda=11.6 -c pytorch -c -nvidia

In addition, there is a corresponding relationship between Python and Pytorch versions . For details, you can check: Correspondence between torch/torchvision/python versions . According to the figure below, it is recommended not to use too high or too low a Python version to be compatible with most Pytorch versions.

2.2.1 CPU version installation:

Students who do not have an NVIDIA discrete graphics card can only install the CPU version, which must be NVIDIA, and it is a discrete graphics card, and integrated graphics cards are not acceptable.

Order:

conda install pytorch torchvision torchaudio cpuonly -c pytorch

• Students who use Tsinghua source must remove the last '-c pytorch' to download from Tsinghua source . This parameter means downloading from pytorch official website.
• In addition, this command installs the latest pytorch, torchvision, and torchaudio by default. You can specify the version according to your own needs (you should specify the pytorch version. Both torchvision and torchaudio depend on pytorch, and there are version dependencies, and the current pytorch support will be automatically installed. highest version).

For another example, we specify that the pytorch version is 1.9.0, the torchvision is 0.10.0, and torchaudio is not installed:

conda install pytorch==1.9.0 torchvision==0.10.0 cpuonly -c pytorch

2.2.2 GPU version installation

To install the GPU version of Pytorch, you first need to have an NVIDIA discrete graphics card and CUDA driver. The driver is generally installed (without Baidu), and then you need to install CudaToolKit. NVIDIA CUDA Toolkit is a professional CUDA toolkit that provides users with a professional development environment to create GPU-accelerated high-performance applications. (It probably means that you can use the GPU to accelerate the neural network with the CUDA Toolkit.)

(1) Determine the independent graphics card, CUDA driver version and CUDA version of the machine

Open the terminal cmd, command:

nvidia-smi

(2) Determine the torch-gpu version and CUDA Toolkit version supported by the native CUDA

Principle : The driver is backward compatible , which determines the highest version of CUDA Toolkit that can be installed . The officially recommended cuda versions are 10.2 and 11.3 , which support most pytorch versions.However, please note that some students have older graphics cards. If CUDA is 10.2, it can only support torch-gpu version 1.10.0, which means that CUDA 10.2 cannot be installed with GPU versions above torch-1.10.0. , but the CPU version is fine, as long as your python version meets the requirements.

• For details, please refer to this article: Correspondence between PyTorch, CUDA Toolkit and graphics card driver version

For example, my NVIDIA driver version is 516.59. According to the figure below, the highest version of CUDA Tookit I can install is 11.7, and it is backward compatible with all versions from 10.1 to 11.7.

• At this time, if I want to install Pytorch1.12.1, according to the command provided by the official website, it can be:

# CUDA 11.3
conda install pytorch==1.12.1 torchvision==0.13.1 torchaudio==0.12.1 cudatoolkit=11.3 -c pytorch

can also be:

# CUDA 11.6
conda install pytorch==1.12.1 torchvision==0.13.1 torchaudio==0.12.1 cudatoolkit=11.6 -c pytorch -c conda-forge

2.3 Local Python direct installation

This step is for students who do not install Conda .

• The CPU version directly uses the pip command to install the corresponding version;

• The GPU version is installed directly with pip according to the CUDA driver version, which is actually the same as conda. Generally, parameters such as 'torch==xxx+cuxxx' are added during installation. For details, please refer to the official website command.

Such as: (Obviously, if you enter the conda environment, you can also use the following command to install the torch-gpu version, because as we said before, using conda install and pip install in the conda environment is basically the same, just pull these libraries The source is different, please experience the function of conda carefully.)

# CUDA 11.6
pip install torch==1.12.0+cu116 torchvision==0.13.0+cu116 torchaudio==0.12.0 --extra-index-url https://download.pytorch.org/whl/cu116
# CUDA 11.3
pip install torch==1.12.0+cu113 torchvision==0.13.0+cu113 torchaudio==0.12.0 --extra-index-url https://download.pytorch.org/whl/cu113
# CUDA 10.2
pip install torch==1.12.0+cu102 torchvision==0.13.0+cu102 torchaudio==0.12.0 --extra-index-url https://download.pytorch.org/whl/cu102
# CPU only
pip install torch==1.12.0+cpu torchvision==0.13.0+cpu torchaudio==0.12.0 --extra-index-url https://download.pytorch.org/whl/cpu

2.4 Verification

import torch
print(torch.cuda.is_available())
# 如果成功安装了CPU版本，会输出False
# 如果成功安装了GPU版本，会输出True，集成显卡也会输出True，但是用不了。
# 没有报错即安装成功。