Basic guide to using Python package management (pip, conda)

Python package management

overview

introduce

Python has a wealth of open source third-party libraries and packages that can help complete various tasks and extend the functionality of Python, such as NumPy for scientific computing, Pandas for data processing, Matplotlib for drawing, etc. Before you start writing Pytlhon programs, you may want to install some commonly used Python libraries so that you can easily use them during programming.

In order to manage third-party libraries and packages conveniently, you need to install a Python package management tool, such as pip, conda, etc. These tools help install, upgrade, and remove Python packages, making it easy to manage Python dependencies.

The difference between Anaconda, conda, pip, virtualenv

• Anaconda

  Anaconda is a distribution containing 180+ scientific packages and their dependencies. The scientific packages it contains include: conda, numpy, scipy, ipython notebook, etc.

• conda

  conda is a management tool for packages, their dependencies and environments.

  Applicable languages: Python, R, Ruby, Lua, Scala, Java, JavaScript, C/C++, FORTRAN

  Applicable platforms: Windows, macOS, Linux

  use:

  • Quickly install, run, and upgrade packages and their dependencies.

  • Create, save, load and switch environments conveniently on your computer.

    If the required package requires a different version of Python, there is no need to switch to a different environment, because conda is also an environment manager. With just a few commands, you can create a completely separate environment to run different Python versions, while continuing to use your usual Python version in your regular environment. ——

  conda was created for Python projects, but is available for many of the languages ​​mentioned above.

  The conda package and environment manager is included with all versions of Anaconda.

• pip

  pip is a package manager for installing and managing software packages.

  Applicable language of pip: Python

  Versions installed by default in Python:

  • Python 2.7.9 and subsequent versions: installed by default, the command is pip
  • Python 3.4 and subsequent versions: installed by default, the command is pip, pip3

  The origin of the pip name: pip is named after a recursive abbreviation . Its name is generally believed to come from 2 sources:

  • "Pip installs Packages"
  • "Pip installs Python" ("pip installs Python")

• virtualenv

  virtualenv is a tool for creating an isolated Python environment.

  Solve the problem:

  • When one program needs to use Python 2.7 version and another program needs to use Python 3.6 version, if all programs are installed in the default path under the system, such as: , when the /usr/lib/python2.7/site-packagesprogram that should not be upgraded is accidentally upgraded, it will affect other programs.
  • Installing a program or modifying its library or library version while the program is running will cause the program to break.
  • When sharing hosting, site-packagespackages cannot be installed in the global directory.

  virtualenv will create an environment for its own installation directory, which does not share libraries with other virtualenv environments; it can also optionally not link installed global libraries.

pip vs. conda

• dependency check

  • pip：

    Other dependencies required may not necessarily be displayed.

    When installing the package, it may simply ignore the dependencies and install them, only to prompt errors in the results.

  • conda：

    List other required dependencies.

    Automatically installs its dependencies when installing a package.

    You can easily switch freely between different versions of the package.

• environmental management

  • pip: It is difficult to maintain multiple environments
  • conda: It is more convenient to switch between different environments, and the environment management is relatively simple

• Impact on the system's built-in Python

  • pip: Updating/reverting/uninstalling packages in the system's own Python will affect other programs.
  • conda: It will not affect the Python that comes with the system.

• Applicable language

  • pip: for Python only
  • conda: Available for Python (mainly), R, Ruby, Lua, Scala, Java, JavaScript, C/C++, FORTRAN.

• The relationship between conda and pip, virtualenv

  conda combines the power of pip and virtualenv.

  conda uses a new package format, and pip cannot install and parse conda's package format. Two tools can be used, but they cannot interact, such as using pip to locally install conda's offline package.

pip (Python package manager)

introduce

• pip is Python's official package manager, and it ships with versions of Python. When installing Python, pip is also installed.
• Various third-party libraries (packages) of Python that are not part of the Python standard library can be easily installed and managed through pip, so that they can be directly introduced and used in the code in the project.
• pip is a command-line tool that can be used directly in the terminal or command prompt.

Basic commands

• python package management

  # 查看已安装的包
  pip list
  # 查看需要升级的库
  pip list -o
  
  # 安装一个 python 包
  pip install package_name			# package_name：具体地包名
  # 安装特定版本的包
  pip install package_name==version_number
  # 安装本地包
  pip install /path/to/package		# /path/to/package：本地包路径
  
  # pip的超时时间默认为15秒，如果下载速度过慢，可以使用以下命令设置超时时间为60秒
  # 方式1：添加参数--default-timeout=60。# 方式：在配置里面[global]下添加timeout=60
  pip install --default-timeout=60 package_name
  
  # 指定国内的源（阿里云）来安装某个包
  pip install -i https://pypi.tuna.tsinghua.edu.cn/simple <package-name>
  # 注意：如果url是http的化，需要信任(因为未加密)，可以通过下面两个方法解决
  # 方式1：安装时加入 --trusted-host 临时参数
  pip install -i http://pypi.douban.com/simple/ --trusted-host pypi.douban.com/simple package_name
  # 方式2：在 pip.conf 中加入 trusted-host 选项，该方法是一劳永逸
  [global]
  index-url = http://mirrors.aliyun.com/pypi/simple/
  [install]
  trusted-host=mirrors.aliyun.com
  
  # 升级包
  pip install --upgrade package_name
  # 升级pip
  pip install --upgrade pip
  
  # 卸载包
  pip uninstall package_name
  
  # 安装包的依赖项
  pip install package_name[dependencies]
  
  # 验证已安装的库是否有兼容依赖问题
  pip check package_name
  
  # 下载某个包到指定的路径下（不安装）
  pip download package_name -d "某个路径"  
  
  # 查看包的详细信息
  pip show package_name
  
  # 导出已安装的包列表
  pip freeze > requirements.txt
  # 从requirements.txt文件中安装包
  pip install -r requirements.txt
  

• Download source management

  # 新增全局下载源。国内最好更换pip的源，以便更快地下载包
  pip config set name value
  # 示例：清华源
  pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple
  
  # 删除全局下载源
  pip config unset name
  
  # 查看pip下载的安装包的默认路径
  python -m site
  
  # 显示pip的配置列表
  pip config list
  # 显示pip文件的所有存储位置
  pip -v config list
  # 输出的不同的目录对应不同的参数 --global（全局）、--user（用户） 、--site
  

• other commands

  # 清理缓存。pip安装包的时候，会下载并缓存一些包以便后续使用，不会自动删除，这可能会占用磁盘空间，需要手动删除
  pip cache purge
  
  # 查看pip版本
  pip --version
  pip -V
  

General parameter options for pip

• -r，--requirement <file>: Install from the given requirements file. This option can be used multiple times.

• -c，--constraint <file>: Use the given constraints file constraint version. This option can be used multiple times.

  Constraints files are smarter than requirements files. Constraints files differ from requirements files in one key respect: putting a package in a constraints file will not cause the package to be installed, whereas a requirements file will install all packages listed. It is often used to put the dependencies of a package.

• --no-deps: Do not install package dependencies

• --pre，pip: Find includes pre-release and development versions. By default, pip only looks for stable releases.

• -e，--editable <path/url>: Install project in editable mode (ie setuptools development mode) from local project road king or VCS url.

• -t，--target <dir>: Install the package into <dir>, by default this will not replace <dir>existing files or folders

• --platform <platform>: Use only <platform>wheels compatible with . Defaults to the platform on which the system is running. Use this option multiple times to specify multiple platforms supported by the target interpreter

• -U，--update: Upgrade all specified packages to the latest available versions. The handling of dependencies depends on the upgrade strategy used.

• --upgrade-strategy <upgrade_strategy>: Determines how dependency upgrades should be handled.

  Two modes:

  • "eager" : Dependencies will be upgraded regardless of whether the currently installed version of the dependency meets the requirements for upgrading the package
  • "only-if-needed": Only upgrade if the requirements of the upgrade package are not met

• --force-reinstall: reinstall all packages even if they are already up to date

• -I,--ignore-installed: Ignore installed packages, overwriting them.

  This can break your system if existing packages are of different versions or have a different package manager installed.

• --compile: compile python source files to bytecode

• --no-compile: Do not compile python source files to bytecode

• --no-binary <format_control>: Do not use binary packages.

  <format_control>can be:

  • all: disable all binary packages
  • none: Empty the previously provided packages, or use the specified packages, separated by commas

  Note that some packages are difficult to compile and may not install when using this option.

• --only-binary <format_control>: Do not use source packages

Conda（Anaconda）

Reference: Anaconda introduction, installation and usage tutorial

introduce

• Conda is an open source package management system and environment management system that runs on Windows, macOS, and Linux.

  Conda quickly installs, runs, and updates packages and their dependencies, so it's easy to create, save, load, and switch environments on your computer.

  It was originally created for Python programs, because there are many versions of Python, and its libraries are also very extensive, and there are many dependencies between libraries, so it is very troublesome to install and manage versions of libraries, so Conda is designed as a tool for managing versions and Python environments, but it can also package and manage software in any language.

  Conda is included in Anaconda, so you can use Conda directly after installing Anaconda, click the link to download , and select the corresponding system and version type.

• Anaconda is a free and open source Python and R distribution for computational science (data science, machine learning, big data processing, and predictive analytics),

  Anaconda is dedicated to simplifying the package management system and deployment, and comes with Conda, python, and more than 150 scientific packages and their related packages.

  Anaconda's packages are managed using the package management system Conda.

  
  

In the process of project development and deployment, because the virtual environment required by the project is different, such as python version, module version, etc., you can create an environment for each project through Conda, and then manage and use it in the corresponding environment.

• Management of packages and virtual environment environments in multiple programming languages

• Very simple to complete package installation, operation, update, deletion, and dependency issues

• Operable 7,500+ packages on repo.anaconda.com

• It is very simple to complete the construction, saving, loading and switching of different environments

• Supported languages: Python, R, Ruby, Lua, Scala, Java, JavaScript, C/C++, FORTRAN

  But generally it is mainly used to manage python packages

• Supported Operating Systems: Windows, macOS and Linux

Conda Common Commands

• Virtual Environment Management

  # 查看所有环境。注：1.*号所在的行表示当前所在环境。2.系统默认虚拟环境为base
  conda env list
  conda info -e
  
  # 创建新的虚拟环境。注：1.遇到yes/no输入yes，即可完成创建。
  conda create -n envs_name python=python_version
  # envs_name			：指定环境名称
  # python_version	：指定Python版本
  
  # 进入虚拟环境
  conda activate envs_name
  # 退出当前环境
  conda deactivate
  
  # 删除虚拟环境
  conda env remove -n envs_name
  
  # 复制虚拟环境
  conda create -n new_envs_name --clone old_envs_name
  
  # 更新python
  conda update python
  
  # 更新conda版本。注：必须在base环境更新
  conda update conda
  

• Module/package management

  # 查看当前环境的包列表
  conda list
  # 查看指定环境的包列表
  conda list -n envs_name
  
  # 在当前环境安装包。注：1.默认安装最新版本，2.同时安装多个包用空格分隔。
  conda install package_name
  # 在指定环境安装包
  conda install --name env_name package_name
  # 安装指定版本的包。注：使用conda安装指定包时，conda可以自动处理相关的依赖包
  conda install numpy=1.19
  # 指定范围内中版本包安装（安装版本处于1.0.4到1.1.1之间的pandas）
  conda install "pandas>1.0.4,<1.1.1"
  # 指定list中版本包安装（安装pandas 1.0.4版或者1.1.1版）
  conda install "pandas[version='1.0.4 |1.1.1']"
  
  # 包安装跳过【y/n】。默认情况下为 false，即安装过程中会请求是否继续安装，设置为yes则不再弹出请求。
  conda config --set always_yes yes
  
  # 卸载当前环境的包
  conda remove package_name
  # 卸载指定环境的包
  conda remove --name env_name package_name
  
  # 升级当前环境的包
  conda update/upgrade package_name
  # 升级指定环境的包
  conda update/upgrade -n env_name package_name
  # 升级全部包
  conda upgrade --all
  
  # 精确查找包
  conda search package_name
  # 模糊查找包，模糊符号为 *
  conda search *<模糊词>*
  # 查看某个范围内版本包
  conda search "PKGNAME [version='>=1.0.0,<1.1']"
  

• conda download source management

  # 查看已配置下载源
  conda config --show channels
  
  # 查看已配置下载源优先级
  conda config --get channels
  
  # 新增下载源（清华大学源）
  conda config --add channels channels_Name
  conda config --add channels https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/free/
  
  # 删除下载源
  conda config --remove channels channels_Name
  
  # 显示包的安装来源
  conda config --set show_channel_urls yes
  

  Note: The Conda download source is actually written into the .condarc file:

  • The path of the linux system is/home/xx/.condarc

  • The path of the windows system isC:\Users\admin\.condarc

    Windows has no .condarc file by default, it needs to be generated by conda config --set show_channel_urls yes

• other commands

  # 检查conda版本
  conda --version
  # 查看conda系统版本等信息
  conda info
  # 查看conda所有配置信息
  conda config --show
  

IDEA configures Conda virtual environment

IntelliJ IDEA supports using Conda to create virtual environments for Python after installing and enabling the Python plugin.

Create a Conda environment:

1. Make sure you have Anaconda or Miniconda downloaded and installed on your computer.

2. Navigate to File | Project Structure

   • Method 1: File >>> Project Structure in the upper left corner
   • Method 2: Shortcut key: Ctrl+Shift+Alt+S

   

3. In the " Project Structure " dialog, select " SDK " under the " Platform Setting " section , then add a new SDK, then select " Python SDK " from the popup menu

   

4. In the left pane of the Add Python Interpreter dialog, select Conda Environment. The following actions depend on whether a Conda environment previously existed.

   If " New environment " is selected:

   1. Specify the location of the new Conda environment in the Location field. Note that the directory where the new Conda environment should be located must be empty!
   2. Select the Python version from the Python version list
   3. Specify the location of the Conda executable in the Conda executable field
   4. If desired, select the "Make available to all projects" checkbox.

   If you select Existing environment ( recommended ):

   1. Expand the Interpreter list and select any existing interpreter. Alternatively, click to select " … " and specify the path to the Conda executable in the file system, e.g.C:\Users\jetbrains\Anaconda3\python.exe

      Note: It is best to use python.exe in the Conda directory, otherwise the installed third-party library may not be scanned

   2. Specify the location of the Conda executable in the Conda executable field

   3. If desired, select the "Make available to all projects" checkbox.

5. Click OK to complete the task.