pip: introduce you to an important tool in the Python development workflow

This article is shared from HUAWEI CLOUD Community " Using Python's pip to manage project dependencies ", author: Yuchuan.

The standard package manager for Python is pip. It allows you to install and manage packages that are not part of the Python standard library. If you're looking for an introduction to pip, you've come to the right place!

In this tutorial, you will learn how to:

pip is set in your working environment
Fix common errors related to using pip
Install and uninstall packages pip
Manage project dependencies using requirements files

get started with pip

So, what exactly does pip do? pip is a package manager for Python. This means that it is a tool that allows you to install and manage libraries and dependencies that are not distributed as part of the standard library. The name pip was coined by Ian Bicking in 2008:

I have renamed pyinstall to a new name: pip. The name pip is [an] acronym and statement: pip installs a package. (source)

Package management is so important that Python's installer, pip, has been installed for Python3 and Python2 since versions 3.4 and 2.7.9, respectively. Many Python projects use pip, which makes it a must-have tool for every Pythonista.

If you come from another programming language, then you are probably familiar with the concept of package managers. JavaScript uses npm for package management, Ruby uses gems, and the .NET platform uses NuGet. In Python, pip has become the standard package manager.

pip finds on your system

The Python3 installer provides installation options for you to install Python on your pip system. In fact, the option for pip to install with Python is checked by default, so it should be available after pip installs Python.

Note: On some Linux (Unix) systems (like Ubuntu), there is a pip file called .interpreter which does not install it by default. python3-pipsudoaptinstallpython3-pip

pip You can verify that the pip3 executable is available by looking for it on your system. Select your operating system below and use your platform specific commands accordingly:

Windows
Linux+macOS

C:\>wherepip3

The where command on Windows will tell you where to find pip3. If Windows cannot find an executable named pip3, then you can also try looking for pip without the triple() at the end. 3

On Windows and Unix systems, pip3 may be found in several places. This can happen when you have multiple Python versions installed. If you can't find pip anywhere on your system, then you might consider reinstalling pip.

In addition to pip running the system directly, you can also run it as a Python module. In the next section, you will learn how.

pip runs as a module

When you pip run your system directly, the command itself doesn't show which Python version pip belongs to. Unfortunately, this means that you can pip install packages into site-packages for older Python versions without noticing. To prevent this from happening, you can run pip as a Python module:

$python3-mpip

Note that you run pip with python3 -m . The -m switch tells Python to run the module as an executable for the python3 interpreter. This way you can ensure that your system default Python3 version runs that pip command. If you want to learn more about this way of running pip, then you can read BrettCannon on using python3-mpip.

Sometimes you may want to be more explicit and restrict packages to specific projects. In this case you should run pip in a virtual environment .

Using pip in a Python virtual environment

To avoid installing packages directly into the system Python installation, you can use a virtual environment. A virtual environment provides a standalone Python interpreter for your project. Any packages you use in this environment will be independent of your system interpreter. This means you can keep your project's dependencies separate from other projects and the entire system.

Using pip in a virtual environment has three main advantages. you can:

Make sure you are using the correct Python version for the project at hand
Make sure you are referencing the correct pip instance pip or pip3 at runtime
Use a specific package version for your project without affecting other projects

venvPython3 has built-in modules for creating virtual environments. This module helps you create virtual environments with isolated Python installations. After activating a virtual environment, you can install packages into this environment. Packages that you install into one virtual environment are isolated from all other environments on the system.

You can create a virtual environment and verify that you pip use the module in the newly created environment by following these steps:

Windows
Linux+macOS

C:\>python-mvenvvenv
C:\>venv\Scripts\activate.bat
(venv)C:\>pip3--version
pip21.2.3from...\lib\site-packages\pip(python3.10)
(venv)C:\>pip--version
pip21.2.3from...\lib\site-packages\pip(python3.10)

venv Here you create a virtual environment using Python's built-in venv module. Then you activate it with the source command. ()Brackets ()venv around your name indicate that you have successfully activated the virtual environment.

Finally, check the version of pip3 and executable in the activated virtual environment. pip both point to the same pip module, so once your virtual environment is activated you can use pip or pip3.

Reinstall on pip error

When running pip commands, errors may appear in some cases. Your specific error message will depend on your operating system:

Error messages like this indicate that pip is installed.

Note: When the pip command doesn't work, before starting any troubleshooting, you can try using the pip3 command with three ( ) at the end. 3

Getting the error shown above can be frustrating because pip is essential for installing and managing external packages. Some common issues are related to how pip is installed on your system.

Although different systems have different error messages, they all point to the same problem: your system cannot be found in the locations listed in the variable pip. PATH On Windows, PATH is part of system variables . On macOS and Linux, PATH is part of the environment variables . You can check the contents of the PATH variable with:

Windows
Linux+macOS

C:\>echo%PATH%

The output of this command will display a list of places (directories) on disk where the operating system looks for executable programs. Depending on your system, locations can be separated by colons (:) or semicolons (;).

By default, after installing Python or creating a virtual environment, the directory containing the pip executable should exist. PATH However, missing pip is a common problem. Two supported methods can help you to pip install again and add it to your PATH:

ensurepip module_
get-pip.py script_

The ensurepip module has been part of the standard library since Python 3.4. It was added to give you a straightforward way to reinstall, pip for example, if you skipped it when installing Python or pip uninstalled it at some point. Select your OS below and ensurepip runs accordingly:

Windows
Linux+macOS

C:\>python-mensurepip--upgrade

If pip is not already installed, this command will install it in your current Python environment. If you are in an active virtual environment, this command installs pip into that environment. Otherwise, it installs pip globally on your system. The --upgrade option ensures that the pip version is the same as the one declared in ensurepip.

Note: The module ensurepip does not go online. The latest version of pip that can be installed is the version of ensurepip that is bundled in your environment's Python installation. For example, running ensurepip with Python3.10.0 will install pip21.2.3. If you want a newer version of pip, then you need to run ensurepip first. After that, you can pip manually update to its latest version.

Another way to fix a pip install is to use the get-pip.py script. The get-pip.py file contains a complete copy of the pip-encoded ZIP file. You can download get-pip.py directly from the PyPA bootstrap page. Once you have the script on your machine, you can run the Python script like this:

Windows
Linux+macOS

C:\>pythonget-pip.py

This script will install the latest versions of pip, setuptools and pip in your current Python environment. wheel If you just want to install pip, then you can add the --no-setuptools and --no-wheel options to your command.

If none of the above methods work, it might be worth trying to download the latest Python version for your current platform. You can follow the Python3 installation and setup guide to make sure pip is properly installed and working.

install package pip

Python is considered a language that contains batteries. This means that the Python standard library contains an extensive set of packages and modules to help developers with their coding projects.

At the same time, Python has an active community that provides a wider set of packages that can help you with your development needs. These packages are published to the Python Package Index, also known as PyPI (pronounced PiePeaEye).

PyPI hosts a large number of packages, including development frameworks, tools and libraries. Many of these packages provide friendly interfaces to the functionality of the Python standard library.

Using the Python Package Index (PyPI)

One of the many packages hosted by PyPI is called requests. This library helps you interact with web services by abstracting away the complexities of HTTP requests. You can learn all about it on its official documentation site. requests

When you want requests to use this package in your project, you must first install it into your environment. If you don't want to install it in your system Python site-packages, then you can first create a virtual environment as shown above.

After a virtual environment is created and activated, the command line prompt displays the virtual environment name in parentheses. Any pip commands you execute from now on will happen in your virtual environment.

To install a package, pip provides an install command. You can run this to install the requests package:

Windows
Linux+macOS

(venv)C:\>python-mpipinstallrequests

In this example, you run pip with the install command followed by the name of the package you want to install. The pip command looks for a package in PyPI, resolves its dependencies, and installs everything in the current Python environment to make sure its requests work.

The pipinstall <package> command always looks for the latest version of a package and installs it. It also searches for dependencies listed in the package metadata and installs them to ensure the package has all the required requirements.

It is also possible to install multiple packages in one command:

Windows
Linux+macOS

(venv)C:\>python-mpipinstallrptreecodetiming

By linking the package rptree and codetiming in the pipinstall command, you can install two packages at once. You can add any number of packages to the pipinstall command. In this case, the requirements.txt file can come in handy. Later in this tutorial, you'll learn how to use a requirements.txt file to install multiple packages at once.

Note: Unless the specific version number of the package is relevant for this tutorial, you will notice that the version string is in xyz which is a placeholder format that can represent 3.1.4, 2.9 or any other version number. As you proceed, the output in the terminal will show your actual package version number.

You can use the list command to display the packages installed in your environment and their version numbers:

Windows
Linux+macOS

(venv)C:\>python-mpiplist
PackageVersion
---------------------------
certifix.y.z
charset-normalizerx.y.z
codetimingx.y.z
idnax.y.z
pipx.y.z
requestsx.y.z
rptreex.y.z
setuptoolsx.y.z
urllib3x.y.z

The piplist command renders a table showing all installed packages in the current environment. The above output shows the version of the package using the xyz placeholder format. When you run the piplist command in your environment, pip displays the specific version number you have installed for each package.

To get more information about a specific package, you can view the package's metadata with the show command pip:

Windows
Linux+macOS

(venv)C:\>python-mpipshowrequests
Name:requests
Version:x.y.z
Summary:PythonHTTPforHumans.
...
Requires:certifi,idna,charset-normalizer,urllib3
Required-by:

The output of this command on your system will list the package's metadata. The Requires line lists packages such as certifi, idna, charset-normalizer, and urllib3. These are installed because requests depend on them to work properly.

Now that you have requests installed and its dependencies, you can import it like any other regular package in Python code. Start the interactive Python interpreter and import the requests package:

>>>
>>>importrequests
>>>requests.__version__
"x.y.z"

After starting the interactive Python interpreter, you imported the requests module. By calling requests.__version__, you confirm that you are using the module in the requests virtual environment.

Use a custom package index

By default, pip uses PyPI to find packages. But pip also gives you the option to define a custom package index.

Using pip with a custom index can be helpful when the PyPI domain is blocked on your network or you want to use a package that is not publicly available. Sometimes sysadmins also create their own internal package index to better control which package versions are available to users on the pip company network.

A custom package index must conform to PEP503 – SimpleRepositoryAPItoworkwithpip. You can get an idea of what such an API (application programming interface) looks like by visiting the PyPI Simple Index - but be aware that this is a huge page with a lot of hard-to-parse stuff. Any custom index that follows the same API can be targeted by this option. Besides typing, you can also use shorthand. --index-url--index-url-i

For example, to install the tool from the TestPyPIrptree package index, you can run the following command:

Windows
Linux+macOS

(venv)C:\>python-mpipinstall-ihttps://test.pypi.org/simple/rptree

With the -i option, you tell pip to look at a different package index than the default PyPI. Here, you installed rptree from TestPyPI, not from PyPI. You can use TestPyPI to fine-tune the release process for Python packages without messing up the production package index on PyPI.

If you need to use an alternate index permanently, you can set this option in the config file index-url. The file is named and you can find its location by running: pippip.conf

Windows
Linux+macOS

(venv)C:\>python-mpipconfiglist-vv

Using the pipconfiglist command, you can list active configurations. This command will only output if you have a custom configuration set. Otherwise, the output is empty. At that time, the additional --verbose or -vv, options may be helpful. When adding -vv, pip shows where it looks for different configuration levels.

If you want to add a pip.conf file, you can choose one of the locations listed by pipconfiglist-vv. The pip.conf file with the custom package index looks like this:

#pip.conf

[global]
index-url=https://test.pypi.org/simple/

When you have such a pip.conf file, pip will use the defined index-url to find the package. With this configuration, you don't need to use the --index-url option in the command pipinstall to specify that you only need packages that can be found in TestPyPI's simple API.

Install the package from the GitHub repository

You are not limited to packages hosted on PyPI or other package indexes. pip also provides the option to install packages from GitHub repositories. But even if the package is hosted on PyPI, such as the RealPython directory tree generator, you can choose to install it from its Git repository:

Windows
Linux+macOS

(venv)C:\>python-mpipinstallgit+https://github.com/realpython/rptree

Using the git+https scheme, you can point to a Git repository containing installable packages. You can verify that you have the package rptree installed correctly by running the interactive Python interpreter and importing:

>>>
>>>importrptree
>>>rptree.__version__
"x.y.z"

After starting the interactive Python interpreter, import the rptree module. By calling rptree.__version__, you can verify that you are using rptree's virtual environment-based modules.

Note: If you are using a version control system (VCS) other than Git, do you know about pip. To learn how to use pipMercurial, Subversion or Bazaar, check out the VCS support section of the documentation for pip.

Installing the package from the Git repository can be helpful if the package is not hosted on PyPI but has a remote Git repository. The remote repository pip you point to can even be hosted on an internal Git server on the company intranet. This can be useful when you are behind a firewall or have other restrictions on your Python project.

Install packages in editable mode to simplify development

When working on your own package, it makes sense to install it in editable mode. By doing this, you can use the command line as in any other package, while working with source code. A typical workflow is to first clone the repository, then pip install it into your environment as an editable package:

Windows
Linux+macOS

1C:\>gitclonehttps://github.com/realpython/rptree
2C:\>cdrptree
3C:\rptree>python3-mvenvvenv
4C:\rptree>venv\Scripts\activate.bat
5(venv)C:\rptree>python-mpipinstall-e.

Using the above command, you installed the rptree package as an editable module. Here's a step-by-step breakdown of what you just did:

Line 1 clones the Git repository for the rptree package.
Line 2 changes the working directory to rptree/.
Lines 3 and 4 create and activate a virtual environment.
Line 5 installs the contents of the current directory as an editable package.

The -e option is shorthand for the option --editable. When you use the -e option with pipinstall, you tell pip that you want to install the package in editable mode. Instead of using a package name, use a dot (.) to point to the pip current directory.

If you don't use the -e flag, pip will normally install packages into your environment's site-packages/ folder. When you install a package in editable mode, you create a link in your site-packages to your local project path:

~/rptree/venv/lib/python3.10/site-packages/rptree.egg-link

Using the pipinstall command with the -e flag is just one of the many options pipinstall offers. You can check out the pipinstall example in the documentation. pip is there and you'll learn how to install a specific version of a package or point to a different index where pip is not PyPI.

In the next section, you'll learn how requirements files can help your pip workflow.

Use requirements file

The pipinstall command always installs the latest released version of a package, but sometimes your code needs a specific package version to work properly.

You want to create a specification of dependencies and versions for developing and testing an application so that there are no surprises when the application is used in production.

Fixed requirements

When you share your Python project with other developers, you probably want them to use the same versions of external packages that you are using. Maybe a specific version of a package contains new functionality that you depend on, or you're using a version of a package that is incompatible with a previous version.

These external dependencies are also called requirements. You will often find Python projects pinning their requirements in a file called requirements.txt or similar. The requirements file format allows you to specify exactly which packages and versions should be installed.

Running piphelp shows that there is a freeze command that outputs installed packages in demand format. You can use this command, redirecting the output to a file to generate a requirements file:

Windows
Linux+macOS

(venv)C:\>python-mpipfreeze>requirements.txt

requirements.txt This command creates a file in your working directory with the following content:

certifi==x.y.z
charset-normalizer==x.y.z
idna==x.y.z
requests==x.y.z
urllib3==x.y.z

Keep in mind that xyz above shows the package version in placeholder format. Your requirements.txt file will contain the real version number.

The freeze command dumps the names and versions of currently installed packages to standard output. You can redirect the output to a file that you can later use to install your exact requirements into another system. You can name the requirements file whatever you want. However, a widely adopted convention is to name it requirements.txt.

When you want to replicate the environment in another system, you can run pipinstall specifying the requirements file with the -r switch:

Windows
Linux+macOS

(venv)C:\>python-mpipinstall-rrequirements.txt

In the command above, you told pip to install the packages listed in requirements.txt into the current environment. The package version will match the version constraints contained in the requirements.txt file. You can run piplist to display the packages just installed and their version numbers:

Windows
Linux+macOS

(venv)C:\>python-mpiplist

PackageVersion
---------------------------
certifix.y.z
charset-normalizerx.y.z
idnax.y.z
pipx.y.z
requestsx.y.z
setuptoolsx.y.z
urllib3x.y.z

Now you are ready to share your project! You can commit requirements.txt to a version control system like Git and use it to replicate the same environment on other machines. But wait, what happens if a new update is released for these packages?

fine-tuning requirements

The problem with hardcoding a package's version and dependencies is that the package is frequently updated with bug and security fixes. You may want to take advantage of these updates as soon as they are released.

The requirements file format allows you to specify dependency versions using comparison operators, which gives you the flexibility to ensure packages are updated while still defining the base version of the package.

Open requirements.txt in your favorite editor and convert the equality operator (==) to a greater than or equals operator (>=), as shown in the following example:

#requirements.txt

certifi>=x.y.z
charset-normalizer>=x.y.z
idna>=x.y.z
requests>=x.y.z
urllib3>=x.y.z

You can change the comparison operator to >= to tell pip to install the exact or higher version that was released. When you set up a new environment with that requirements.txt file, pip looks for the latest version that meets the requirements and installs it.

Next, you can upgrade the packages in the requirements file by running the install command with the switch or shorthand: --upgrade-U

Windows
Linux+macOS

(venv)C:\>python-mpipinstall-U-rrequirements.txt

If a new version of a listed package is available, that package will be upgraded.

Ideally, new versions of packages would be backward compatible and never introduce new bugs. Unfortunately, new versions may introduce changes that will break your application. To fine-tune your requirements, the requirements file syntax supports additional version specifiers.

Imagine a new version, 3.0, was released but requests introduced an incompatible change that broke your application. You can modify the requirements file to prevent version 3.0 or higher from being installed:

#requirements.txt

certifi==x.y.z
charset-normalizer==x.y.z
idna==x.y.z
requests>=x.y.z,<3.0
urllib3==x.y.z

Changing the version specifier of a package ensures that any version greater than or equal to requests will not be installed. 3.0 The pip documentation has extensive information on the requirements file format, you can consult it to learn more.

Separating production and development dependencies

Not all packages you install during application development are production dependencies. For example, you might want to test your application, so you need a testing framework. A popular testing framework is pytest. You want to install it in your development environment, but you don't want it in your production environment because it's not a production dependency.

You create a second requirements file, requirements_dev.txt, to list additional tools for setting up the development environment:

#requirements_dev.txt

pytest>=x.y.z

Having two requirements files will require you to install them using pip, requirements.txt and requirements_dev.txt. Fortunately, pip allows you to specify additional parameters in the requirements file, so you can modify requirements_dev.txt to install production requirements. Requirements in txt file:

#requirements_dev.txt

-rrequirements.txt
pytest>=x.y.z

Note that you use the same -r switch to install the production requirements.txt file. Now, in your development environment, you can install all the requirements by simply running this command:

Windows
Linux+macOS

(venv)C:\>python-mpipinstall-rrequirements_dev.txt

Because requirements_dev.txt contains the -rrequirements.txt line, you not only have to install, but also pytest to install requirements.txt. In a production environment, just installing the production requirements is enough:

Windows
Linux+macOS

(venv)C:\>python-mpipinstall-rrequirements.txt

With this command you can install requirements.txt. Your production environment will not have pytest installed compared to your development environment.

Production refrigeration requirements

You created production and development requirements files and added them to source control. These files use flexible version specifiers to ensure you take advantage of bug fixes released by dependencies. You've also tested your application and can now deploy it to production.

You know that all tests pass and that the application works with the dependencies you used during development, so you may want to make sure that the same versions of dependencies are deployed to production.

The current version specifier does not guarantee that the same version will be deployed to production, so you want to freeze production requirements until the project is released.

After completing development based on current requirements, the workflow for creating a new version of the current project is as follows:

With such a workflow, the requirements_lock.txt file will contain the exact version specifiers and can be used to replicate your environment. You have ensured that when your users install the packages listed in requirements_lock.txt into their own environment, they will use the version you want them to use.

Freezing your requirements is an important step in ensuring that your Python project works the same way in the user environment as it does in your environment.

uninstall package pip

Sometimes, you must uninstall a package. Either you found a better library to replace it, or it's something you don't need. Uninstalling packages can be a little tricky.

Note that when you install requests, you also have to pip install other dependencies. The more packages you have installed, the greater the chance that multiple packages depend on the same dependencies. This is where the show command inpip comes in handy.

Before uninstalling the package, make sure to run the show package command:

Windows
Linux+macOS

(venv)C:\>python-mpipshowrequests

Name:requests
Version:2.26.0
Summary:PythonHTTPforHumans.
Home-page:https://requests.readthedocs.io
Author:KennethReitz
Author-email:[email protected]
License:Apache2.0
Location:.../python3.9/site-packages
Requires:certifi,idna,charset-normalizer,urllib3
Required-by:

Note the last two fields Requires and Required-by. The show command tells you that requests require certifi, idna, charset-normalizer and urllib3. You might want to uninstall these too. Note that requests are not required for any other package. So it is safe to uninstall it.

You should run the show command against all dependencies, requests to make sure no other libraries also depend on them. Once you know the order of dependencies of the packages you want to uninstall, you can remove them with the uninstall command:

Windows
Linux+macOS

(venv)C:\>python-mpipuninstallcertifi

The uninstall command shows you the files that will be removed and asks for confirmation. If you're sure you want to remove the package because you've checked its dependencies and know that nothing else is using it, then you can pass a -y switch to suppress the file listing and confirmation dialog:

Windows
Linux+macOS

(venv)C:\>python-mpipuninstallurllib3-y

Here you uninstall urllib3. Using the -y switch, you can suppress the confirmation dialog asking if you want to uninstall this package.

In one call you can specify all packages to uninstall:

Windows
Linux+macOS

(venv)C:\>python-mpipuninstall-ycharset-normalizeridnarequests

You can pass multiple packages to the pipuninstall command. If you don't add any extra switches then you need to confirm uninstalling each package. By passing the -y switch you can uninstall them all without any confirmation dialog.

You can also uninstall all packages listed in the requirements file by providing the -r <requirementsfile> option. This command will prompt for a confirmation request for each package, but you can suppress it with the -y switch:

Windows
Linux+macOS

(venv)C:\>python-mpipuninstall-rrequirements.txt-y

Remember to always check the dependencies of the package you are uninstalling. You might want to uninstall all dependencies, but uninstalling packages used by others will break your working environment. As a result, your project may no longer work correctly.

If you are working in a virtual environment, you can reduce your workload by simply creating a new virtual environment. Then you can install the packages you need instead of trying to uninstall the ones you don't need. However, pipuninstall is useful when you need to uninstall a package from your system Python installation. Using pipuninstall is a great way to tidy up your system if you accidentally installed a system-wide package.

Explore the alternative pip

The Python community provides excellent tools and libraries for you to use in pip.pip These include alternatives that try to simplify and improve package management.

Here are some other package management tools available for Python:

Only pip is bundled in standard Python installations. If you want to use any of the alternatives listed above, you must follow the installation guidelines in their documentation. With so many options, you're sure to find the right tool for your programming journey!

in conclusion

Many Python projects use the pip package manager to manage their dependencies. It is included in the Python installer and is an essential tool for dependency management in Python.

In this tutorial, you learned how to:

pip is set up and running in your working environment
Fix common errors related to using pip
Install and uninstall packages pip
Define project and application requirements
Pin dependencies in requirements file

Additionally, you learned about the importance of keeping your dependencies up to date and the alternatives pip can help you manage those dependencies.

By taking a closer look at pip, you've explored an important tool in your Python development workflow. Using , you can install and manage any other package found on PyPIpip. You can use external packages from other developers as requirements and focus on the code that makes your project unique.

Click Follow to learn about HUAWEI CLOUD's new technologies for the first time~