Python automatically import the missing library

Original: Deep: Python how to automatically import the missing library?

Author: Pea cat

When writing Python projects, we may often encounter an error Import Module failure: ImportError: No module named 'xxx'or ModuleNotFoundError: No module named 'xxx'.

Import failures, usually divided into two types: one is to import the module to write their own (ie as a suffix .py file), and the other is import-party libraries. This article focuses on the second case, in the future have the opportunity, we discuss in detail other related topics.

Importing Python library failed to solve the problem, in fact, the key is on the operating environment are installed with missing libraries (note whether the virtual environment), or use an appropriate alternative. The issue is divided into three cases:

A single missing library module

In the preparation of the code, if we need to use a tripartite libraries (such as requests), but not sure of the actual operating environment is installed it, you can do:

try:
    import requests
except ImportError:
    import os
    os.system('pip install requests')
    import requests

Wrote the effect that, if requests can not find the library, on the first installation, and then import.

In some open source projects, we may also see the following lines (to json for example):

try:
    import simplejson as json
except ImportError:
    import json

Wrote effect, priority tripartite import library simplejson, if not, then use the built-in standard library json.

The benefits of this writing there is no need to import additional libraries, but it has the disadvantage that the need to ensure that the use of two libraries are compatible, if no alternative library in the standard library, it is not feasible .

If you really can not find a compatible standard library, you can also write your own modules (such as my_json.py), to achieve what you want, and then import it except statement.

try:
    import simplejson as json
except ImportError:
    import my_json as json

Second, the entire project missing libraries

The above idea is for the development of the project, but it has several disadvantages: 1, in the code for each tripartite library may be missing are pip install, is not desirable; 2, a tripartite standard library or libraries can not be yourself handwritten Librarian, how to do? 3, the project has been formed, these modifications are not allowed to do how to do?

So the question here is: have a project that you want to deploy to the new machine, it involves a lot of tripartite library, but they are not pre-installed on the machine, how to do?

For a project compliance, in accordance with the agreement, it will usually contain a " requirements.txt " documents, records of all the project dependencies and their version numbers required. This is before the project release, use the command pip freeze > requirements.txt generates.

Use the command pip install -r requirements.txt (executed in the file directory, or write the whole file in the command path), it will automatically give all the dependent libraries installed.

However, if the project is not compliant, or for other reasons bad, we do not have such a document, should you do?

A stupid way is to put the project up and running, wait for it to go wrong, you encounter a guide library fails, a manually loaded, then run again once the project encountered guide library failed to install it, and so on ...... (omitted here 1 ten thousand bad language) ......

Third, the automatic import any missing libraries

Is there a better method can be imported automatically missing libraries it?

Without modifying the original code, without the need for "requirements.txt" file, there is no way to automatically import library need it?

Of course there is! First look at the results:

We tornado, for example, the first operation can be seen, we have not been installed tornado, after the second operation, import tornado again, the program will help us to automatically download and install the tornado, so no error.

autoinstall our handwritten module code is as follows:

# 以下代码在 python 3.6.1 版本验证通过
import sys
import os
from importlib import import_module


class AutoInstall():
    _loaded = set()

    @classmethod
    def find_spec(cls, name, path, target=None):
            if path is None and name not in cls._loaded:
                cls._loaded.add(name)
                print("Installing", name)
                try:
                    result = os.system('pip install {}'.format(name))
                    if result == 0:
                        return import_module(name)
                except Exception as e:
                    print("Failed", e)
            return None

sys.meta_path.append(AutoInstall)

This code is used in sys.meta_path our first print to see if it is what?

Python import mechanism 3 is in the discovery process, substantially in the following order:

Find in sys.modules, which caches all imported modules
Find in sys.meta_path, and it supports custom loader
Find in sys.path, which recorded some of the name of the directory where the library
If not found, throw ImportErroran exception

Which should be noted, sys.meta_path vary in different versions of Python, such as its large in Python 2 and Python 3 the difference; in the newer Python 3 version (3.4+), a custom loader needs to implement find_specmethod, and the early version used is find_module.

The code above is a custom classloader AutoInstall, you can achieve the purpose of automatically import library constituents. Need to explain, this approach would "hijack" all newly imported library, destroy the original import way, so some weird problem may also occur, please pay attention.

sys.meta_path use Python probe belongs. Probe, that import hookis the Python almost unnoticed, but it can do many things, such as loading the library on the network, the module is modified when importing module, automatically install the missing library, upload audit information, such as lazy loading Wait.

Due to space limitations, we are no longer lays out in detail. Last Summary about:

You can try ... except way to achieve a simple three library or import replacement
When the total deletion of the known dependencies (e.g. requirements.txt), can manually install
Using sys.meta_path, you can automatically import any deletion library

References:

https://github.com/liuchang0812/slides/tree/master/pycon2015cn

http://blog.konghy.cn/2016/10/25/python-import-hook

https://docs.python.org/3/library/sys.html#sys.meta_path