In python, explain the import mechanism in detail (1)

Get into the habit of writing together! This is the 12th day of my participation in the "Nuggets Daily New Plan · April Update Challenge", click to view the details of the event .

If a language wants to become bigger and stronger, it needs a module management system. For example, the manifold of js is indispensable for the package management tool npm. Rust has cargo and golang language has been complained before, but there is no official package management tool.

Python programs usually need to introduce some standard libraries or third-party libraries, and projects also need to be organized and managed in modules. Therefore, understanding python's import mechanism and how to manage modules is a problem faced by developers. The sys package provides an import mechanism to help import modules.

Get into the habit of writing together! This is the 12th day of my participation in the "Nuggets Daily New Plan · April Update Challenge", click to view the details of the event .

I have shared a few articles about what happened behind importing modules in python before. The content of the articles is a bit scattered. Today, I will try to sort out the ideas.

import socket
import numpy as np
from tqdm import tqdm
复制代码

The process of loading modules __import__()is ,

• First, it will search for the module sys.modulesto be loaded. If the module has already been loaded, then you can sys.modulesget the module directly from the cache
• Usually the path we want to import module is a hierarchical, similar file path, so that is to start from the top-level directory according to the module path, then during the loading process, it will first traverse sys.meta_paththe MetaPathFinder to select a suitable Finder. Think of it as a searcher. After selecting the finder, you can call the find_modulemethod and pass the path to this function. This function will return a loader object. This loader object has a load_modulemethod to load the module you want to load.
• Loading a module will also store the module in sys.modulesthis cache. The next time the module is loaded again, the module will be sys.modulesobtained directly from , to avoid repeatedly loading the same module.

>>> print(sys.meta_path)
[<class '_frozen_importlib.BuiltinImporter'>, <class '_frozen_importlib.FrozenImporter'>, <class '_frozen_importlib_external.PathFinder'>]
复制代码

• BuiltinImporter is used to load kernel-level modules such as sys, os such modules
• FrozenImporter is used to import frozen bytecode modules. In fact, these modules have been compiled into python's interpreter. This finder is usually only used to initialize Python's import mechanism. Directly load modules that can be run without a python interpreter
• PathFinder We know from the name that this is a support path search. This is a filesystem-based resource loading implementation for importing, which can be used to import .py, .pyc, and .zip files. This is a meta path finder, most modules are imported this way

class CustomFinder(object):
    def find_module(self, fullname,path=None):
        print(f"Looking for {fullname},{path}")
        return None


import sys
sys.meta_path.insert(0,CustomFinder())

import math
import xml.etree.ElementTree
复制代码

You can define one yourself. You CustomFinderneed to implement a find_module method inside the class. This method should be called by the interpreter, and two parameters will be injected when calling the function. Then simply output these two parameters inside the function. sys.meta_pathThen register it, and then we import the statement to try to import the two modules are math and xml.etree.ElementTree.

print(sys.meta_path)
复制代码

You can search through the sys.meta_path command to find that CustomFinder already exists in the meta_path list.

[<__main__.CustomFinder object at 0x10c0d8430>, <class '_frozen_importlib.BuiltinImporter'>, <class '_frozen_importlib.FrozenImporter'>, <class '_frozen_importlib_external.PathFinder'>]
复制代码

We found that we can search the xml path and return None. Why continue to search down? This is because if CustomFinder continues to search down through other finder.

Looking for xml,None
Looking for xml.etree,['/anaconda3/envs/py38/lib/python3.8/xml']
Looking for xml.etree.ElementTree,['/anaconda3/envs/py38/lib/python3.8/xml/etree']
复制代码