Python dynamic loading

Python generally has the following three ways:

Built-in __import__() function
importlib module
exec function

Finally, I will introduce the inspect module that is often used during dynamic loading.

import function

__import__ (name[, globals[, locals[, fromlist[, level]]]])

Parameter Description:

name (required): the name of the loaded module
globals (optional): A dictionary containing global variables. This option is rarely used and uses the default value global()
locals (optional): A dictionary containing local variables, which are not used by the internal standard implementation, and use the default value local()
fromlist (Optional): The name of the imported submodule
level (Optional): Import path option. The default in Python 2 is -1, which means that both absolute import and relative import are supported. The default value in Python 3 is 0, which means that only absolute import is supported. If it is greater than 0, it means the relative level of the imported parent directory, that is, 1 is similar to'.' and 2 is similar to'..'.

# test.py
def test():
	print("test")

class Test(object):
	def __init__(self):
		print("Test Create")
		
class SubTest(Test):
	def __init__(self):
		print("Test2 Create")

Load test.py dynamically

c = __import__('test')

print(c)

c.test()
obj = c.Test()
obj_sub = c.SubTest()
'''
<module 'test' from '**************\\test.py'>
test
Test Create
Test2 Create
'''

If the input parameter contains ".", using __import__ to directly import the module can easily cause unexpected results. OpenStack's oslo.utils encapsulates __import__ and supports dynamic import of class, object, etc.

# Copyright 2011 OpenStack Foundation.
# All Rights Reserved.
#
#    Licensed under the Apache License, Version 2.0 (the "License"); you may
#    not use this file except in compliance with the License. You may obtain
#    a copy of the License at
#
#         http://www.apache.org/licenses/LICENSE-2.0
#
#    Unless required by applicable law or agreed to in writing, software
#    distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
#    WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
#    License for the specific language governing permissions and limitations
#    under the License.

"""
Import related utilities and helper functions.
"""

import sys
import traceback


def import_class(import_str):
    """Returns a class from a string including module and class.
    .. versionadded:: 0.3
    """
    mod_str, _sep, class_str = import_str.rpartition('.')
    __import__(mod_str)
    try:
        return getattr(sys.modules[mod_str], class_str)
    except AttributeError:
        raise ImportError('Class %s cannot be found (%s)' %
                          (class_str,
                           traceback.format_exception(*sys.exc_info())))


def import_object(import_str, *args, **kwargs):
    """Import a class and return an instance of it.
    .. versionadded:: 0.3
    """
    return import_class(import_str)(*args, **kwargs)


def import_object_ns(name_space, import_str, *args, **kwargs):
    """Tries to import object from default namespace.
    Imports a class and return an instance of it, first by trying
    to find the class in a default namespace, then failing back to
    a full path if not found in the default namespace.
    .. versionadded:: 0.3
    .. versionchanged:: 2.6
       Don't capture :exc:`ImportError` when instanciating the object, only
       when importing the object class.
    """
    import_value = "%s.%s" % (name_space, import_str)
    try:
        cls = import_class(import_value)
    except ImportError:
        cls = import_class(import_str)
    return cls(*args, **kwargs)


def import_module(import_str):
    """Import a module.
    .. versionadded:: 0.3
    """
    __import__(import_str)
    return sys.modules[import_str]


def import_versioned_module(module, version, submodule=None):
    """Import a versioned module in format {module}.v{version][.{submodule}].
    :param module: the module name.
    :param version: the version number.
    :param submodule: the submodule name.
    :raises ValueError: For any invalid input.
    .. versionadded:: 0.3
    .. versionchanged:: 3.17
       Added *module* parameter.
    """

    # NOTE(gcb) Disallow parameter version include character '.'
    if '.' in '%s' % version:
        raise ValueError("Parameter version shouldn't include character '.'.")
    module_str = '%s.v%s' % (module, version)
    if submodule:
        module_str = '.'.join((module_str, submodule))
    return import_module(module_str)


def try_import(import_str, default=None):
    """Try to import a module and if it fails return default."""
    try:
        return import_module(import_str)
    except ImportError:
        return default


def import_any(module, *modules):
    """Try to import a module from a list of modules.
    :param modules: A list of modules to try and import
    :returns: The first module found that can be imported
    :raises ImportError: If no modules can be imported from list
    .. versionadded:: 3.8
    """
    for module_name in (module,) + modules:
        imported_module = try_import(module_name)
        if imported_module:
            return imported_module

    raise ImportError('Unable to import any modules from the list %s' %
                      str(modules))

importlib module

Load a module dynamically

import importlib

itertools = importlib.import_module('itertools')

importlib.import_module(name, package=None) parameter description:

name: What module to import in absolute or relative import mode (for example, either like this pkg.mod or this ..mod)
package: If the parameter name is specified by relative import, then the parameter packages must be set to the package name, which serves as the anchor point for parsing the package name (for example, import_module('..mod','pkg.subpkg' ) Will import pkg.md).

The import_module() function is a simplified wrapper for importlib.__import__(). This means that all the ideology of this function comes from importlib.__import__(). The most important difference between these two functions is that import_module() returns the specified package or module (such as pkg.mod), while __import__() returns the highest-level package or module (such as pkg).

If you dynamically import a module that has been created since the interpreter started executing (that is, a Python source code file is created), in order to let the import system know about the new module, invalidate_caches() may need to be called.

Determine whether a module can be loaded

import importlib.util
import sys

# For illustrative purposes.
name = 'itertools'

spec = importlib.util.find_spec(name)
if spec is None:
    print("can't find the itertools module")
else:
    # If you chose to perform the actual import ...
    module = importlib.util.module_from_spec(spec)
    spec.loader.exec_module(module)
    # Adding the module to sys.modules is optional.
    sys.modules[name] = module

The importlib.util.find_spec(name, package=None) parameter has the same meaning as importlib.import_module(name, package=None). It returns a namespace that contains relevant import information for loading modules. importlib.util.module_from_spec(spec) creates a new module from spec, and then you can use module as itertools. spec.loader.exec_module(module) Execute a module.

Load directly from file

import importlib.util
import sys

# For illustrative purposes.
import tokenize
file_path = tokenize.__file__
module_name = tokenize.__name__

spec = importlib.util.spec_from_file_location(module_name, file_path)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
# Optional; only necessary if you want to be able to import the module
# by name later.
sys.modules[module_name] = module

importlib.util.spec_from_file_location(name, location, *, loader=None, submodule_search_locations=None) It creates a ModuleSpec instance based on the path to a certain file. In the example, module can be used directly as tokenize.

Manage finder and loader through importer

import importlib.machinery
import sys

# For illustrative purposes only.
SpamMetaPathFinder = importlib.machinery.PathFinder
SpamPathEntryFinder = importlib.machinery.FileFinder
loader_details = (importlib.machinery.SourceFileLoader,
                  importlib.machinery.SOURCE_SUFFIXES)

# Setting up a meta path finder.
# Make sure to put the finder in the proper location in the list in terms of
# priority.
sys.meta_path.append(SpamMetaPathFinder)

# Setting up a path entry finder.
# Make sure to put the path hook in the proper location in the list in terms
# of priority.
sys.path_hooks.append(SpamPathEntryFinder.path_hook(loader_details))

Implement importlib.import_module function

import importlib.util
import sys

def import_module(name, package=None):
    """An approximate implementation of import."""
    absolute_name = importlib.util.resolve_name(name, package)
    try:
        return sys.modules[absolute_name]
    except KeyError:
        pass

    path = None
    if '.' in absolute_name:
        parent_name, _, child_name = absolute_name.rpartition('.')
        parent_module = import_module(parent_name)
        path = parent_module.__spec__.submodule_search_locations
    for finder in sys.meta_path:
        spec = finder.find_spec(absolute_name, path)
        if spec is not None:
            break
    else:
        msg = f'No module named {absolute_name!r}'
        raise ModuleNotFoundError(msg, name=absolute_name)
    module = importlib.util.module_from_spec(spec)
    spec.loader.exec_module(module)
    sys.modules[absolute_name] = module
    if path is not None:
        setattr(parent_module, child_name, module)
    return module

exec function

eval(expression, globals=None, locals=None)

exec(object[, globals[, locals]])

Parameter Description:

globals: Optional parameter, representing the global namespace (storing global variables). If provided, it must be a dictionary object. Usually passed in globals(), which returns a dictionary representing the current global identifier table. This is always the dictionary of the current module (inside a function or method, this refers to the module that defines the function or method, not the module that calls the function or method)
locals: Optional parameter, representing the current local namespace (storing local variables), if provided, it can be any mapping object. If this parameter is omitted, it will take the same value as globals. Usually passed in locals(), which updates and returns a dictionary representing the current local identifier table. When a free variable is called inside a function, it will be returned by the locals() function; when a free variable is not called, it will not be returned by the locals() function.

There are many similarities between the exec function and the eval function. The main differences are as follows:

The eval() function can only calculate the value of a single expression, while the exec() function can dynamically run code segments.
The eval() function can have a return value, and the exec() function return value is always None.

x = 10

def func():
    y = 20
    a = eval('x + y')
    print('a: ', a)
    b = eval('x + y', {'x': 1, 'y': 2})
    print('b: ', b)
    c = eval('x + y', {'x': 1, 'y': 2}, {'y': 3, 'z': 4})
    print('c: ', c)
    d = eval('print(x, y)')
    print('d: ', d)

func()

'''
a:  30
b:  3
c:  4
10 20
d:  None
'''

x = 10
expr = """
z = 30
sum = x + y + z
print(sum)
"""
def func():
    y = 20
    exec(expr)
    exec(expr, {'x': 1, 'y': 2})
    exec(expr, {'x': 1, 'y': 2}, {'y': 3, 'z': 4})
    
func()
'''
60
33
34
'''

inspect module

Judgment of operations on classes, modules, members, classes, and module types
Get the source code
Get parameter information of a class or function
Parsing stack

inspect.getmembers(object[, predicate])

It implements the extraction of all members in an object and returns it as a list of (name, value) pairs. The second parameter can usually call the following 16 methods as needed:

inspect.ismodule(object): whether it is a module
inspect.isclass(object): whether it is a class
inspect.ismethod(object): Whether it is a method (bound method written in python)
inspect.isfunction(object): Whether it is a function (python function, including lambda expression)
inspect.isgeneratorfunction(object): Whether it is a python generator function
inspect.isgenerator(object): Whether it is a generator
inspect.istraceback(object): Whether it is traceback
inspect.isframe(object): whether it is a frame
inspect.iscode(object): whether it is code
inspect.isbuiltin(object): Whether it is a built-in function or a built-in method
inspect.isroutine(object): Whether it is a user-defined or built-in function or method
inspect.isabstract(object): Whether it is an abstract base class
inspect.ismethoddescriptor(object): Is it a method identifier
inspect.isdatadescriptor(object): Whether it is a digital identifier, digital identifiers have __get__ and __set__ attributes; usually they also have __name__ and __doc__ attributes
inspect.isgetsetdescriptor(object): Whether it is a getset descriptor
inspect.ismemberdescriptor(object): Whether it is a member descriptor