Before a simple analysis of closures in Python, let's first understand the scoping rules in Python. There are many blog posts about the detailed knowledge of scope in Python. Here we start with a simple example.
Scope in Python
Suppose the following function is defined on the interactive command line:
>>> a = 1
>>> def foo():
b = 2
c = 3
print "locals: %s" % locals()
return "result: %d" % (a + b +c)
>>> a = 1
>>> def foo():
b = 2
c = 3
print "locals: %s" % locals()
return "result: %d" % (a + b +c)
The above code first assigns a value of 1 to a, and then defines a function: foo(). In the function foo(), we define two integers b and c, and the return value of the function is the sum of three numbers a, b, and c.
Verify the above function:
# result
>>> foo()
locals: {'c': 3, 'b': 2}
result: 6
# result
>>> foo()
locals: {'c': 3, 'b': 2}
result: 6
According to the result of the verification, the return value of the foo() function is 6. In the above function definition, there are only two assignments of variables b and c. How does the calling function judge the value of a? This involves scoping rules for functions. This article excerpts the relevant discussion in the "Python Reference Manual (4th Edition)":
Each time a function is executed, a local namespace is created. The namespace represents a local environment that contains the names of function parameters and the names of variables assigned within the function body. When parsing these names:
The interpreter will first search the local namespace;
If no matching name is found, it searches the global namespace (the global namespace for a function is always the module in which the function is defined);
If the interpreter also cannot find a match in the global namespace, it will eventually check the built-in namespace;
If no matching value is found in the built-in namespace, a NameError exception is raised.
Corresponding to the above example, the foo function will first look for matching values for three variables in the local namespace. The locals() method in the above code gives the contents of the local namespace of the foo function. As you can see, the local namespace is a dictionary containing the values of b and c, because we defined these two variables in the foo function. However, the local namespace does not contain the value of a, so it needs to be looked for in the global namespace. You can use __globals__ to get the local namespace of a function.
# The global namespace for the foo function
>>> foo.__globals__
{'a': 1, '__builtins__': <module '__builtin__' (built-in)>, '__package__': None, '__name__': '__main__', 'foo': <function foo at 0x0000000004613518>, '__doc__': None}
# The global namespace for the foo function
>>> foo.__globals__
{'a': 1, '__builtins__': <module '__builtin__' (built-in)>, '__package__': None, '__name__': '__main__', 'foo': <function foo at 0x0000000004613518>, '__doc__': None}
The global namespace of the foo function contains the built-in function module, the foo function, the variable a, and some other parameters. Since the variable a is found in the global namespace of the foo function, the foo function returns the sum of the three variables.
Python closures
The above Python scoping rules are general. However, in Python "everything is an object", and functions are no exception. This means that functions can be passed as parameters to other functions, placed in data structures, and used as the return result of functions. What happens to Python's scoping rules in this case? Let's take another example:
>>> def foo():
a = 1
def bar():
b = 2
c = 3
return a + b + c
return bar
>>> def foo():
a = 1
def bar():
b = 2
c = 3
return a + b + c
return bar
In this example, we define a function foo and assign a value to the variable a. But the difference from the previous example is that in the function foo, we also nest a function bar, and also define two variables, this function is used as the return value of the function foo. According to the above scope rules, the local scope of the function foo is neither the local scope of the function bar nor its global scope. Can the function bar correctly match the value of the variable a? Let's verify that this function works correctly.
# call function foo() >>> bar = foo() # return value bar is a function >>> bar <function bar at 0x00000000045F3588> # call bar() >>> bar() # The result is displayed as the sum of the three variables 6
The above verification results show that, in the above nested functions, the inner function can correctly refer to the variables of the outer function, even if the outer function has returned.
This behavior that the local scope of the inner function can access the variables in the local scope of the outer function is called: closure. The feature that the inner function can access the variables of the outer function is much like "packaging" the variables of the outer function directly into the inner function. We can also understand the closure in this way: the statements that make up the function and the environment that executes these statements are "packaged" together The resulting object is called a closure.
Several objects related
to closures In order to understand how closures implement the reference of internal functions to external function variables, it is necessary to introduce several objects related to closures. Regarding these objects, the underlying implementation of Python will be involved, which will not be described in detail in this article. You can refer to the following articles:
However, in order to intuitively explain the implementation process of the closure (without analyzing the underlying implementation), here is a brief introduction to the following code objects. A code object refers to a code object that represents executable Python code compiled into bytes, or bytecode. It has several more important properties:
co_name: the name of the
function co_nlocals: the number of local variables used by the function
co_varnames: a tuple containing the names of local variables
co_cellvars: a tuple containing the names of local variables referenced by nested functions
co_freevars: a tuple , which holds the variable names used in the outer scope
co_consts: is a tuple containing the literals used by the bytecode
The key ones are the two attributes co_varnames and co_freevars. We slightly modify the above example:
Python
>>> def foo():
a = 1
b = 2
def bar():
return a + 1
def bar2():
return b + 2
return bar
>>> bar = foo()
# outer function
>>> foo.func_code.co_cellvars
('a', 'b')
>>> foo.func_code.co_freevars
()
# inner nested function
>>> bar.func_code.co_cellvars
()
>>> bar.func_code.co_freevars
('a',)
>>> def foo():
a = 1
b = 2
def bar():
return a + 1
def bar2():
return b + 2
return bar
>>> bar = foo()
# outer function
>>> foo.func_code.co_cellvars
('a', 'b')
>>> foo.func_code.co_freevars
()
# inner nested function
>>> bar.func_code.co_cellvars
()
>>> bar.func_code.co_freevars
('a',)
The above shows that the co_cellvars of the code object of the outer function saves the names of the variables that the inner nested function needs to refer to, and the co_freevars of the code object of the inner nested function saves the names of the variables that need to be referenced in the scope of the outer function. Specifically, two functions are nested in the foo function, and they both need to refer to variables in the local scope of the foo function, so foo.func_code.co_cellvars contains the names of variable a and variable b. The function bar is the return value of foo and only references the variable a, so bar.func_code.co_freevars only contains the variable a.
The correspondence between the co_freevars and co_cellvars of the inner function and the outer function makes the inner function have a special attribute __closure__ of a closure during the function compilation process (there is a related implementation in the bottom layer). The __closure__ attribute is a tuple of cell objects containing variables referenced by multiple scopes. The following verifications can be done:
>>> foo.__closure__ #None # The reference of the internal function bar to the variable a >>> bar.__closure__ (<cell at 0x00000000044F6798: int object at 0x0000000003FA4B38>,) # The value of the variable a referenced by the inner function bar >>> bar.__closure__[0].cell_contents 1