Functions in Python are also Objects

The doubts encountered when studying the CS 61A Higher-Order Functions chapter were suddenly enlightened after reading this Zhihu answer .

Everything is an object in Python

This is probably the most useful sentence to learn Python. Presumably you already know Python's built-in data structures such as list, tuple, dict, etc. When you execute:

alist = [1, 2, 3]

, you create a list object and reference it with the alist variable:

Of course you can also define a class yourself:

class House(object):
    def __init__(self, area, city):
        self.area = area
        self.city = city

    def sell(self, price):
        [...]  #other code
        return price

Then create an object of the class:

house = House(200, 'Shanghai')

OK, you immediately have a 200-square-meter house in Shanghai. It has some properties (area, city)and some methods (__init__, self):

Functions are first-class objects

Like list, tuple, dict, and objects created with House, when you define a function, functions are objects:
def func(a, b):
return a+b

In the global scope, the function object is referenced by the function name, it receives two parameters a and b, and calculates the sum of these two parameters as the return value.
The so-called first-class object means that the object can be named with an identifier, and the object can be treated as data, such as assigning a value, passing it as a parameter to a function, or returning it as a return value, etc.

Therefore, you can refer to the function object by any other variable name:
add = func

This way you can call the function with the new reference just as you would call func(1, 2):
print func(1, 2)
print add(1, 2) #the same as func(1, 2)

Or pass a function object as a parameter to another function:
def caller_func(f):
return f(1, 2)

if name == "main":
print caller_func(func)

It can be seen that the
function object func is passed as a parameter to the caller_func function, and the parameter passing process is similar to an assignment operation f=func;
so the func function object is referenced by the local variable f in the scope of the caller_func function, and f actually points to the function func; When cc
executes return f(1, 2), it is equivalent to executing return func(1, 2);
therefore, the output result is 3.

1. Function Objects vs Function Calls
   Whether a function is assigned to a new identifier or passed as a parameter to a new function, it is the function object itself, not the function call.

Let's illustrate this with a simpler, but visually, more confusing example. For example, the following function is defined:
def func():
return "hello,world"

Then perform two assignments respectively:
ref1 = func #Assign the function object to ref1
ref2 = func() #Call the function and assign the return value of the function ("hello,world" string) to ref2

Many beginners will confuse these two assignments. Through the built-in type function in Python, you can check the results of these two assignments:
In [4]: ​​type(ref1)
Out[4]: function

In [5]: type(ref2)
Out[5]: str

As you can see, ref1 refers to the function object itself, while ref2 refers to the return value of the function. Through the built-in callable function, you can further verify that ref1 is callable and ref2 is not:
In [9]: callable(ref1)
Out[9]: True

In [10]: callable(ref2)
Out[10]: False

The effect of passing parameters is similar.

1. Closure & LEGB Law
   The so-called closure is the object obtained when the statements that make up a function and the execution environment of these statements are packaged together

It does sound a bit complicated, so let's use a chestnut to help understand it. Suppose we have the following definition in the foo.py module:

foo.py

filename = "foo.py"

def call_func(f):
return f() #As described earlier, f refers to a function object and then calls it

In another func.py module, write code like this:

func.py

import foo #import foo.py

filename = "func.py"
def show_filename():
return "filename: %s" % filename

if name == " main ":
print foo.call_func(show_filename) #Note: The actual location of the call is in the foo.call_func function

When we execute func.py with the python func.py command, the output is:
chiyu@chiyu-PC :~$ python func.py
filename:func.py

Obviously the value of the filename variable used by the show_filename() function is the one defined in the same environment as it (the func.py module). Although the filename variable with the same name is also defined in the foo.py module, and the actual call to show_filename is also inside the call_func of foo.py.

For nested functions, the mechanism is more obvious: the closure will capture the entire environment required for the inner function to execute:

enclosed.py

import foo
def wrapper():
filename = "enclosed.py"
def show_filename():
return "filename: %s" % filename
print foo.call_func(show_filename) #输出：filename: enclosed.py

In fact, each function object has a __globals__ attribute that points to the global namespace where the function is defined:

show_filename inside wrapper

show_filename.globals

{
' builtins ': <module ' builtin ' (built-in)>, #built-in scope environment
' file ': 'enclosed.py',
'wrapper': , #direct peripheral environment
' package ': None,
' name ': ' main ',
'foo': <module 'foo' from '/home/chiyu/foo.pyc'>, #global environment
' doc ': None
}

When the code executes to the return "filename: %s" % filename statement in show_filename, the parser looks for the filename variable in the following order:
Local - inside a local function (show_filename), assigned by any means, and not used by the global keyword The filename variable declared as a global variable;
Enclosing - the local scope of the immediate surrounding space (upper function wrapper), look for the filename variable (if there are multiple levels of nesting, search from the inside out to the outermost function) ;
Global - the global space (module enclosed.py), the filename variable assigned at the top level of the module;
Builtin - the filename variable is found in the predefined variable names in the built-in module ( builtin
); the filename variable that meets the requirements is found first at any level , then no longer look to the outer layers. If the required variable is not found until the Builtin layer, a NameError exception is thrown. That's what variable name resolution is for: LEGB's Law.

Summary:
The most important use value of closures is to seal the context of function execution;
in the execution environment captured by the closure (the context where the def statement block is located), the closure also follows the LEGB rules to search layer by layer until it finds a variable that meets the requirements. Or throw an exception.

1. Decorators & Syntax Sugar
   So what do closures have to do with decorators?

The important feature of closures mentioned above: sealing the context, this feature can be cleverly used for the packaging of existing functions, so as to make existing functions more functional. And that's the decorator.

Or as an example, the code is as follows:

alist = [1, 2, 3, ..., 100] --> 1+2+3+...+100 = 5050

def lazy_sum():
return reduce(lambda x, y: x+y, alist)

We define a function lazy_sum that sums all the elements in the alist and returns it. alist is assumed to be a list of integers from 1 to 100:
alist = range(1, 101)

But for some reason, I don't want to return the result of the calculation right away, but somewhere later, output the result via the displayed call. So I wrap it with a wrapper function:
def wrapper():
alist = range(1, 101)
def lazy_sum():
return reduce(lambda x, y: x+y, alist)
return lazy_sum

lazy_sum = wrapper() #wrapper() returns the lazy_sum function object

if name == "main":
lazy_sum() #5050

This is a typical example of Lazy Evaluation. We know that in general, local variables will be reclaimed by the garbage collector when the function returns, and can no longer be used. But the alist here is not, it is returned with the return of the lazy_sum function object (this statement is not accurate, it is actually included in the execution environment of lazy_sum, through __globals__), thus extending the life cycle.

When lazy_sum() is called in the if block, the parser will find the alist list from the context (here, the local scope of the wrapper function of the Enclosing layer), calculate the result, and return 5050.

When you need to dynamically add functionality to a defined function, such as parameter checking, a similar principle becomes useful:
def add(a, b):
return a+b

This is a very simple function: calculate the sum and return of a+b, but we know that Python is a dynamically typed + strongly typed language. You cannot guarantee that the parameters a and b passed in by the user must be two integers. He has An integer and a string may be passed in:
In [2]: add(1, 2)
Out[2]: 3

In [3]: add(1.2, 3.45)
Out[3]: 4.65

In [4]: add(5, 'hello')

TypeError Traceback (most recent call last)
/home/chiyu/ in ()
----> 1 add(5, 'hello')

/ home / chiyu / in add(a, b)
1 def add(a, b):
----> 2 return a+b

TypeError: unsupported operand type(s) for +: 'int' and 'str'

So, the parser mercilessly throws a TypeError exception.

Dynamic typing: determine the type of a variable at runtime, python determines the type of a variable when you first assign a value to it;
strong typing: there is a forced type definition, you have an integer, unless explicitly cast, otherwise Never treat it as a string (e.g. try + operation directly with an integer and a string);

Therefore, in order to use the add function more elegantly, we need to check the parameters of a and b before executing the + operation. This is where decorators are very useful:
import logging

logging.basicConfig(level = logging.INFO)

def add(a, b):
return a + b

def checkParams(fn):
def wrapper(a, b):
if isinstance(a, (int, float)) and isinstance(b, (int, float)): #Check whether parameters a and b are both integers or floats Point type
return fn(a, b) #If yes, call fn(a, b) to return the calculation result

    #否则通过logging记录错误信息，并友好退出
    logging.warning("variable 'a' and 'b' cannot be added")   
    return
return wrapper     #fn引用add，被封存在闭包的执行环境中返回

if name == " main ": #Pass in
the add function object, fn points to add
#add on the left side of the equal sign, points to the return value of checkParams wrapper
add = checkParams(add)
add(3, 'hello') #pass type Check, don't calculate the result, but log and exit

Pay attention to the checkParams function:
first look at the parameter fn, when we call checkParams(add), it will become a local (Local) reference to the function object add;
inside checkParams, we define a wrapper function that adds parameter type checking function, then fn(a, b) is called, according to the LEGB law, the interpreter will search several scopes and finally find fn in the local scope of the (Enclosing layer) checkParams function;
note the return wrapper at the end, which will Create a closure, the fn variable (a reference to the add function object) will be sealed in the execution environment of the closure and will not be recycled with the return of checkParams;
when add = checkParams(add) is called, add points to The new wrapper object, which adds the function of parameter checking and logging, and can continue to call the original add for + operation through the sealed fn.

So calling add(3, 'hello') will not return the calculation result, but print out the log:
chiyu@chiyu-PC :~$ python func.py
WARNING:root:variable 'a' and 'b' cannot be added

Some people think that the way add = checkParams(add) is too cumbersome, so python provides a more elegant way of writing, called syntactic sugar:
@checkParams
def add(a, b):
return a + b

This is just a writing optimization, the interpreter will still convert it to add = checkParams(add) to execute.

1. Regression problem
   def addspam(fn):
   def new( args):
   print "spam,spam,spam"
   return fn( args)
   return new

@addspam
def useful(a,b):
print a2+b2

First look at the second piece of code:
@addspam decorator, which is equivalent to executing useful = addspam(useful). There is a misunderstanding here: the parameter passed to addspam is the useful function object itself, not a result of its invocation;

Go back to the addspam function body:
return new returns a closure, fn is enclosed in the execution environment of the closure, and will not be recycled with the return of the addspam function;
and fn is a reference of useful at this time, when the return fn is executed ( args), it is actually equivalent to executing return useful( args);

Finally, a reference diagram of the code execution process is attached, hoping to help you understand: