Python learning--coroutine

Coroutine, also known as microthread, fiber. The English name is Coroutine.

The concept of coroutines was put forward very early, but it has not been widely used in certain languages ​​(such as Lua) until recent years.

Subroutines, or functions, are called hierarchically in all languages. For example, A calls B, B calls C during execution, C returns after execution, B returns after execution, and finally A is executed.

So the subroutine call is realized through the stack, and a thread is to execute a subroutine.

A subroutine call is always an entry, one return, and the calling sequence is clear. The call of a coroutine is different from a subroutine.

The coroutine seems to be a subroutine, but in the process of execution, it can be interrupted inside the subroutine, and then switch to execute other subroutines, and then return to continue execution when appropriate.

Note that interrupting in a subroutine to execute other subroutines is not a function call, but is somewhat similar to CPU interrupts . For example, subroutine A, B:

def A():
    print('1')
    print('2')
    print('3')

def B():
    print('x')
    print('y')
    print('z')

Assuming that it is executed by a coroutine, during the execution of A, it can be interrupted at any time to execute B. B may also be interrupted during the execution and then execute A. The result may be:

1
2
x
y
3
z

But in A, B is not called, so the call of the coroutine is more difficult to understand than the function call.

It seems that the execution of A and B is a bit like multi-threading, but the characteristic of coroutine is that it is executed by one thread. Compared with multi-threading, what are the advantages of coroutine?

The biggest advantage is the extremely high execution efficiency of the coroutine . Because subprogram switching is not thread switching, it is controlled by the program itself. Therefore, there is no thread switching overhead . Compared with multithreading, the more threads, the more obvious the performance advantage of coroutines.

The second major advantage is that there is no need for a multi-threaded locking mechanism , because there is only one thread, and there is no conflict of writing variables at the same time. In the coroutine, the shared resources are controlled without locking, and only the state is judged, so the execution efficiency is much higher. The thread is much higher .

Because the coroutine is executed by a thread, how to use a multi-core CPU? The simplest method is multi-process + coroutine , which not only makes full use of multi-core, but also gives full play to the high efficiency of coroutine to obtain extremely high performance.

Python's support for coroutines is achieved through generators.

(I said how to continue execution from the last yield position when it is executed next time, it turned out to be a coroutine)

In the generator, we can not only iterate through the for loop, but also continuously obtain the next value returned by the yield statement through the next() function.

But Python's yield can not only return a value, it can also receive parameters sent by the caller.

Look at the example:

The traditional producer-consumer model is that one thread writes messages, one thread fetches messages, and controls the queue and waiting through the lock mechanism, but it may deadlock if you are not careful.

If you switch to a coroutine, after the producer generates a message, it directly jumps to the consumer through yield to start execution. After the consumer finishes executing, the switch will cause the producer to continue production, which is extremely efficient:

def consumer():
	r = ''
	while True:
		n = yield r
		if not n:
			return
		print('[CONSUMER] Consuming %s...' % n)
		r = '200 OK'
	
def produce(c):
	c.send(None)
	n = 0
	while n < 5:
		n = n + 1
		print('[PRODUCER] Producing %s...' % n)
        r = c.send(n)
        print('[PRODUCER] Consumer return: %s' % r)
    c.close()

c = consumer()
produce(c)

Results of the:

[PRODUCER] Producing 1...
[CONSUMER] Consuming 1...
[PRODUCER] Consumer return: 200 OK
[PRODUCER] Producing 2...
[CONSUMER] Consuming 2...
[PRODUCER] Consumer return: 200 OK
[PRODUCER] Producing 3...
[CONSUMER] Consuming 3...
[PRODUCER] Consumer return: 200 OK
[PRODUCER] Producing 4...
[CONSUMER] Consuming 4...
[PRODUCER] Consumer return: 200 OK
[PRODUCER] Producing 5...
[CONSUMER] Consuming 5...
[PRODUCER] Consumer return: 200 OK

Note that the consumer function is a generator. After passing a consumer to produce:

1. First call c.send(None) to start the generator;
2. Then, once something is produced, switch to consumer execution via c.send(n);
3. The consumer gets the message through yield, processes it, and sends the result back through yield;
4. Produce gets the result of consumer processing and continues to generate the next message;
5. Produce decides not to produce, and closes the consumer through c.close(), and the whole process ends.

The entire process is lock-free and is executed by a thread. Produce and consumer cooperate to complete the task, so it is called "coroutine" instead of thread preemptive multitasking.

(This is what the generator does)

Finally, I use Donald Knuth's sentence to summarize the characteristics of the coroutine:

"A subroutine is a special case of a coroutine."