When it comes to concurrent programming, what everyone can easily think of is: process, thread, coroutine, asynchronous IO. The four have something in common in terms of implementation, which is nothing more than the word scheduling.
Process: operating system process system scheduling, scheduling number: pid, basically provided by the operating system for scheduling support
Thread: operating system thread scheduling, scheduling number: TCB, virtual machine provides some support
Coroutine: the program schedules itself, scheduling number: function name, all done by the program itself.
Asynchronous IO: The message middleware is responsible for scheduling, scheduling number: message queue.
Processes, threads, and coroutines implement time multiplexing to achieve logical synchronization; while asynchronous IO only implements decoupling of applications, and must cooperate with one or more of the first three to achieve concurrent deal with;
The following is a special talk about coroutines
Coroutines, as the name suggests, are programs that cooperate with each other. There can be many coroutines in a program. These coroutines behave as if they are executed concurrently under the scheduling of the main thread; the advantage of this is that it is lightweight.
People often say: Threads are lightweight processes; when it comes to coroutines, we can say that coroutines are the most lightweight unit of execution; nothing is lighter than coroutines; a coroutine can be a function Or any callable object, its scheduling is completely controlled by the program itself, so its scheduling overhead can be minimized according to its own needs. This is its advantage;
As mentioned at the beginning of this article, the core idea of dealing with concurrency is: scheduling; so don't think how new coroutines are; you have used it as early as when you wrote Tetris programs in C; Tetris needs a special The functions that count and display the score also need a function that controls the movement and flipping of the block. These two functions need to be executed concurrently; your approach at that time is definitely not multi-threading, but like the following:
while(1){
sleep(0.05)
counter();
move();
}Now you know what a coroutine is, the counter() move() above
Let's talk about the three realms of python coroutines from the shallower to the deeper: "coroutines implemented by themselves", "", "coroutines implemented by third-party libraries"
When it comes to concurrency, the simplest model is the producer-consumer model: the producer produces a product at regular intervals, and the consumer also consumes a product at regular intervals. The producer generally thinks that there is only one, and the consumer can have multiple indivual;
self-implemented coroutines
Go directly to the code, a bit similar to the implementation of Tetris
from time import sleep,time
from Queue import Queue
queue=Queue(10)
class producer():
def __init__(self, interval):
self.interval = interval
self.count = 1
self.state = "free"
def run(self):
if self.state == "free":
if not queue.full():
self.start_time = time()
print "producer is making cake[%d]"%(self.count)
self.state = "busy"
else:
print "more than 10 cakes exits!,producer is waiting"
if self.state == "busy":
if time()-self.start_time > self.interval:
queue.put("cake[%s]"%self.count)
self.count+=1
self.state = "free"
class consumer():
def __init__(self,name, interval, producer):
self.interval=interval
self.producer=producer
self.state="free"
self.cake=None
self.name=name
def run(self):
if self.state=="free":
if not queue.empty():
self.cake=queue.get()
self.start_time = time()
self.state="busy"
else:
print "no cake exits, %s is waiting!"%self.name
if self.state=="busy":
if time() - self.start_time > self.interval:
print "%s finished eating %s"%(self.name, self.cake)
self.state="free"
def main():
p = producer(1)
c1 = consumer("mom",2, p)
c2 = consumer("child",3, p)
while(True):
sleep(0.1)
p.run()
c1.run()
c2.run()
main()
Coroutines implemented using the language feature yield
Again, to achieve concurrency is to achieve scheduling; and what is the essence of scheduling? It is the traffic police uncle's gesture to stop a car and make a car run; our car here is a function; if there is no yield function in python, we can't make him stop halfway through the execution, so we will be like As above, this function is called every time in the while loop, and the state is stored through the class object, making it seem like something is running and listening; with the language feature of yield, we can directly treat the state as a function state; this makes us have to do a lot less work, such as sleep(0.1) above
code show as below:
from Queue import Queue
from time import time
queue=Queue(10)
def producer(name, interval):
num=0
state="free"
while True:
if state=="free":
if not queue.full():
state="busy"
start_time=time()
print "%s is making cake[%d]"%(name, num)
num+=1
else:
print "more than 10 cakes exists, %s is waiting"%name
yield
continue
if state=="busy":
if time()-start_time > interval:
queue.put("cake[%d]"%num)
state="free"
yield
continue
def consumer(name, interval):
state = "free"
while True:
if state == "free":
if not queue.empty():
cake = queue.get()
start_time = time()
state = "busy"
else:
print "no cake exists, %s is waiting"%name
yield
continue
if state == "busy":
if time()-start_time > interval:
print "%s finished eating %s"%(name,cake)
state = "free"
yield
continue
sister=consumer("sister",2)
brother=consumer("brother",3)
mom = producer("mom",1)
while True:
mom.next()
brother.next()
sister.next()
Let's make another improvement to make it look more thread-like, that is, instead of while True like the above in the code, use start()
from Queue import Queue
from time import time
class Routine(object):
def __init__(self, name, target=None, parent=None):
self.name=name
self.parent=parent
self.children=[]
self.target=target
def add(self, child):
self.children.append(child)
def next(self):
if self.target:
return self.target.next()
def start(self):
if self.parent:
self.parent.add(self)
else:
while True:
for i in self.children:
i.next()
def producer(name, interval):
num=0
state="free"
while True:
if state=="free":
if not queue.full():
state="busy"
start_time=time()
print "%s is making cake[%d]"%(name, num)
num+=1
else:
print "more than 10 cakes exists, %s is waiting"%name
yield
continue
if state=="busy":
if time()-start_time > interval:
queue.put("cake[%d]"%num)
state="free"
yield
continue
def consumer(name, interval):
state = "free"
while True:
if state == "free":
if not queue.empty():
cake = queue.get()
start_time = time()
state = "busy"
else:
print "no cake exists, %s is waiting"%name
yield
continue
if state == "busy":
if time()-start_time > interval:
print "%s finished eating %s"%(name,cake)
state = "free"
yield
continue
queue=Queue(10)
sister=consumer("sister",2)
brother=consumer("brother",3)
mom = producer("mom",1)
main_routine=Routine(__name__)
routineA=Routine("A",sister,main_routine)
routineB=Routine("B",brother,main_routine)
routineC=Routine("C",mom,main_routine)
routineA.start()
routineB.start()
routineC.start()
main_routine.start()
In the above improvements, we have encapsulated the implementation of the coroutine to a certain extent, but there are still some shortcomings; for example, it needs to define the generator itself; of course, if it is encapsulated in the way of "self-implemented coroutine", it is not necessary. generator; so is there a wrapper that someone else has already written? Or is there a better coroutine library? Because we need professional people to do professional things, building a car yourself is mostly thankless, requiring a lot of testing and optimization, and it will not work without the power of a community;
Coroutines implemented using third-party libraries
The code below is taken from:
from gevent import monkey; monkey.patch_all()
import gevent
import urllib2
def f(url):
print('GET: %s' % url)
resp = urllib2.urlopen(url)
data = resp.read()
print('%d bytes received from %s.' % (len(data), url))
gevent.joinall([
gevent.spawn(f, 'https://www.python.org/'),
gevent.spawn(f, 'https://www.yahoo.com/'),
gevent.spawn(f, 'https://github.com/'),
])Its output is as follows:
GET: https://www.python.org/
GET: https://www.yahoo.com/
GET: https://github.com/
45661 bytes received from https://www.python.org/.
14823 bytes received from https://github.com/.
304034 bytes received from https://www.yahoo.com/.It can be seen that when it encounters blocking (such as IO operation, or gevent.sleep(0)), gevent's scheduler will automatically switch the coroutine to execute; and its coroutines are functions one by one, which is very nice Now: what looks like a thread is actually a coroutine.