python --- coroutine programming (use of third-party library gevent)

<div class="iteye-blog-content-contain" style="font-size: 14px"><div id="cnblogs_post_body">
 < ;span style="font-size: 16px">1. What is a coroutine? 
 Coroutine, also known as microthread. A coroutine is not a thread or a process, and its context switching is not controlled by the CPU. A coroutine is switched from the current task to other tasks controlled by the current task. A thread can contain multiple coroutines. For the CPU, there is no concept of coroutines. It is a lightweight user-mode thread (that is, only for users). The coroutine has its own register context and stack. When the coroutine scheduling switches to other coroutines, the register context and stack are saved. When switching back to the current coroutine, the previously saved register context and stack are restored. 
 2. Why use coroutines in programming? 
 The advantages of using coroutines: (1) The CPU does not need to bear the overhead of the context; (2) There is no need to lock (multiple threads) (3) The programmer switches the control flow to facilitate programming; (4) High concurrency, high expansion, low Cost (a CPU supports tens of thousands of coroutines is not a problem). 
   There are downsides. Disadvantages of coroutines: (1) The coroutine itself cannot use the CPU multi-core resources (unless it cooperates with multi-process or multi-threading); (2) encountering blocking operations will block the entire program. 
   
 < ;span style="font-size: 16px">Example 1 (using yield to switch between tasks): 

 <img id="code_img_closed_868047cc-5571-4a37-9978-938f91cb06d2" class="code_img_closed" src="http://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt="" />
 <img id="code_img_opened_868047cc-5571-4a37-9978-938f91cb06d2" class="code_img_opened" style="display: none" src="http://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="" /> 
 <div id="cnblogs_code_open_868047cc-5571-4a37-9978-938f91cb06d2" class="cnblogs_code_hide"> 
 <pre> 1 import time
 2 
 3 def func1(name):
 4 print("----func1 start...----")
 5 for i in range(6):
 6 temp = yield #每次遇到yield，func1在此处阻塞，直到temp接收到func2中con.send()传来的值
 7 print("%s in the func1" % (str(temp)))
 8 time.sleep(1)
 9 
10 
11 def func2():
12 print("----func2 start...----")
13 con.__next__() #The real func1 call starts here
14 for i in range(5):
15 con.send(i+1)
16 print("%s in the func2" % i)
17 
18 
19 if __name__ == '__main__':
20 con = func1(1) # This is not a real function call in a function with yield, print con to know

How should coroutines be used in ? 
 greenlet is a coroutine module implemented in C, Compared with the yield that comes with PYTHON , it allows you to switch between arbitrary functions without declaring this function as generator (con=func1(1) in example 1) just do this). 
   
 < ;span style="font-size: 16px">Example 2: 
 <div class="cnblogs_code">
 <img id="
 <img id="code_img_opened_f0929bd7-dab0-49a3-8a44-d97faf62506a" class="code_img_opened" style="display: none" src="http://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="" /> 
 <div id="cnblogs_code_open_f0929bd7-dab0-49a3-8a44-d97faf62506a" class="cnblogs_code_hide"> 
 <pre> 1 import greenlet
 2 
 3 def func1():
 4 for i in range(1,6):
 5 print(i)
 6 g2.switch() #切换到g2
 7 
 8 def func2():
 9 words = ['a', 'b', 'c', 'd', 'e']color: #800000">e']color: #800000">e']
10 for w in words:
11 print(w)
12 g1.switch() #切换到g1
13 
14 g1 = greenlet.greenlet(func1)
15 g2 = greenlet.greenlet(func2)
16 g1.switch() #切换到g1</pre> 
 </div> 
 Use greenlent module for task switching
 </div> 
 Note: Using greenlent can easily switch between multitasking, but the most time-consuming program running is I/ O operation, to use coroutines to achieve high concurrency, it should be switch to other tasks once an I/O operation is encountered, and switch back to the current task after the I/O operation is completed (this process should be automatic ). 
   
 4. So in PYTHON , how to make tasks switch when encountering I/O operations? 
 We use the third-party library gevent for this. 
 　　gevent的官方定义：gevent is a <a class="reference external" href="https://en.wikipedia.org/wiki/Coroutine">coroutine</a> -based <a class="reference external" href="http:// PYTHON .org/"> PYTHON </a> networking library that uses <a class="reference external" href="https://greenlet.readthedocs.io/">greenlet</a> to provide a high-level synchronous API on top of the libev event loop. 
   
 例三（gevent的简单使用）： 
 <div class="cnblogs_code">
 <img id="code_img_closed_26c74430-ff34-4e2e-af26-e8a3c235b1f0" class="code_img_closed" src="http://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt="" />
 <img id="code_img_opened_26c74430-ff34-4e2e-af26-e8a3c235b1f0" class="code_img_opened" style="display: none" src="http://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="" /> 
 <div id="cnblogs_code_open_26c74430-ff34-4e2e-af26-e8a3c235b1f0" class="cnblogs_code_hide"> 
 <pre> 1 import gevent
 2 import time
 3 
 4 def func1():
 5 print("func1: start.....")
 6 # Put the current greenlet to sleep for at least *seconds*.(模拟I/O操作，任务在此处自动切换)
 7 gevent.sleep(3)
 8 print("func1: end")
 9 
10 def func2():
11 print("func2: start.....")
12 gevent.sleep(0.5)
13 print("func2: end")
14 
15 start_time = time.time()
16 # joinall(greenlets, timeout=None, raise_error=False, count=None)
17 # Wait for the ``greenlets`` to finish.
18 # :return: A sequence of the greenlets that finished before the timeout (if any)expired.
19 gevent.joinall([gevent.spawn(func1),
20 gevent.spawn(func2)])
21 # spawn(cls, *args, **kwargs)
22 # Create a new :class:`Greenlet` object and schedule it to run ``function(*args, **kwargs)``.
23 # This can be used as ``gevent.spawn`` or ``Greenlet.spawn``.
24 
25 print("cost:", time.time()-start_time)
26 # 通过计算程序运行的时间可以发现程序确实是以单线程达模拟出了多任务并行的操作。</pre> 
 </div> 
 gevent的简单使用
 </div> 
   
 例四（gevent和urllib配合同时下载多个网页）： 
 <div class="cnblogs_code">
 <img id="code_img_closed_04c7985d-8718-4d81-b37a-847a2a20e510" class="code_img_closed" src="http://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt="" />
 <img id="code_img_opened_04c7985d-8718-4d81-b37a-847a2a20e510" class="code_img_opened" style="display: none" src="http://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="" /> 
 <div id="cnblogs_code_open_04c7985d-8718-4d81-b37a-847a2a20e510" class="cnblogs_code_hide"> 
 <pre> 1 import urllib.request
 2 import gevent,time
 3 import gevent.monkey
 4 
 5 def func(url="", filename=""):
 6 print("Download:%s" % url)
 7 result = urllib.request.urlopen(url) #请求打开一个网页
 8 data = result.read() #读取内容
 9 with open(filename, 'wb') as fp: #写入文档
10 fp.write(data)
11 print("Finish:%s" % url)
12 
13 if __name__ == "__main__":
14 # Do all of the default monkey patching (calls every other applicablefunction in this module).
15 # 相当与做一个标记，做完此操作gevent就可以检测到此程序中所有的I/O操作
16 gevent.monkey.patch_all()
17 
18 async_time = time.time()
19 gevent.joinall([
20 gevent.spawn(func, "http://www.cnblogs.com/ [url=http://www.cppentry.com/list.php?fid=77]PYTHON [/url]d-Li/p/7774497.html", "7774497.html"),
21 gevent.spawn(func, "http://www.gevent.org/", "gevent.html"),
22 gevent.spawn(func, "https://www. PYTHON .org/", " PYTHON .html"),
23 ])
24 print("async download cost:", time.time()-async_time)
25 
26 start_time = time.time()
27 func("http://www.cnblogs.com/ [url=http://www.cppentry.com/list.php?fid=77]PYTHON [/url]d-Li/p/7774497.html", "7774497.html")
28 func("http://www.gevent.org/", "gevent.html")
29 func("https://www. PYTHON .org/", " PYTHON .html")
30 print("download cost:", time.time()-start_time)</pre> 
 </div> 
 gevent and urllib cooperate to download multiple pages at the same time
 </div> 
 Note: A simple multi-concurrency crawler can be implemented by slightly modifying the code of the above example and adding the parsing function of the html source code. 
   
 Yes<a id="cb_post_title_url" class="postTitle2" href="http://www.cnblogs.com/ [url=http://www.cppentry.com/list. php?fid=77]PYTHON [/url]d-Li/p/7598850.html"> PYTHON --- Network Programming Socket</a> The socket_server2 in the second example can be transformed into a large concurrent socket server by using gevent.
 Example 5 (using gevent to implement concurrent socket server) : 
 <div class="cnblogs_code">
 <img id="code_img_closed_45c1bc41-9757-42bd-8796-a6b582a5ef7c" class="code_img_closed" src="http://images.cnblogs.com/OutliningIndicators/ContractedBlock.gif" alt="" />
 <img id="code_img_opened_45c1bc41-9757-42bd-8796-a6b582a5ef7c" class="code_img_opened" style="display: none" src="http://images.cnblogs.com/OutliningIndicators/ExpandedBlockStart.gif" alt="" /> 
 <div id="cnblogs_code_open_45c1bc41-9757-42bd-8796-a6b582a5ef7c" class="cnblogs_code_hide"> 
 <pre> 1 #服务端
 2 import socket
 3 import gevent
 4 import gevent.monkey
 5 
 6 gevent.monkey.patch_all()
 7 
 8 def request_handler(conn):
 9 
10 '''
11 Wait for an incoming connection. Return a new socket
12 representing the connection, and the address of the client.
13 '''
14 while True:
15 # print("ok")
16 data = conn.recv(1024) #接收信息，写明要接收信息的最大容量，单位为字节
17 print("server recv:", data)
18 conn.send(data.upper()) #对收到的信息处理，返回到客户端
19 
20 
21 
22 if __name__ == "__main__":
23 address = ("localhost", 6666) # 写明服务端要监听的地址，和端口号
24 server = socket.socket() # 生成一个socket对象
25 server.bind(address) # 用socket对象绑定要监听的地址和端口
26 server.listen() # 开始监听
27 
28 while True:
29 conn, addr = server.accept() # 等带新连接接入服务端，返回一个新的socket对象和地址，地址格式同前面格式
30 gevent.spawn(request_handler, conn)
31 
32 server.close() # 关闭服务端</pre> 
 </div> 
 socket_server2的并发实现
 </div> 
  　　注：可使用<a id="cb_post_title_url" class="postTitle2" href="http://www.cnblogs.com/ </div></div>--- Network Programming Socket</a> ;The socket_client2 of the second example is tested. PYTHON[/url]d-Li/p/7598850.html"> [url=http://www.cppentry.com/list.php?fid=77]PYTHON

python --- coroutine programming (use of third-party library gevent)

Guess you like