The performance of the relevant reptiles

1.1 common methods of concurrency

　　1 Introduction

　　　　　　1. When writing reptiles, performance is mainly consumed in the IO request, when the request URL using single-threaded mode will inevitably lead to wait, so that the overall slow request.

　　　　　　2. Process: Enable process is a waste of resources

　　　　　　3. Thread: multi-thread, and can not perform other tasks in the process of blocking

　　　　　　4. Coroutine: gevent only from a thread, when the request is sent out after gevent on the matter, it is always only one thread work, the first to come back to deal with

　　2, concurrency compare several methods

　　　　1) Use the thread pool concurrency

# ! / Usr / bin / Python the env 
# - * - Coding: UTF-. 8 - * - 
Import Requests
 from concurrent.futures Import the ThreadPoolExecutor 

DEF fetch_request (URL): 
    Result = requests.get (URL)
     Print (result.content) 

the pool ThreadPoolExecutor = (10)        # create a thread pool, open up to 10 threads 
URL_LIST = [
     ' www.google.com ' ,
     ' http://www.baidu.com ' , 
] 

for url in URL_LIST:
     # go to the thread pool Gets a thread 
    #Fetch_request thread to perform the method 
    pool.submit (fetch_request, url) 

pool.shutdown (True)      # main thread shuts itself down, let the child thread up their own task execution

Use thread pool concurrency

　　　　2) the use of process pool concurrency

#! /usr/bin/env python
# -*- coding: utf-8 -*-
import requests
from concurrent.futures import ProcessPoolExecutor

def fetch_request(url):
    result = requests.get(url)
    print(result.text)

url_list = [
    'www.google.com',
    'http://www.bing.com',
]

if __name__ == '__main__':
    pool = ProcessPoolExecutor(10)  # 线程池
    #Disadvantages: multi-thread, and can not perform other tasks in the process of blocking 
    for url in URL_LIST:
         # to get a thread pool process 
        # process to execute fetch_request method 
        pool.submit (fetch_request, url) 
    pool.shutdown (True)

The use of process pool concurrency

　　　　3) the callback function to perform multi-threaded +

#! /usr/bin/env python
# -*- coding: utf-8 -*-
from concurrent.futures import ThreadPoolExecutor
import requests

def fetch_async(url):
    response = requests.get(url)
    return response

def callback(future):
    print(future.result().content)

if __name__ == '__main__':
    url_list = ['http://www.github.com', 'http://www.bing.com']
    pool = ThreadPoolExecutor(5)
    for url in url_list:
        v = pool.submit(fetch_async, url)
        v.add_done_callback(callback)
    pool.shutdown(wait=True)

Callback function to perform multi-threaded +

　　　　4) coroutine: Asynchronous microthread

# ! / Usr / bin / env Python 
# - * - Coding: UTF-8 - * - 
Import gevent
 Import Requests
 from gevent Import Monkey 

monkey.patch_all () 

# these requests come back to deal with whoever who 
DEF fetch_async (Method,, url , req_kwargs):
     Print (Method, URL, req_kwargs) 
    Response = requests.request (Method = Method, URL = URL, ** req_kwargs)
     Print (response.url, response.content) 


IF  the __name__ == ' __main__ ' :
     # # ### transmission request ##### 
     gevent.joinall ([
        gevent.spawn (fetch_async, Method='get', url='https://www.python.org/', req_kwargs={}),
        gevent.spawn(fetch_async, method='get', url='https://www.yahoo.com/', req_kwargs={}),
        gevent.spawn(fetch_async, method='get', url='https://github.com/', req_kwargs={}),
    ])

Coroutine: micro-threaded asynchronous