Python process as much as multithreading

Process is a collection of resources

The thread is the process in which specific work

Between threads and threads are independent of each other

Multithreading: For IO intensive tasks

Multi-process: for CPU-intensive tasks

First, multithreading

Threads need to use threading module

The thread start method:

threading.Thread (target = XXX, args = ( 'xxx', 'xxx')) #target take the name of the function, args parameter is passed then, if there is only one parameter to be written so args = ( 'xxx', )

By threading.Thread examples are child threads out of the thread, only one thread First up is the main thread

Write a simple multi-threaded

import threading

def down_load():
    time.sleep(1)
    Print ( ' run over ' )

for i in the Range ( 5 ): # 5 cycles that started five threads
    T = of the threading.Thread (target = down_load) instantiating a thread #
    t.start () # start the thread

print (threading.active_count ()) # View the current line a number of processes
print (threading.current_thread ()) # View the current thread

If I now want to look at all the threads start executing the deals, which involves thread to wait, the thread is waiting for join, use jion would be more trouble, you can use a while loop manner to handle.

import threading
import time

def down_load():
    time.sleep(1)
    Print ( ' run over ' )

start_time = time.time ()
 for i in the Range ( 5 ): # 5 cycles that started five threads
    T = of the threading.Thread (target = down_load) instantiating a thread #
    t.start () # start the thread
the while threading.active_count () =! 1 : # 1 is not equal to the number of threads judge has been circulating, equal to 1 until the end of the cycle
    pass
print (threading.active_count ()) # View the current number of threads
print (threading.current_thread ()) # View the current thread

end_time=time.time()
Print ( ' Time: ' , END_TIME-START_TIME)

example:

Respectively with single and multithreaded execution are two ways to look at the time to download the pictures:

mport requests,time,threading
from hashlib import md5
def down_load_pic(url):
    req = requests.get(url)
    m = md5(url.encode())
    with open( m.hexdigest()+'.png','wb') as fw:
        fw.write(req.content)

url_list = ['http://www.nnzhp.cn/wp-content/uploads/2019/10/f410afea8b23fa401505a1449a41a133.png',
            'http://www.nnzhp.cn/wp-content/uploads/2019/11/481b5135e75c764b32b224c5650a8df5.png',
            'http://www.nnzhp.cn/wp-content/uploads/2019/11/b23755cdea210cfec903333c5cce6895.png',
            'http://www.nnzhp.cn/wp-content/uploads/2019/11/542824dde1dbd29ec61ad5ea867ef245.png']

# Serial download operation (single-threaded) 
# start_time = time.time () 
# for url in URL_LIST: 
#      down_load_pic (url) 
# end_time = time.time ()
#
# print(end_time-start_time)


# Parallel fashion (multi-threaded) 
start_time = time.time ()
 for url in URL_LIST:
    t = threading.Thread (target = down_load_pic, args = (url,))   # only one parameter to be written so (url,) 
    t.start ()
    
while threading.activeCount()!=1:
    pass

end_time=time.time()
print(end_time-start_time)

• Thread Pool

Need to use threadpool thread pool module, the need to manually install it: pip install threadpool

Thread pool can automatically calculate the distribution of data, we do not need to manually start, or to download the picture as an example:

import threadpool
import requests,time,threading
from hashlib import md5
def down_load_pic(url):
    req = requests.get(url)
    m = md5(url.encode())
    with open( m.hexdigest()+'.png','wb') as fw:
        fw.write(req.content)

url_list = ['http://www.nnzhp.cn/wp-content/uploads/2019/10/f410afea8b23fa401505a1449a41a133.png',
            'http://www.nnzhp.cn/wp-content/uploads/2019/11/481b5135e75c764b32b224c5650a8df5.png',
            'http://www.nnzhp.cn/wp-content/uploads/2019/11/b23755cdea210cfec903333c5cce6895.png',
            'http://www.nnzhp.cn/wp-content/uploads/2019/11/542824dde1dbd29ec61ad5ea867ef245.png']

the pool = threadpool.ThreadPool (20 is) # instantiating a thread pool 
reqs = threadpool.makeRequests (down_load_pic, URL_LIST) # allocation data, the first is a function name, the second data 
# [pool.putRequest (REQ) for REQ in reqs] # listing formula, with the following two lines of code 
for REQ in reqs:
    pool.putRequest(req)
    
pool.wait () # Wait until they have finished printing is executed end, if you do not will wait to print End 
Print ( ' End ' )

• Daemon thread

Daemon thread relies on the main thread, the main thread ends, a daemon thread will end immediately.

import threading,time

def down_load():
    the time.sleep ( . 1 )
     Print ( ' run over ' )

for i in range(5):
    t=threading.Thread(target=down_load)
    t.setDaemon (True) # Set the child thread as a daemon thread 
    t.start ()

print('over')

• Thread lock

Multiple threads operating data of the same time, we must lock,

import threading

a = 0
Lock = threading.Lock () # apply a lock

DEF the Add ():
     , Ltd. Free Join NUM
     # lock.acquire () # lock 
    # NUM + = 1 
    # lock.release () # unlock, do not unlock the lock if not there will be a deadlock 

    with Lock: # shorthand, also with with will help you automatically lock, unlock 
        NUM + = 1 

for i in the Range (10 ):
    t=threading.Thread(target=add)
    t.start()
while threading.activeCount()!=1:
    pass
print('over:',num)

Second, multi-process

Process uses multiprocessing module

import multiprocessing,time

def down_load():
    the time.sleep ( . 1 )
     Print ( ' run over ' )

IF  __name__ == ' __main__ ' : # under windows need to write multiple processes in main, otherwise it will error, mac does not need

    for i in range(5):
        the p- = multiprocessing.Process (target = down_load) # example of a process 
        p.start ()

    the while len (multiprocessing.active_children ()) =! 0:
         Pass 
    Print (multiprocessing.current_process ()) # View the current process 
    # Print (multiprocessing.cpu_count ()) # View the number of cpu 
    Print ( ' End ' )