我们知道python下的多进程做异步还是可以的，但是做并发利用多核处理器是行不通的，而且速度还会更慢。那么我们来试试多进程的效果吧。简单看下多进程的几种实现方法。
1. 普通进程启动与测试
#!/usr/bin/env python
####################################
# Process is used execute a standalone task, which is different with thread
# common methods:
# start() used to start process
# join() used to synchronous process to wait all process finish
# close() used to prevent more process to add in process pool
####################################
import multiprocessing
import os
def process_task(name):
print('Children process {0}({1}) is running'.format(name, os.getpid()))
if __name__ == '__main__':
print('Parent process {0} is running'.format(os.getpid()))
for i in range(10):
p = multiprocessing.Process(target=process_task, args=("task%d" % i,))
p.start()
p.join()
print('All processes have been finished')
采用4和机器，我们发现进程被充分利用了。结果如下：
Parent process 21952 is running
Children process task1(24440) is running
Children process task2(20548) is running
Children process task0(20744) is running
Children process task3(16556) is running
Children process task5(24364) is running
Children process task4(17044) is running
Children process task6(17212) is running
Children process task9(23964) is running
Children process task7(8460) is running
Children process task8(21452) is running
All processes have been finished
2. 使用Task Pool
#!/usr/bin/env python
####################################
# Process is used execute a standalone task, which is different with thread
# we use poll
# common methods:
# start() used to start process
# join() used to synchronous process to wait all process finish
# close() used to prevent more process to add in process pool
#
####################################
from multiprocessing import Pool
import os, time, random
# process to work
def process_task(i, task_name):
print('Runing task %s (%s)...' % (task_name, os.getpid()))
start = time.time()
time.sleep(i)
end = time.time()
print('Task %s has been runing %0.2f seconds.' % (task_name, (end - start)))
def async_pool():
print('Parent process %s.' % os.getpid())
p = Pool()
# start multiprocess with pool
for i in range(10):
p.apply_async(process_task, args=(i, "task"+ str(i)))
print('Waiting for all subprocesses...')
p.close()
p.join()
print('All have been done!')
def sync_pool():
print('Parent process %s.' % os.getpid())
p = Pool()
# start multiprocess with pool
for i in range(10):
p.apply(process_task, args=(i, "task"+ str(i)))
print('Waiting for all subprocesses...')
p.close()
p.join()
print('All have been done!')
if __name__ == '__main__':
# 1. Testing async pool
print('Testing async_pool...')
async_pool()
# 2. Testing sync pool
print('Tesint sync_pool...')
sync_pool()
运行结果如下
Testing async_pool...
Parent process 23348.
Waiting for all subprocesses...
Runing task task0 (10788)...
Task task0 has been runing 0.00 seconds.
Runing task task1 (10788)...
Runing task task2 (18224)...
Runing task task3 (5212)...
Runing task task4 (24308)...
Task task1 has been runing 1.00 seconds.
Runing task task5 (10788)...
Task task2 has been runing 2.00 seconds.
Runing task task6 (18224)...
Task task3 has been runing 3.00 seconds.
Runing task task7 (5212)...
Task task4 has been runing 4.00 seconds.
Runing task task8 (24308)...
Task task5 has been runing 5.00 seconds.
Runing task task9 (10788)...
Task task6 has been runing 6.00 seconds.
Task task7 has been runing 7.00 seconds.
Task task8 has been runing 8.00 seconds.
Task task9 has been runing 9.00 seconds.
All have been done!
Tesint sync_pool...
Parent process 23348.
Runing task task0 (5440)...
Task task0 has been runing 0.00 seconds.
Runing task task1 (5440)...
Task task1 has been runing 1.00 seconds.
Runing task task2 (15116)...
Task task2 has been runing 2.00 seconds.
Runing task task3 (12916)...
Task task3 has been runing 3.00 seconds.
Runing task task4 (10264)...
Task task4 has been runing 4.00 seconds.
Runing task task5 (5440)...
Task task5 has been runing 5.00 seconds.
Runing task task6 (15116)...
Task task6 has been runing 6.00 seconds.
Runing task task7 (12916)...
Task task7 has been runing 7.00 seconds.
Runing task task8 (10264)...
Task task8 has been runing 8.00 seconds.
Runing task task9 (5440)...
Task task9 has been runing 9.00 seconds.
Waiting for all subprocesses...
All have been done!
3. 使用queue进行进程间通信
#!/usr/bin/env python
####################################
# Process is used execute a standalone task, which is different with thread
# common methods:
# start() used to start process
# - join() used to synchronous process to wait all process finish
# - close() used to prevent more process to add in process pool
# Queue has put() and get() method.
# - put() used to add member to queue
# - get() used to fetch member from queue
# - blocked = True to wait add and fetch until time out
####################################
from multiprocessing import Process, Queue, Pool
import multiprocessing, os, time, random
# Write data to queue
def process_task_write(q, urls):
print('Process (%s) is writing..' % os.getpid())
for url in urls:
q.put(url)
print('Put %s to queue' % url)
time.sleep(random.random())
# Read data from queue
def process_task_read(q):
print('Process (%s) is reading...' % os.getpid())
while True:
url = q.get(True)
print('Get %s from queue' % url)
# using queue to communicate between parent and children, or children and children for sync process
def queue_subprocess():
# define process
q = Queue()
task_writer1 = Process(target=process_task_write, args=(q, ['url_1', 'url_2', 'url_3', 'url_4']))
task_writer2 = Process(target=process_task_write, args=(q, ['url_5', 'url_6', 'url_7', 'url_8']))
task_reader = Process(target=process_task_read, args=(q,))
# start processes
task_reader.start()
task_writer1.start()
task_writer2.start()
# wait write finish
task_writer1.join()
task_writer2.join()
# wait a moment to ternminate reader
time.sleep(1)
task_reader.terminate()
print('All done!')
# Write data to queue
def process_task_asyn_write(q, urls):
print('Process (%s) is writing..' % os.getpid())
for url in urls:
q.put(url)
print('Put %s to queue' % url)
time.sleep(random.random())
# Read data from queue
def process_task_async_read(q, lock):
print('Process (%s) is reading...' % os.getpid())
while True:
lock.acquire()
url = q.get(True)
lock.release()
print('Get %s from queue' % url)
# using manager queue to communicate between parent and children, or children and children for async process
def manager_queue_subprocess():
manager = multiprocessing.Manager()
q = manager.Queue()
lock = manager.Lock()
p = Pool()
p.apply_async(process_task_asyn_write, args=(q, ['manager_url_1', 'manager_url_2', 'manager_url_3', 'manager_url_4']))
p.apply_async(process_task_asyn_write, args=(q, ['manager_url_5', 'manager_url_6', 'manager_url_7', 'manager_url_8']))
task_reader = p.apply_async(process_task_async_read, args=(q, lock))
p.close()
p.join()
print('All done!')
if __name__ == '__main__':
# 1. using queue for sync process
print('Testing with queue...')
queue_subprocess()
# 2. using manager queue for async process, because queue can't be shared between pararent between children in async process
print('Testing with manager queue...')
manager_queue_subprocess()
运行结果如下
Testing with queue...
Process (16272) is writing..
Process (17656) is reading...
Put url_1 to queue
Get url_1 from queue
Process (19280) is writing..
Put url_5 to queue
Get url_5 from queue
Put url_2 to queue
Get url_2 from queue
Put url_6 to queue
Get url_6 from queue
Put url_3 to queue
Get url_3 from queue
Put url_4 to queue
Get url_4 from queue
Put url_7 to queue
Get url_7 from queue
Put url_8 to queue
Get url_8 from queue
All done!
Testing with manager queue...
Process (16736) is writing..
Process (9184) is writing..
Put manager_url_1 to queue
Put manager_url_5 to queue
Process (14340) is reading...
Get manager_url_1 from queue
Get manager_url_5 from queue
Put manager_url_6 to queue
Get manager_url_6 from queue
Put manager_url_7 to queue
Get manager_url_7 from queue
Put manager_url_8 to queue
Get manager_url_8 from queue
Put manager_url_2 to queue
Get manager_url_2 from queue
Put manager_url_3 to queue
Get manager_url_3 from queue
Put manager_url_4 to queue
Get manager_url_4 from queue
4. 使用Pipe进行进程间通信
#!/usr/bin/env python
from multiprocessing import Process, Pipe
# single way to send
def single_proc(pipe):
pipe.send("data from process pipe")
pipe.close()
# single way to send in process and receive from main process
def single_way():
(pipe1, pipe2) = Pipe()
sender = Process(target = single_proc, args = (pipe1, ))
sender.start()
print("we get from pipe: %s" % pipe2.recv())
pipe2.close()
# round way to send and receive
def round_proc(pipe):
pipe.send('data from process pipe')
reply = pipe.recv()
print('child got: ', reply)
# round way main process
def round_way():
(pipe1, pipe2) = Pipe()
child = Process(target=round_proc, args=(pipe1,))
child.start()
print('parent got:', pipe2.recv())
pipe2.send({x * 2 for x in 'test'})
child.join()
print('parent exit!')
if __name__ == '__main__':
# 1. single way sample
single_way()
# 2. round way sample
round_way()
运行结果如下：
we get from pipe: data from process pipe
parent got: data from process pipe
child got: {'ee', 'ss', 'tt'}
parent exit!