EDITORIAL: multithreading in python is not really true multi-threaded, if you want to get the most from multicore CPU resources, in most cases require the use of multiple processes in python. Python provides a very easy to use multi-process packages multiprocessing, only need to define a function, Python will do all the other things. With this package, it can be easily done from a single process to execute concurrently conversion.
1. The multiprocessingmodule provides a Processclass to represent a process object
os Import Import Time DEF run_proc (name): Code # child process to execute Print ' run sub-process% s, the child process ID (% S) ... ' % (name, os.getpid ()) Print " I processing contents are:?% S% S + = " % (name, name) return name IF __name__ __ == ' __main__ ' : Print ' parent process number S%. ' % os.getpid () Print ( ' --- ------------------------------------- ' ) for I in Range ( . 3 ): PMultiprocessing.Process = (= run_proc target, args = (i,)) # multiple processes Print ' child process% d open ... ' % i p.start () #join()method can wait after the end of the child process continues to run down , commonly used for synchronization between processes p.join () Print ' sub-process ends% d ... ' % I Print
Results are as follows:
Parent process ID is 3378 . ---------------------------------------- child process open 0 ... run sub-process 0 , the child process number ( 3642 ) ... my processing content is: 0 + 0 ? = child process ends ... 0 child process open ... 1 run sub-process 1 , the child process No. ( 3649 ) ... my processing content is: 1 + 1 ? = child process 1 end ... child process open ... 2 run sub-process 2 , sub-process ID ( 3656 ) ... my processing content is: 2 + 2 =? child process 2 ends ...
If you want to return the results obtained by processing multi-process process, only you need to use the Manager class to multiprocessing, slightly change the code:
os Import Import Time from multiprocessing Import Manager DEF run_proc (name, return_list): Code # child process to execute Print ' run sub-process% s, the child process ID (% S) ... ' % (name, os.getpid ()) Print " I was handling content:% S% S + =? " % (name, name) return_list.append (name) IF __name__ __ == ' __main__ ' : Print ' parent process number S%. ' % os .getpid () Print ( ' ---------------------------------------- ' ) Manager = Manager () return_list = manager.list () #return_dict = manager.dict () you can also use the dictionary dict for i in the Range ( 3 ): the p- = multiprocessing.Process (= run_proc target, args = (i, return_list)) # multi-process print ' subprocess% d open ... ' % I p.start () p.join () Print ' sub-process ends% d ... ' % I Print Print " result of all the child process are obtained in return_list, value: " , return_list
Results are as follows:
Parent process ID is 3378 . ---------------------------------------- child process open 0 ... run sub-process 0 , the child process number ( 4569 ) ... my processing content is: 0 + 0 ? = child process ends ... 0 child process open ... 1 run sub-process 1 , the child process No. ( 4579 ) ... my processing content is: 1 + 1 ? = child process 1 end ... child process open ... 2 run sub-process 2 , sub-process ID ( 4589 ) ... my processing content is: 2 + 2 =? child process ... 2 end result of all child processes have been treated in return_list, the values are: [ 0 , 1 ,2]
2.Pool: If you want to start a large number of child processes can be used to process pool way to batch create child process:
from multiprocessing Pool Import Import os, Time, Random DEF long_time_task (name): Print ' running task% s, the child process ID (% S) ... ' % (name, os.getpid ()) Print " I am the son No. process (% s) processed content " % (os.getpid ()) Start = the time.time () the time.sleep (random.random () * . 3 ) End = the time.time () Print ' tasks% s % 0.2f second run. ' % (name, (End - Start)) return name IF __name__ __ == ' __main__' : Print ' parent process number S%. ' % Os.getpid () RST = [] P = Pool ( 4 ) # 4 sub-process process pool contains for I in Range ( 5 ): 4 sub-process is completed # 5 tasks, so there is a task that requires a process such as idle reprocessing a = p.apply_async (long_time_task, args = (i,)) #A is the process handler long_time_task results returned rst.append (a) # the times the result is added to the array rst go Print ' wait for all child processes end ... ' p.close () p.join () # wait for all child process is finished. After the first call must close before calling join () (), call the close () can not continue to add a new Process. Print ' all child processes end ... '
Results are as follows:
Parent process ID is 3378 . Run the task 0 , the child process number ( 4621 ) ... run the task 2 , the child process number ( 4624 ) ... run the task 1 , the child process number ( 4622 ) ... and I'm child process ID ( 4621 ) deal with the contents of my son is the process ID ( 4622 ) deal with content running task 3 , the child process number ( 4627 ) ... I am the child process ID ( 4624 ) deal with content and I'm child process ID ( 4627 contents) processing task 1 running 0.16 seconds to run the task 4 , the child process number ( 4622 ) ... I am the child process ID ( 4622) Content processing of waiting for all child processes end ... Task 2 ran 0.98 seconds. Task 4 ran 0.89 seconds. Task 3 ran 2.25 seconds task 0 ran 2.89 seconds to end all child processes ...
Direct output rst will not get the desired results:
rst 运行结果： [<multiprocessing.pool.ApplyResult at 0x7ffa6c682c90>, <multiprocessing.pool.ApplyResult at 0x7ffa6c587590>, <multiprocessing.pool.ApplyResult at 0x7ffa6c587610>, <multiprocessing.pool.ApplyResult at 0x7ffa6c5876d0>, <multiprocessing.pool.ApplyResult at 0x7ffa6c587790>]
This is the need to use .get () method:
= RST [. I GET () for I in RST] RST result: [ 0 , 1 , 2 , 3 , 4 ]