Recently, I've been experimenting with the multiprocessing module. I wrote this script to test it:
import multiprocessing
from time import sleep
import datetime
def b(m):
print(m)
def int_val(a):
b(a)
def char_val(a):
sleep(15)
b(a)
list_val = [1,2,'c',6,10,1,'e',11,78,'a', 'b']
if __name__ == '__main__':
p = multiprocessing.Pool(4)
for index, val in enumerate(list_val):
if isinstance(val, str):
p.map(char_val, [val])
print(datetime.datetime.now())
else:
p.map(int_val, [val])
print(datetime.datetime.now())
The output looks like this :
1
2020-03-29 01:45:37.099114
2
2020-03-29 01:45:37.099114
c
2020-03-29 01:45:52.114733
6
2020-03-29 01:45:52.114733
10
2020-03-29 01:45:52.114733
1
2020-03-29 01:45:52.114733
e
2020-03-29 01:46:07.115963
11
2020-03-29 01:46:07.115963
78
2020-03-29 01:46:07.115963
a
2020-03-29 01:46:22.117232
b
2020-03-29 01:46:37.118046
If we see the output, the program has not really achieved multi-processing I was looking for.
I was hoping that while the program waits for the 15 seconds for the element 'c', other elements will be processed and almost all the characters will be printed at the same time.
It might be a silly question, but I am really lost!
Any help would be highly appreciated!
You're calling p.map
iteratively for each element of the list, so essentially you're spawning a new process for each element. Since p.map
is a blocking call, it is actually waiting for execution of char_val('c')
before moving ahead.
If you give the whole list_val
to p.map()
, you should get the expected execution sequence
import multiprocessing
from time import sleep
import datetime
def b(m):
print(m)
def int_val(a):
b(a)
def char_val(a):
sleep(15)
b(a)
def f(val):
if isinstance(val, str):
char_val(val)
print(datetime.datetime.now())
else:
int_val(val)
print(datetime.datetime.now())
list_val = [1,2,'c',6,10,1,'e',11,78,'a', 'b']
if __name__ == '__main__':
p = multiprocessing.Pool(4)
p.map(f, list_val)
# for index, val in enumerate(list_val):
# if isinstance(val, str):
# p.map(char_val, [val])
# print(datetime.datetime.now())
# else:
# p.map(int_val, [val])
# print(datetime.datetime.now())
Output:
1
2020-03-29 11:38:15.373607
2
2020-03-29 11:38:15.373764
6
2020-03-29 11:38:15.374008
10
1
2020-03-29 11:38:15.374117
2020-03-29 11:38:15.374108
11
2020-03-29 11:38:15.374233
78
2020-03-29 11:38:15.374438
c
a
2020-03-29 11:38:30.388652
2020-03-29 11:38:30.388761
e
b
2020-03-29 11:38:30.389465
2020-03-29 11:38:30.389566