Multiprocessing in a loop, "Pool not running" error

Anna :

I'm trying to run some calculation in loop, each calculation creates, uses and closes a pool. But the calculation only runs once and then throws an error: "Pool not running". Of course the old one is not running, but shouldn't the new one be created?

Below is a simplified example, similar to my code. More freakishly, in my actual code calculation runs 7 times before crashing, so I'm really confused what's the problem. Any advice appreciated!

from pathos.multiprocessing import ProcessingPool as Pool

def add_two(number):  
    return (number + 2)

def parallel_function(numbers):
    pool = Pool(10)
    result = pool.imap(add_two, numbers)
    pool.close()
    pool.join()    
    return(result)

sets=[
    [1, 2, 3],
    [2, 3, 4],
    [3, 4, 5]
]

for one_set in sets:
    x = parallel_function(one_set)
    for i in x:
        print(i)

noxdafox :

This is a pathos limitation which implements the Pool using the singleton pattern.

This is the related issue ticket.

I would recommend you to use another Pool of Workers implementation.