Python Multiprocessing with Pool()

Question

I can't seem to get this to work, I'm trying to use multiprocessing to call my evaluate function for all combinations of a list;

import itertools
from multiprocessing import Pool

def combinations(inputs):
    pool = Pool()
    combi = list(itertools.combinations(inputs, 5))
    outputs = pool.map(evaluate, combi)
    return outputs

def evaluate(input):
    <do stuff>
    return output

a = [[2,2],[4,3],[5,1],[6,3],[7,2],[12,4],[12,2]]
b = combinations(a)

This works if I put it into a for loop and process each one at a time. It's just the pool.map() and how to pass each individual list item I can't figure out.

Alex Lopatin · Accepted Answer · 2019-03-09 13:19:51Z

Here is with the original list of 21 combinations:

import itertools
from multiprocessing import Pool
import os

def combinations(inputs):
    combi = list(itertools.combinations(inputs, 5))
    pool = Pool(len(combi))
    outputs = pool.map(evaluate, combi)
    return outputs


def evaluate(input):
    return ['processed by {0}'.format(os.getpid()), input]


a = [[2,2],[4,3],[5,1],[6,3],[7,2],[12,4],[12,2]]
b = combinations(a)
for i in b:
    print(i)

Output:

['processed by 5185', ([2, 2], [4, 3], [5, 1], [6, 3], [7, 2])]
['processed by 5186', ([2, 2], [4, 3], [5, 1], [6, 3], [12, 4])]
['processed by 5187', ([2, 2], [4, 3], [5, 1], [6, 3], [12, 2])]
['processed by 5188', ([2, 2], [4, 3], [5, 1], [7, 2], [12, 4])]
['processed by 5189', ([2, 2], [4, 3], [5, 1], [7, 2], [12, 2])]
['processed by 5190', ([2, 2], [4, 3], [5, 1], [12, 4], [12, 2])]
['processed by 5191', ([2, 2], [4, 3], [6, 3], [7, 2], [12, 4])]
['processed by 5192', ([2, 2], [4, 3], [6, 3], [7, 2], [12, 2])]
['processed by 5193', ([2, 2], [4, 3], [6, 3], [12, 4], [12, 2])]
['processed by 5194', ([2, 2], [4, 3], [7, 2], [12, 4], [12, 2])]
['processed by 5195', ([2, 2], [5, 1], [6, 3], [7, 2], [12, 4])]
['processed by 5196', ([2, 2], [5, 1], [6, 3], [7, 2], [12, 2])]
['processed by 5197', ([2, 2], [5, 1], [6, 3], [12, 4], [12, 2])]
['processed by 5198', ([2, 2], [5, 1], [7, 2], [12, 4], [12, 2])]
['processed by 5199', ([2, 2], [6, 3], [7, 2], [12, 4], [12, 2])]
['processed by 5200', ([4, 3], [5, 1], [6, 3], [7, 2], [12, 4])]
['processed by 5201', ([4, 3], [5, 1], [6, 3], [7, 2], [12, 2])]
['processed by 5202', ([4, 3], [5, 1], [6, 3], [12, 4], [12, 2])]
['processed by 5203', ([4, 3], [5, 1], [7, 2], [12, 4], [12, 2])]
['processed by 5204', ([4, 3], [6, 3], [7, 2], [12, 4], [12, 2])]
['processed by 5205', ([5, 1], [6, 3], [7, 2], [12, 4], [12, 2])]

razimbres · Accepted Answer · 2019-03-09 11:31:58Z

1

Run:

import multiprocessing

output=[]

a = [[2,2],[4,3],[5,1],[6,3],[7,2],[12,4],[12,2]]


def evaluate(x):
    output=x[0]+x[1]
    return output

def combinations():
    pool = multiprocessing.Pool(64)
    outputs=pool.map(evaluate, a)
    return outputs

if __name__ == '__main__':
    output.append(combinations())

output

Output:

#Out[99]: [[4, 7, 6, 9, 9, 16, 14]]

answered Mar 9, 2019 at 11:31

razimbres

5,0855 gold badges29 silver badges55 bronze badges

Collectives™ on Stack Overflow

Python Multiprocessing with Pool()

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related