python multiple processing using pool

Question

I have a assignment for my course in Python.

The question is this :

You will write a program creating multiple processes (p=10). Each process will receive integer value (y) and compute y multiplying by 4 (y*4). A collection of y values is a list of integers [0, 1, 2, 3, 4, 5]. You must use a process pool, so you need to map these 10 processes to take the list of y integers. Do the following:

Your program should print out the following three. (1) Print out the original list. (2) Print each running process name and its output of y*4 in random order of execution

Output of (1): Input list: [0, 1, 2, 3, 4, 5]

Output of (2): Output in random order:

PoolWorker-10 output: [0, 4, 8, 12, 16, 20]

PoolWorker-11 output: [0, 4, 8, 12, 16, 20]

PoolWorker-12 output: [0, 4, 8, 12, 16, 20]

My first attempt was:

import time
from multiprocessing import Pool
from multiprocessing import Process, current_process
def f(number):
    result=4*number
    Process_name=current_process().name
    print(f"Poolworker -{Process_name} output: {result} ")

    return result

if __name__=='__main__':
    p= Pool(processes=10)
    numbers=[0,1,2,3,4,5]
    result=p.map(f,numbers)

The output was:

Poolworker -SpawnPoolWorker-2 output: 0 
Poolworker -SpawnPoolWorker-2 output: 4 
Poolworker -SpawnPoolWorker-2 output: 8 
Poolworker -SpawnPoolWorker-2 output: 12 
Poolworker -SpawnPoolWorker-2 output: 16 
Poolworker -SpawnPoolWorker-2 output: 20

My second attempt:

import time
from multiprocessing import Pool
from multiprocessing import Process, current_process
def f(*number):
	numbers=list(number)
	i=0
	for x in numbers:
		numbers[i]=4*numbers[i]
		i+=1
	Process_name=current_process().name
	print(f"Poolworker -{Process_name}output: {numbers} ")

	return numbers

if __name__ == '__main__':
		array=[0,1,2,3,4,5]
		p=Pool(processes=10)
	
		result=p.map(f,array)

I still have the wrong output with the changes.

How can I get the desired output?

You might find this useful. stackoverflow.com/questions/48162230/… — johnashu
– johnashu, Commented Apr 21, 2020 at 10:56
i was really having hard time with this so maybe i was ranting a lot sorry . and also im not native at english. having hard time to explain in english :/ — sushisonjderek
– sushisonjderek, Commented Apr 21, 2020 at 10:57
@ johnashu thx haha. i think i was inapproriate asking question like that . just too much stress from my hw — sushisonjderek
– sushisonjderek, Commented Apr 21, 2020 at 11:07
@sushisonjderekDon't worry about it.. We have all been there.. Good luck in your studies! — johnashu
– johnashu, Commented Apr 21, 2020 at 11:23

gold_cy · Accepted Answer · 2020-04-21 11:37:49Z

2

The issue is that f is applied to each number in your list whereas you want every worker to do the calculations on the whole list. Made some minor modifications to your code in order to get those results.

from itertools import repeat
from multiprocessing import Pool
from multiprocessing import Process, current_process

def f(numbers):
    result= [4 * num for num in numbers]
    name=current_process().name
    print(f"Poolworker - {name} output: {result} ")
    return result

if __name__=='__main__':
    WORKERS = 10
    p = Pool(processes=WORKERS)
    numbers = [0, 1, 2, 3, 4, 5]
    print(f"Original - {numbers}")
    result = p.map(f, repeat(numbers, WORKERS))

This returns the following:

Original - [0, 1, 2, 3, 4, 5]
Poolworker - ForkPoolWorker-1 output: [0, 4, 8, 12, 16, 20] 
Poolworker - ForkPoolWorker-2 output: [0, 4, 8, 12, 16, 20] 
Poolworker - ForkPoolWorker-3 output: [0, 4, 8, 12, 16, 20] 
Poolworker - ForkPoolWorker-4 output: [0, 4, 8, 12, 16, 20] 
Poolworker - ForkPoolWorker-5 output: [0, 4, 8, 12, 16, 20] 
Poolworker - ForkPoolWorker-6 output: [0, 4, 8, 12, 16, 20] 
Poolworker - ForkPoolWorker-7 output: [0, 4, 8, 12, 16, 20] 
Poolworker - ForkPoolWorker-8 output: [0, 4, 8, 12, 16, 20] 
Poolworker - ForkPoolWorker-9 output: [0, 4, 8, 12, 16, 20] 
Poolworker - ForkPoolWorker-10 output: [0, 4, 8, 12, 16, 20]

edited Apr 21, 2020 at 11:37

answered Apr 21, 2020 at 11:32

gold_cy

14.2k4 gold badges27 silver badges55 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

sushisonjderek Over a year ago

i'd really appreciate ! but there's a few thing i couldn't understand .[4 * num for num in numbers] does this mean multiplying whole array elements? (idk much about python.) and also, result = p.map(f, repeat(numbers, WORKERS)) in this pool map , it has 'repeat' . does it mean repeating 10times ?

gold_cy Over a year ago

[4 * num for num in numbers] is known as a list-comprehension so yes it is multiplying every number in the list by 4. repeat is a function in the standard library which takes an iterable and repeats it n times where the user provides n. in our case we want it to repeat the number of times we have a process so we use that number as the argument

sushisonjderek Over a year ago

thanks a lot. i think i could understand ur answer easily .

DarrylG · Accepted Answer · 2020-04-21 11:32:48Z

From your description you want all processes to multiply array by 4.

So use parallel processes that are each given an array of numbers to multiply.

Reference

Code

from multiprocessing import Process

import multiprocessing as mp

def f(numbers, output):
    " Places process name and numbers multiplied by 4 into output queue "
    output.put((mp.current_process().name, [4*n for n in numbers]))

# Define an output queue
output = mp.Queue()

# Setup a list of processes that we want to run
numbers = [1, 2, 3, 4, 5]
processes = [mp.Process(target=f, args=([1, 2, 3, 4, 5], output)) for x in range(10)]

# Run processes
for p in processes:
    p.start()

# Exit the completed processes
for p in processes:
    p.join()

# Get process results from the output queue
results = [output.get() for p in processes]


for id, values in results:
  print(f'PoolWorker {id} output: {values}')

Output

PoolWorker Process-1 output: [4, 8, 12, 16, 20]
PoolWorker Process-3 output: [4, 8, 12, 16, 20]
PoolWorker Process-9 output: [4, 8, 12, 16, 20]
PoolWorker Process-6 output: [4, 8, 12, 16, 20]
PoolWorker Process-7 output: [4, 8, 12, 16, 20]
PoolWorker Process-5 output: [4, 8, 12, 16, 20]
PoolWorker Process-4 output: [4, 8, 12, 16, 20]
PoolWorker Process-8 output: [4, 8, 12, 16, 20]
PoolWorker Process-2 output: [4, 8, 12, 16, 20]
PoolWorker Process-10 output: [4, 8, 12, 16, 20]

Collectives™ on Stack Overflow

python multiple processing using pool

2 Answers 2

3 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related