Why Python multiprocessing is running sequencially?

Question

I got some really simple code below. #!/usr/bin/python from multiprocessing import Pool import time

def worker(job):
    if job in range(25,30):
        time.sleep(10)
    print "job:%s" %job
    return (job)

pool = Pool(processes=10)
result = []

for job in range(1, 1000):
    result.append(pool.apply_async(worker(job)))
pool.close()
pool.join()

As you can see, I have a worker to handle 1000 jobs use multiprocessing. If the job is 25-30, then the worker will sleep 10s. This is try to simulate a time/resource cost job.

When I run the above code, the out put is like below. From job 25. The whole process is running like a sequencial process.Because every 10s there is output after the job 24. Until the job 30 is finished.

But why? Shouldn`t multiprocessing process run concurrently?

[root@localhost tmp]# ./a.py 
job:1
job:2
job:3
job:4
job:5
job:6
job:7
job:8
job:9
job:10
job:11
job:12
job:13
job:14
job:15
job:16
job:17
job:18
job:19
job:20
job:21
job:22
job:23
job:24


job:25
job:26
...

Daniel Roseman · Accepted Answer · 2016-01-12 13:05:56Z

6

Because you're calling it on instantiation. You should pass the callable and the arguments, not the result, to apply_async.

result.append(pool.apply_async(worker, [job]))

answered Jan 12, 2016 at 13:05

Daniel Roseman

602k68 gold badges910 silver badges923 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Why Python multiprocessing is running sequencially?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related