Python pool process management

Question

I was running some Python code and realized when I instantiate Pool with only one process, for instance:

from multiprocessing.pool import Pool
from time import sleep

def f(i):
  print(i)
  sleep(10)

with Pool(1) as p:
    p.map(f, [i for i in range(100)])

Actually five processes are currently running. I've also noticed a pattern going on: if I instantiate Pool with 1, 2,3,... processes the number of processes launched by Python are 5,6,7,... I'm curious: Does Pool use three process for management?

@Javier, I think so. When I look at top (linux) they have different pids. — Daniel
– Daniel, Commented Feb 25, 2018 at 0:00

georgexsh · Accepted Answer · 2018-02-27 06:48:00Z

1

with Pool(1), you will get 2 processes, the main process (pid 31070), and one worker process (pid 31071), but 3 extra threads/LWPs in the main process (LWP/thread id 31072/31073/31074):

  PID  PPID   LWP  NLWP CMD
31070 21240 31070     4 python3 so_48968836_mp.py
31070 21240 31072     4 python3 so_48968836_mp.py
31070 21240 31073     4 python3 so_48968836_mp.py
31070 21240 31074     4 python3 so_48968836_mp.py
31071 31070 31071     1 python3 so_48968836_mp.py

those three threads are for pool workers maintaining, async task and result handling.

edited Feb 27, 2018 at 6:48

answered Feb 27, 2018 at 4:10

georgexsh

16.8k3 gold badges45 silver badges71 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Python pool process management

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related