Python running multiple processes

Question

I have a script (a.py) that needs to call another script (b.py) multiple times, each execution of b.py takes 1 minute. If I run in a for loop, this takes a lot of time.

I want to know how I can optimize this, to reduce the time. Any help or suggestions would be very helpful.

Source code:

# a.py
import os

if __name__ == '__main__':
    inputs = ['file1', 'file2', 'file3', 'file4']
    script_path = 'b.py'

    # Some logging statements. 
    for infile in inputs:
    os.system("{} -i {}".format(script_path, infile))  


# b.py
    # take arguments
    # perform some operations
    # log the execution

So far, I have been using os.system to call the other script. How can I call the script b.py n times in parallel ?

It would be helpful if you edited your post to show what you have tried. All of the generous programmers on stackoverflow are much more helpful if you provide something as a base for us to work with. The least you can do is provide a minimal, complete, and verifiable example of your code. — Jebby
– Jebby, Commented Jan 20, 2018 at 11:22

clemens · Accepted Answer · 2018-01-20 15:45:17Z

4

You may use muliprocessing.Process to run it in parallel:

from multiprocessing import Process

inputs = ['file1', 'file2', 'file3', 'file4']
script_path = 'b.py'

def run(script, name):
    os.system("{} -i {}".format(script, name))  

if __name__ == '__main__':
    inputs = ['file1', 'file2', 'file3', 'file4']
    script_path = 'b.py'
    for infile in inputs:
        p = Process(target=run, args=(script_path, infile))
        p.start()
    p.join()

Note: Executing a Python script from a Python script with os.system is not very elegant. You should modify your script b.py that it works as a module which provides its main functionality with functions or classes as well. Then you can import b and use these functions or classes instead.

answered Jan 20, 2018 at 15:45

clemens

17.9k12 gold badges52 silver badges71 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

R George Over a year ago

Subprocess looks good, I haven't used this module before. I will give it a go. Also, even if I make it a module and import the functions, it still works serially, right ? How can I invoke it in parallel.

clemens Over a year ago

The parallel execution is done by subprocess. Of course, my suggestion is to execute the code of module b in parallel.

Philip Adler · Accepted Answer · 2018-01-20 15:33:47Z

0

There isn't a way you can optimise this as such. If you want to run script b for each item in a list, then iteration is the only way to do this. It might be that you can alter the contents of script b to run faster, but that is a different problem.

answered Jan 20, 2018 at 15:33

Philip Adler

2,20620 silver badges27 bronze badges

Collectives™ on Stack Overflow

Python running multiple processes

2 Answers 2

2 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related