How to launch a couple of python scripts from a first python script and then terminate them all at once?

Question

I have a function in a python script which should launch another python script multiple times, I am assuming this can be done like this(Script is just my imagination of how this would work.)

iterations = input("Enter the number of processes to run")
for x in range(0, iterations):
    subprocess.call("python3 /path/to/the/script.py", shell=True)

but, I also need to pass over some defined variables into the other script, for example, if

x = 1

in the first script, then, I need x to have the same value in the second script without defining it there, I have NO idea how to do that.

And then also killing them, I have read about some method using PIDs, but don't those change every time?

Most of the methods I found on Google looked overly complex and what I want to do is really simple. Can anyone guide me in the right direction as to what to use and how I should go at accomplishing it?

How about using multiprocessing.Pool? You can then use map_async to pass arguments, or terminate to finish them. — Kolmar
– Kolmar, Commented Jan 30, 2015 at 21:40

cmidi · Accepted Answer · 2015-01-30 21:36:07Z

1

I have a function in a python script which should launch another python script multiple times, I am assuming this can be done like this(Script is just my imagination of how this would work.)

** Here is the subprocess manual page which contains everything I will be talking about https://docs.python.org/2/library/subprocess.html

One of the way to call one script from other is using subprocess.Popen something on the lines

   import subprocess
   for i in range(0,100):
       ret = subprocess.Popen("python3 /path/to/the/script.py",stdout=subprocess.PIPE,stderr=subprocess.PIPE,shell=True)

you can use the return value from Open to make the call synchronous using the communicate method.

  out,err = ret.communicate()

This would block the calling script until the subprocess finishes.

I also need to pass over some defined variables into the other script??

There are multiple ways to do this. 1. Pass parameters to the called script and parse it using OptionPraser or sys.args in the called script have something like

     from optparse import OptionParser
     parser = OptionParser()
     parser.add_option("-x","--variable",action="store_true",dest="xvalue",default=False)
     (options,args) = parser.parse_args()

     if options.xvalue == True:
          ###do something

in the callee script use subprocess as

     ret = subprocess.Popen("python3 /path/to/the/script.py -x",stdout=subprocess.PIPE,stderr=subprocess.PIPE,shell=True)

Note the addition of -x parameter

You can use args parse

https://docs.python.org/2/library/argparse.html#module-argparse
Pass the subprocess a environment variable which can be used to configure the subprocess. This is fast but this only works one way, i.e. from parent process to child process. in called script
```
 import os
 x = int(os.enviorn('xvalue'))
```

in callee script set the environment variable

    import os
    int x = 1
    os.environ['xvalue'] = str(x)

Use sockets or pipes or some other IPC method

And then also killing them, I have read about some method using PIDs, but don't those change every time?

again you can use subprocess to hold the process id and terminate it this will give you the process id

ret.pid

you can then use .terminate to terminate the process if it is running

ret.terminate()

to check if the process is running you can use the poll method from subprocess Popen. I would suggest you to check before you terminate the process

ret.poll()

poll will return a None if the process is running

edited Jan 30, 2015 at 21:36

answered Jan 30, 2015 at 20:53

cmidi

2,0603 gold badges24 silver badges37 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

user3907837 Over a year ago

But the script that is called contains an infinite while loop, I need a way to terminate that, and I don't really understand how I would pass variables onto the other script using that, can you elaborate on how I'd do that?

cmidi Over a year ago

updated the answer. You can always check the manual pages for more information

user3907837 Over a year ago

Ok, using your post, I did exactly what I wanted, minus one thing, say, I execute 5 processes (all under the ret variable), and then I do ret.terminate(), only the last opened process is closed, with 4 that I cannot close, how can I close the remaining 4 with just one command?

cmidi Over a year ago

I think one answer explains it. Store the returned object in a list or a dictionary whenever you call pOpen at different indexes. Now because u have objects for all created processes you can terminate whichever you want. There is one answer in this post that does that already

thek33per · Accepted Answer · 2015-01-30 21:07:18Z

0

If you just need to pass some values to second script, and you need to run that by means of subprocess module, then you may simply pass the variables as command line arguments:

for x in range(0, iterations):
    subprocess.call('python3 /path/to/second_script.py -x=%s'%x, shell=True)

And recieve the -x=1 via sys.argv list inside second_script.py (using argparse module)

On the other hand, If you need to exchange something between the two scripts dynamically (while both are running), You can use the pipe mechanism or even better, use the multiprocessing (wich requires some changes in your current code), it would make communication with and controlling it (terminating it) much cleaner.

answered Jan 30, 2015 at 21:07

thek33per

1424 bronze badges

Comments

tdelaney · Accepted Answer · 2015-01-30 21:37:19Z

You can pass variables to subprocesses via the command line, environment variables or passing data in on stdin. Command line is easy for simple strings that aren't too long and don't themselves have shell meta characters in them. The target script would pull them from sys.argv.

script.py:

import sys
import os
import time

x = sys.argv[1]
print(os.getpid(), "processing", x)
time.sleep(240)

subprocess.Popen starts child processes but doesn't wait for them to complete. You could start all of the children, put their popen objects in a list and finish with them later.

iterations = input("Enter the number of processes to run")
processes = []
for x in range(0, iterations):
    processes.append(subprocess.Popen([sys.executable, "/path/to/the/script.py", str(x)])
time.sleep(10)
for proc in processes:
    if proc.poll() is not None:
        proc.terminate()

for proc in processes:
    returncode = proc.wait()

Collectives™ on Stack Overflow

How to launch a couple of python scripts from a first python script and then terminate them all at once?

3 Answers 3

4 Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

4 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related