Python subprocess call with output and timeout

Question

Summary: I want to start an external process from Python (version 3.6), poll the result nonblocking, and kill after a timeout.

Details: there is an external process with 2 "bad habits":

It prints out the relevant result after an undefined time.
It does not stop after it printed out the result.

Example: maybe the following simple application resembles mostly the actual program to be called (mytest.py; source code not available):

import random
import time

print('begin')
time.sleep(10*random.random())
print('result=5')
while True: pass

This is how I am trying to call it:

import subprocess, time
myprocess = subprocess.Popen(['python', 'mytest.py'], stdout=subprocess.PIPE)
for i in range(15):
    time.sleep(1)
    # check if something is printed, but do not wait to be printed anything
    # check if the result is there
    # if the result is there, then break
myprocess.kill()

I want to implement the logic in comment.

Analysis

The following are not appropriate:

Use myprocess.communicate(), as it waits for termination, and the subprocess does not terminate.
Kill the process and then call myprocess.communicate(), because we don't know when exactly the result is printed out
Use process.stdout.readline() because that is a blocikg statement, so it waits until something is printed. But here at the end does not print anything.

The type of the myprocess.stdout is io.BufferedReader. So the question practically is: is there a way to check if something is printed to the io.BufferedReader, and if so, read it, but otherwise do not wait?

Orsiris de Jong · Accepted Answer · 2023-01-19 13:43:12Z

1

I think I got the exact package you need. Meet command_runner, which is a subprocess wrapper and allows:

Live stdout / stderr output
timeouts regardless of execution
process tree including child processes killing in case of timeout
stdout / stderr redirection to queues, files or callback functions

Install with pip install command_runner

Usage:

from command_runner import command_runner

def callback(stdout_output):
    # Do whatever you want here with the output
    print(stdout_output)

exit_code, output = command_runner("python mytest.py", timeout=300, stdout=callback, method='poller')

if exit_code == -254:
    print("Oh no, we got a timeout")
    print(output)

# Check for good exit_code and full stdout output here

If timeout is reached, you'll get exit_code -254 but still get to have output filled with whatever your subprocess wrote to stdout/stderr.

Disclaimer: I'm the author of command_runner

Additional non blocking examples using queues can be seen on the github page.

answered Jan 19, 2023 at 13:43

Orsiris de Jong

3,0581 gold badge29 silver badges52 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

Csaba Faragó Over a year ago

Thank you! Forgot to mention: there is no internet connection, so I cannot install any packages, and the command_runner is not installed.

Orsiris de Jong Over a year ago

Without internet connection, just download the zipped version from git and put the command_runner directory inside your project root so you can simply import it locally. psutil dependency is not mandatory, only helps better child process killing.

Csaba Faragó Over a year ago

But to download the zip an internet connection is needed. I can download it to the laptop I am working on right now, but cannot copy to the server I am working on. Similarly, I have Python 3.6 only, and cannot install newer.

Orsiris de Jong Over a year ago

Solution works on python 2.7+ so that's good. Of course I meant download zip on your current computer than copy to server. There must be a way for you to exchange data with the server, since you connect to it, you could copy paste via SSH or RDP, whatever you're using. You don't need more files than command_runner/__init__.py, you can even just copy paste file content. I have no better way of doing that, my solution fully answers your problem I guess. You could also analyze the code and write your own implementation, whatever.

Csaba Faragó Over a year ago

Oh, sorry; thank you! It works! I try to convince our system administrator to use your module.

|

Collectives™ on Stack Overflow

Python subprocess call with output and timeout

1 Answer 1

6 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

6 Comments

Your Answer

Sign up or log in

Post as a guest

Related