How to execute a bash script from within python in a non-blocking way, and allowing to see its output?

Question

On a Ubuntu Linux system I want to start a bash script from within a python script. I need to run it in a non-blocking way so that the python script reports all output from the bash script to the standard output.

I have found a very similar question here which does not work for me (of course). Here is my bash script testbash.sh which just prints a row of numbers:

#!/bin/bash
i=0

while [ $i -le 10 ]
do
  echo Number: $i
  ((i++))
  sleep 1
done

Here is the python script:

import sys
import time
from subprocess import PIPE, Popen
from threading  import Thread

try:
    from queue import Queue, Empty
except ImportError:
    from Queue import Queue, Empty  # python 2.x


ON_POSIX = 'posix' in sys.builtin_module_names

def enqueue_output(out, queue):
    for line in iter(out.readline, b''):
        queue.put(line)
    out.close()

p = Popen(['.','testbash.sh'], stdout=PIPE, bufsize=1, close_fds=ON_POSIX, shell=True)
q = Queue()
t = Thread(target=enqueue_output, args=(p.stdout, q))
t.daemon = True # thread dies with the program
t.start()

# read line without blocking
running = True
while running:
    time.sleep(1)
    try:
        line = q.get_nowait() # or q.get(timeout=.1)
    except Empty:
        print('no output yet')
    else:
        # got line
        print(line)

When running this script, I only get the output 'no output yet' and no actual output from the bash script.

I want the python script to run as long as the bash script runs, redirecting the output from the bash script to the python standard-output (so that I see the numbers 1..10 in like every 1-2 seconds), and after ~10 seconds the python script stops. I also need to check if the bash script did finish fine or with an error

What am I missing? Is the script actually running? How to get its output? How to check the exit code?

Try Popen(['./testbash.sh','testbash.sh'],... or simply Popen('./testbash.sh',... The first argument is the program to run. Your original program tries to run '.' which is the current directory. — Bodo
– Bodo, Commented Jan 28, 2020 at 16:42

Philippe · Accepted Answer · 2020-01-28 19:23:41Z

1

A few changes need to be made, in particular, the calling of shell script:

import sys
import time
from subprocess import PIPE, Popen
from threading  import Thread

try:
    from queue import Queue, Empty
except ImportError:
    from Queue import Queue, Empty  # python 2.x


ON_POSIX = 'posix' in sys.builtin_module_names

def enqueue_output(out, queue):
    for line in iter(out.readline, b''):
        queue.put(line.decode('ascii'))
    out.close()

p = Popen(['./testbash.sh'], stdout=PIPE, bufsize=1, close_fds=ON_POSIX, shell=True)
q = Queue()
t = Thread(target=enqueue_output, args=(p.stdout, q))
t.daemon = True # thread dies with the program
t.start()

# read line without blocking
while t.is_alive():
    time.sleep(1)
    try:
        line = q.get_nowait() # or q.get(timeout=.1)
    except Empty:
        print('no output')
    else:
        # got line
        print(line, end='')

p.wait()
print(f'returncode = {p.returncode}')

edited Jan 28, 2020 at 19:23

answered Jan 28, 2020 at 18:05

Philippe

27.7k4 gold badges28 silver badges45 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Alex Over a year ago

Thanks a lot for the fixes - is there a way to get the exit status of the bash script?

Philippe Over a year ago

Updated, at the end

Collectives™ on Stack Overflow

How to execute a bash script from within python in a non-blocking way, and allowing to see its output?

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related