Interaction between Python script and linux shell

Question

I have a Python script that needs to interact with the user via the command line, while logging whatever is output.

I currently have this:

# lots of code

popen = subprocess.Popen(
    args,
    shell=True,
    stdin=sys.stdin,
    stdout=sys.stdout,
    stderr=sys.stdout,
    executable='/bin/bash')

popen.communicate()

# more code

This executes a shell command (e.g. adduser newuser02) just as it would when typing it into a terminal, including interactive behavior. This is good.

Now, I want to log, from within the Python script, everything that appears on the screen. But I can't seem to make that part work.

I've tried various ways of using subprocess.PIPE, but this usually messes up the interactivity, like not outputting prompt strings.

I've also tried various ways to directly change the behavior of sys.stdout, but as subprocess writes to sys.stdout.fileno() directly, this was all to no avail.

As a side note, is there a reason you're using shell=True? Especially when the thing you're launching is itself /bin/bash? You're basically telling Python to start a copy of bash to tell it to start another copy of bash to (presumably, since you didn't show the whole code) tell it to start a copy of adduser. You probably don't need even one shell involved, but I can't imagine why you'd need two. — abarnert
– abarnert, Commented Jul 24, 2014 at 17:36
@abarnert: executable doesn't start a new bash process: it tells subprocess module to use its value instead of /bin/sh if shell=True. args is presumably some shell command with bash-isms such as 'use_tty &>/dev/null'. As I understand the intent is: script -c '/bin/bash -c "$args"' log. Though as my answer shows, you don't necessarily need even one additional bash process to run a command here. — jfs
– jfs, Commented Apr 25, 2015 at 22:35

Community · Accepted Answer · 2017-05-23 11:51:03Z

Popen might not be very suitable for interactive programs due to buffering issues and due to the fact that some programs write/read directly from a terminal e.g., to retrieve a password. See Q: Why not just use a pipe (popen())?.

If you want to emulate script utility then you could use pty.spawn(), see the code example in Duplicating terminal output from a Python subprocess or in log syntax errors and uncaught exceptions for a python subprocess and print them to the terminal:

#!/usr/bin/env python
import os
import pty
import sys

with open('log', 'ab') as file:
    def read(fd):
        data = os.read(fd, 1024)
        file.write(data)
        file.flush()
        return data

    pty.spawn([sys.executable, "test.py"], read)

Or you could use pexpect for more flexibility:

import sys
import pexpect # $ pip install pexpect

with open('log', 'ab') as fout:
    p = pexpect.spawn("python test.py")
    p.logfile = fout # or .logfile_read
    p.interact()

If your child process doesn't buffer its output (or it doesn't interfere with the interactivity) and it prints its output to its stdout or stderr then you could try subprocess:

#!/usr/bin/env python
import sys
from subprocess import Popen, PIPE, STDOUT

with open('log','ab') as file:
    p = Popen([sys.executable, '-u', 'test.py'],
              stdout=PIPE, stderr=STDOUT,
              close_fds=True,
              bufsize=0)
    for c in iter(lambda: p.stdout.read(1), ''):
        for f in [sys.stdout, file]:
            f.write(c)
            f.flush()
    p.stdout.close()
    rc = p.wait()

To read both stdout/stderr separately, you could use teed_call() from Python subprocess get children's output to file and terminal?

rogue-one · Accepted Answer · 2014-03-05 10:23:19Z

0

This should work

import subprocess
f = open('file.txt','w')
cmd = ['echo','hello','world']
subprocess.call(cmd, stdout=f)

answered Mar 5, 2014 at 10:23

rogue-one

11.6k8 gold badges56 silver badges79 bronze badges

4 Comments

HetMes Over a year ago

This doesn't show the output in the console. Nor does this work with commands that require user input, such as 'adduser'

rogue-one Over a year ago

Your question suggested that you want the output to be logged into a file. Do you want it to be logged in both file and in the console?

HetMes Over a year ago

Yes, the idea is that at the end of the python script, I have a complete log of everything that happened, including any logging explicitly generated by the script itself.

jfs Over a year ago

@user480441: adduser might expect the input directly from terminal. Otherwise the child process started by call() should inherit its parent stdin i.e., you should be able to provide input e.g., try subprocess.call('python').

Collectives™ on Stack Overflow

Interaction between Python script and linux shell

2 Answers 2

Comments

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related