python, subprocess: reading output from subprocess

Question

I have following script:

#!/usr/bin/python

while True:
    x = raw_input()
    print x[::-1]

I am calling it from ipython:

In [5]: p = Popen('./script.py', stdin=PIPE)

In [6]: p.stdin.write('abc\n')
cba

and it works fine.

However, when I do this:

In [7]: p = Popen('./script.py', stdin=PIPE, stdout=PIPE)

In [8]: p.stdin.write('abc\n')

In [9]: p.stdout.read()

the interpreter hangs. What am I doing wrong? I would like to be able to both write and read from another process multiple times, to pass some tasks to this process. What do I need to do differently?

EDIT 1

If I use communicate, I get this:

In [7]: p = Popen('./script.py', stdin=PIPE, stdout=PIPE)

In [8]: p.communicate('abc\n')
Traceback (most recent call last):
  File "./script.py", line 4, in <module>
    x = raw_input()
EOFError: EOF when reading a line
Out[8]: ('cba\n', None)

EDIT 2

I tried flushing:

#!/usr/bin/python

import sys

while True:
        x = raw_input()
        print x[::-1]
        sys.stdout.flush()

and here:

In [5]: from subprocess import PIPE, Popen

In [6]: p = Popen('./script.py', stdin=PIPE, stdout=PIPE)

In [7]: p.stdin.write('abc')

In [8]: p.stdin.flush()

In [9]: p.stdout.read()

but it hangs again.

Daniel Stutzbach · Accepted Answer · 2010-09-27 21:28:33Z

15

I believe there are two problems at work here:

1) Your parent script calls p.stdout.read(), which will read all data until end-of-file. However, your child script runs in an infinite loop so end-of-file will never happen. Probably you want p.stdout.readline()?

2) In interactive mode, most programs do buffer only one line at a time. When run from another program, they buffer much more. The buffering improves efficiency in many cases, but causes problems when two programs need to communicate interactively.

After p.stdin.write('abc\n') add:

p.stdin.flush()

In your subprocess script, after print x[::-1] add the following within the loop:

sys.stdout.flush()

(and import sys at the top)

edited Sep 27, 2010 at 21:28

answered Sep 27, 2010 at 16:13

Daniel Stutzbach

77.1k17 gold badges90 silver badges79 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Daniel Stutzbach Over a year ago

@gruszczy: Thanks for the update. I've updated my answer to address another problem.

gruszczy Over a year ago

Adding sys.stdout.flush() to the script and using p.stdout.readline finally helped. Thanks a lot for your help.

Benji Mizrahi Over a year ago

Thank you Daniel Statzbash, for making my life easier. (flushing even after writing "\n")

JoshD · Accepted Answer · 2010-09-27 19:06:11Z

3

The subprocess method check_output can be useful for this:

output = subprocess.check_output('./script.py')

And output will be the stdout from the process. If you need stderr, too:

output = subprocess.check_output('./script.py', stderr=subprocess.STDOUT)

Because you avoid managing pipes directly, it may circumvent your issue.

answered Sep 27, 2010 at 19:06

JoshD

12.9k3 gold badges46 silver badges54 bronze badges

Comments

jfs · Accepted Answer · 2010-09-27 21:11:32Z

3

If you'd like to pass several lines to script.py then you need to read/write simultaneously:

#!/usr/bin/env python
import sys
from subprocess import PIPE, Popen
from threading  import Thread

def print_output(out, ntrim=80):
    for line in out:
        print len(line)
        if len(line) > ntrim: # truncate long output
            line = line[:ntrim-2]+'..'
        print line.rstrip() 


if __name__=="__main__":
    p = Popen(['python', 'script.py'], stdin=PIPE, stdout=PIPE)
    Thread(target=print_output, args=(p.stdout,)).start()
    for s in ['abc', 'def', 'ab'*10**7, 'ghi']:
        print >>p.stdin, s
    p.stdin.close()
    sys.exit(p.wait()) #NOTE: read http://docs.python.org/library/subprocess.html#subprocess.Popen.wait

Output:

4
cba
4
fed
20000001
bababababababababababababababababababababababababababababababababababababababa..
4
ihg

Where script.py:

#!/usr/bin/env python
"""Print reverse lines."""
while True:
    try: x = raw_input()
    except EOFError:
        break # no more input
    else:
        print x[::-1]

Or

#!/usr/bin/env python
"""Print reverse lines."""
import sys

for line in sys.stdin:
    print line.rstrip()[::-1]

Or

#!/usr/bin/env python
"""Print reverse lines."""
import fileinput

for line in fileinput.input(): # accept files specified as command line arguments
    print line.rstrip()[::-1]

edited Sep 27, 2010 at 21:11

answered Sep 27, 2010 at 20:32

jfs

417k210 gold badges1k silver badges1.7k bronze badges

2 Comments

Civilian Over a year ago

What does the "print >>p.stdin, s" syntax do? I'm unfamiliar with ">>"

jfs Over a year ago

@Civilian: "print chevron" is similar to print(s, file=p.stdin) which is available after from __future__ import print_function.

ssokolow · Accepted Answer · 2010-09-27 14:33:18Z

1

You're probably tripping over Python's output buffering. Here's what python --help has to say about it.

-u     : unbuffered binary stdout and stderr; also PYTHONUNBUFFERED=x
         see man page for details on internal buffering relating to '-u'

answered Sep 27, 2010 at 14:33

ssokolow

15.5k7 gold badges57 silver badges61 bronze badges

3 Comments

gruszczy Over a year ago

Can't I somehow force Python inside the code not to flush output? And why is it buffered, when it is pushed into pipe and not, when simply printed to the screen? Can I somehow flush the buffer programatically?

ssokolow Over a year ago

Output sent to pipes is buffered for performance reasons. (Flushing line-by-line is very inefficient, but you don't notice when the program spends most of its time waiting on the user) When flushing to pipes, buffering improves performance significantly, either improving completion times or reducing CPU load, depending on how I/O-bound the operation is.

ssokolow Over a year ago

As for flushing the buffer programmatically, you could try sys.stdout.flush() in the child process, but I've never used it and it's not really good architecture to use it in this manner. (It's more for use as part of a "sync to disk" process in concert with os.fsync) Better to let the parent process disable buffering as needed via either -u or the PYTHONUNBUFFERED environment variable.

Thud Foo · Accepted Answer · 2010-09-27 17:03:40Z

1

When you are through writing to p.stdin, close it: p.stdin.close()

answered Sep 27, 2010 at 17:03

Thud Foo

111 bronze badge

Comments

David Narayan · Accepted Answer · 2010-09-27 14:50:09Z

-3

Use communicate() instead of .stdout.read().

Example:

from subprocess import Popen, PIPE
p = Popen('./script.py', stdin=PIPE, stdout=PIPE, stderr=PIPE)
input = 'abc\n'
stdout, stderr = p.communicate(input)

This recommendation comes from the Popen objects section in the subprocess documentation:

Warning: Use communicate() rather than .stdin.write, .stdout.read or .stderr.read to avoid deadlocks due to any of the other OS pipe buffers filling up and blocking the child process.

answered Sep 27, 2010 at 14:50

David Narayan

5,4582 gold badges23 silver badges6 bronze badges

2 Comments

gruszczy Over a year ago

This doesn't work for me. I have made an edit in my question.

Daniel Stutzbach Over a year ago

He wants to communicate multiple times with the subprocess. communicate can't do that.

Collectives™ on Stack Overflow

python, subprocess: reading output from subprocess

6 Answers 6

3 Comments

Comments

Output:

2 Comments

3 Comments

Comments

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

6 Answers 6

3 Comments

Comments

Output:

2 Comments

3 Comments

Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related