Capture output of complex shell command in python

Question

I would like to embed a command in a python script and capture the output. In this scenario I'm trying to use "find" to find an indeterminate number of files in an indeterminate number of subdirs, and grep each matching file for a string, something like:

grep "rabbit" `find . -name "*.txt"`

I'm running Python 2.6.6 (yeah, I'm sorry too, but can't budge the entire organization for this right now).

I've tried a bunch of things using subprocess, shlex, etc. that have been suggested in here, but I haven't found a syntax that will either swallow this, or ends up sucking the "find..."as the search string forgrep`, etc. Suggestions appreciated. Ken

It can be done easily using subprocess.call(..., shell=True) but such use of the shell actually makes for unsafe Python code. Why not traverse the directory from within Python and also do what grep does in Python? It's fairly simple and makes your code more portable. — 5gon12eder
– 5gon12eder, Commented Sep 19, 2014 at 16:01
@5gon12eder I may do this eventually, but kind of wanted to figure out how to do this mess as command output, too. Been looking at os.walk(). etc. Also just guessing that if I need to parse 5 or 6 hundred files, that "mygrep" might not be as quick as "realgrep". Still, I agree that traverse/python is an excellent choice, since what happens after I find the rabbits will take ages longer than finding them, however I do it. — Ken
– Ken, Commented Sep 19, 2014 at 16:23
related: How do I use subprocess.Popen to connect multiple processes by pipes? — jfs
– jfs, Commented Sep 22, 2014 at 23:19
@5gon12eder: if the command is trusted e.g., it is hardcoded in the source then there is nothing unsafe about shell=True compared to other ways to run it. It probably improves readability -- in a sense it might make the code safer (less bugs). — jfs
– jfs, Commented Sep 22, 2014 at 23:22
@J.F.Sebastian I was assuming that . will be replaced by some variable. If so, all doors for shell code injection are wide open. If the command is indeed a hard-coded string it will probably be fine as you say. Except that it ties your Python program to the POSIX platform for no good reason. Personally, I would almost never use the shell. Python is so powerful that it doesn't really add any convenience to spawn a shell. — 5gon12eder
– 5gon12eder, Commented Sep 23, 2014 at 13:17

GP89 · Accepted Answer · 2014-09-19 16:05:55Z

2

import subprocess


find_p = subprocess.Popen(["find", ".", "-name", "*.txt"], stdout=subprocess.PIPE)
grep_p = subprocess.Popen(["xargs", "grep", "rabbit"], stdin=find_p.stdout)

grep_p.wait()

edited Sep 19, 2014 at 16:05

answered Sep 19, 2014 at 16:00

GP89

6,7604 gold badges39 silver badges66 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

5gon12eder Over a year ago

This greps for "rabbit" in the list of file names instead of in the files. That is to say, it mimics find . -name "*.txt" | grep "rabbit" instead of the OP's command.

GP89 Over a year ago

@5gon12eder ah true I overlooked that, you could use xargs. Not identical, but should work

Ken Over a year ago

Thanks, this seems to work just great, in its second incarnation. I may eventually do the traverse/parse with python suggested above, but this will help get things off the dime quickly. - Ken

Collectives™ on Stack Overflow

Capture output of complex shell command in python

1 Answer 1

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related