21

I have a program that writes to fd3 and I want to process that data with grep and sed. Here is how the code looks so far:


exec 3> >(grep "good:"|sed -u "s/.*:\(.*\)/I got: \1/")
echo "bad:data1">&3
echo "good:data2">&3

Nothing is output until I do a

exec 3>&-

Then, everything that I wanted finally arrives as I expected:

I got: data2

It seems to reply immediately if I use only a grep or only a sed, but mixing them seems to cause some sort of buffering. How can I get immediate output from fd3?

Improve this question

6 Answers 6

Reset to default
16

You only need to tell grep and sed to not bufferize lines:

grep --line-buffered

and

sed -u
Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

12

I think I found it. For some reason, grep doesn't automatically do line buffering. I added a --line-buffered option to grep and now it responds immediately.

Improve this answer

6 Comments

grep expects to be fed either a huge file or a huge pipe-stream. Fully buffered is much more efficient in those cases. (Be glad you're on Linux. This OSX computer I'm typing this on doesn't have line-buffering options for either grep or sed.)
Wow. I'm so sorry about your OSX machine. Even more sorry that OSX actually costs money. Less features, more money, how did Apple pull that off?
OSX isn't really for command line hacking. I'm glad they kept the BSD underpinnings around, but on this computer I spend far more time in things which are not CLI.
Really? I love the BSD license concept, but from the little that I've seen, much of BSD software is missing several nice features. Are there some nice features in BSD not in GNU or are you just saying that BSD bash is nicer than AppleScript (or whatever Mac used to have)?
I think you misunderstand. I have a Mac ... mainly because my job standardized on Macs. And it is nice for the things it is actually optimized for, i.e. big GUI applications. I am a Unix weenie at heart, though, and I'm happy that there is a Unix shell environment for dinking around with, but if I want to do anything serious I go use my Linux box. The BSDness of that environment is accidental; that's just what Apple happened to use for their kernel. (Apple does have a serious hate-on for the GPL, alas. I do think the GNU shell environment is superior.)
|
6

An alternate means to stop sed from buffering is to run it through the s2p sed-to-Perl translator and insert a directive to have it command-buffered, perhaps like

BEGIN { $| = 1 }

The other reason to do this is that it gives you the more convenient notation from EREs instead of the backslash-annoying legacy BREs. You also get the full complement of Unicode properties, which is often critical.

But you don’t need the translator for such a simple sed command. And you do not need both grep and sed, either. These all work:

perl -nle 'BEGIN{$|=1} if (/good:/) { s/.*:(.*)/I got: $1/; print }'

perl -nle 'BEGIN{$|=1} next unless /good:/; s/.*:(.*)/I got: $1/; print'

perl -nle 'BEGIN{$|=1} next unless /good:/; s/.*:/I got: /; print'

Now you also have access to the minimal quantifier, *?, +?, ??, {N,}?, and {N,M}?. These now allow things like .*? or \S+? or [\p{Pd}.]??, which may well be preferable.

Improve this answer

1 Comment

I actually went the perl route as you recommended. I was not aware of its usefulness in bash scripts. perl and bash definitely go well together.
4

Turn off buffering in pipe seems to be the easiest and most generic answer. Using stdbuf (coreutils) :

exec 3> >(stdbuf -oL grep "good:" | sed -u "s/.*:\(.*\)/I got: \1/")
echo "bad:data1">&3
echo "good:data2">&3
I got: data2

Buffering has other dependencies, for example depending on mawk either gawk reading this pipe :

exec 3> >(stdbuf -oL grep "good:" | awk '{ sub(".*:", "I got: "); print }')

In that case, mawk would retain the input, gawk wouldn't.

See also How to fix stdio buffering

Improve this answer

Comments

3

You can merge the grep into the sed like so:

exec 3> >(sed -une '/^good:/s//I got: /p')
echo "bad:data1">&3
echo "good:data2">&3

Unpacking that a bit: You can put a regexp (between slashes as usual) before any sed command, which makes it only be applied to lines that match that regexp. If the first regexp argument to the s command is the empty string (s//whatever/) then it will reuse the last regexp that matched, which in this case is the prefix, so that saves having to repeat yourself. And finally, the -n option tells sed to print only what it is specifically told to print, and the /p suffix on the s command tells it to print the result of the substitution.

The -e option is not strictly necessary but is good style, it just means "the next argument is the sed script, not a filename".

Always put sed scripts in single quotes unless you need to substitute a shell variable in there, and even then I would put everything but the shell variable in single quotes (the shell variable is, of course, double-quoted). You avoid a bunch of backslash-related grief that way.

Improve this answer

5 Comments

Great info -- I used to know all that!
sed, awk, grep, perl..they are all so powerful. I wish it were all just one utility because I just don't have bandwidth to learn all of them.
The right way to think about it is, /bin/sh + everything in "coreutils" is one utility that happens to be divided into many executables. Awk and perl are their own things, though, and honestly, nowadays I don't think it's worth bothering to learn awk, because perl is strictly better, especially if you learn its command line options. (You used to have to worry about systems that didn't have perl, but I don't think that's relevant anymore.) Also, a critical skill in this area is knowing when to abandon shell and use perl or python for the whole thing instead.
Yeah, I'm starting to notice the failing of awk myself. I really liked it's C-ish syntax, but perl definitely has the extra features when you need them. I'm still so-so on python. Does it have useful extras missing from perl?
Perl versus Python is almost entirely a matter of personal taste, IMO; both have nearly identical capabilities, and oodles of library. Personally I find Python a much better match for the way I want to think about problems that need nontrivial data structures, and nowadays I reach for it for text bashing (beyond what sed can do reasonably) as well, just because I'm more in practice with it. But I know plenty of people who feel just the opposite.
2

On a Mac, brew install coreutils and use gstdbuf to control buffering of grep and sed.

Improve this answer

1 Comment

JFTR, it's just stdbuf on GNU/Linux systems.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.