Bash: how to filter tee output in realtime for ssh command?

Question

My goal is to make a timestamped log of the commands used in my ssh sessions.

Tee works in realtime as long as the output is not filtered. When I tail -F test.log, the following command appends output in realtime:

#!/bin/bash
ssh "$@" | tee -a test.log

However, when I try to modify tee's output by methods suggested in this question, the output is no longer in realtime. For example:

#!/bin/bash
ssh "$@" | tee >(grep -e '.*\@.*\:.*\$.*' --color=never >> test.log)

Strangely, if I substitute the "yes" command instead of the ssh command, the output is filtered properly in realtime.

Realtime processing is important because my script needs to append a current timestamp to each line and remove as much output as possible. Here is my script so far:

#!/bin/bash
logfile=~/test.log
desc="sshlog ${@}"
tab="\t"
format_line() {
    while IFS= read -r line; do
        echo "$(date +"%Y-%m-%d %H:%M:%S %z")${tab}${desc}${tab}${line}"
    done
}
echo "[START]" | format_line >> $logfile
ssh "$@" | tee >(grep -e '.*\@.*\:.*\$.*' --color=never | format_line >> $logfile)
echo "[END]" | format_line >> $logfile

How can I fix this, and why is the ssh command acting differently with tee than the yes command?

ruakh · Accepted Answer · 2014-06-23 22:54:36Z

13

The problem is most likely that grep is buffering its output — selecting large chunks of input, filtering them, and outputting the result — so it handles the output of yes smoothly (because yes quickly generates a lot of input for it to filter and output), whereas your ssh command probably doesn't generate as much output as quickly.

Many versions of grep offer a mechanism to tweak this buffering. Since you're on Linux, you're likely using GNU Grep, which offers a --line-buffered flag for this purpose (see the "Other Options" section of the GNU Grep Manual), so that the output is buffered just one line at a time:

ssh "$@" | tee >(grep -e '.*\@.*\:.*\$.*' --color=never --line-buffered >> test.log)

answered Jun 23, 2014 at 22:54

ruakh

185k29 gold badges292 silver badges324 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

phyatt Over a year ago

if you were using awk instead of grep you would use fflush(); . This question is related: stackoverflow.com/questions/21098382/…

GrayFox Over a year ago

@ruakh I've tested your command of tee output redirection Generally speaking: 'some command' | tee >( 'some filtering' > 'output' ) It works using /bin/bash it works. But I got syntax errors using it in /bin/sh, such as: Syntax error: "(" unexpected. Any suggest? Thanks for your support :)

ruakh Over a year ago

@GrayFox: The process substitution (the >(...) feature) here is actually in the question -- I just included it here to show the --line-buffered option in context -- so I think it's off-topic for this question to ask how to approximate process substitution in a shell that doesn't support it. Fortunately, you don't need to ask: this question already links to stackoverflow.com/q/12205250/978917, which has several answers showing non-process-substitution-based approaches.

Collectives™ on Stack Overflow

Bash: how to filter tee output in realtime for ssh command?

1 Answer 1

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related