ssh breaks out of while-loop in bash [duplicate]

Question

I use this bash-code to upload files to a remote server, for normal files this works fine:

for i in `find devel/ -newer $UPLOAD_FILE`
do
    echo "Upload:" $i
    if [ -d $i ]
    then
        echo "Creating directory" $i
        ssh $USER@$SERVER "cd ${REMOTE_PATH}; mkdir -p $i"
        continue
    fi
    if scp -Cp $i $USER@$SERVER:$REMOTE_PATH/$i
    then
        echo "$i OK"
    else
        echo "$i NOK"
        rm ${UPLOAD_FILE}_tmp
    fi
done

The only problem is that for files with a space in the name, the for-loop fails, so I replaced the first line like this:

find devel/ -newer $UPLOAD_FILE | while read i
do
    echo "Upload:" $i
    if [ -d $i ]
    then
        echo "Creating directory" $i
        ssh $USER@$SERVER "cd ${REMOTE_PATH}; mkdir -p $i"
        continue
    fi
    if scp -Cp $i $USER@$SERVER:$REMOTE_PATH/$i
    then
        echo "$i OK"
    else
        echo "$i NOK"
        rm ${UPLOAD_FILE}_tmp
    fi
done

For some strange reason, the ssh-command breaks out of the while-loop, therefore the first missing directory is created fine, but all subsequent missing files/directories are ignored.

I guess this has something to do with ssh writing something to stdout which confuses the "read" command. Commenting out the ssh-command makes the loop work as it should.

Does anybody know why this happens and how one can prevent ssh from breaking the while-loop?

As an aside -- all-uppercase variable names are reserved by convention for environment variables and builtins; they shouldn't be used for variables local to a script. — Charles Duffy
– Charles Duffy, Commented Mar 3, 2015 at 17:15
while read will break badly when your filenames contain literal backslashes. Safer to use read -r. Similarly, read will strip trailing whitespace from names; to avoid that, you need to clear IFS. — Charles Duffy
– Charles Duffy, Commented Mar 3, 2015 at 17:16
Also, find | while read emits a newline-delimited stream, but legitimate filenames on UNIX are allowed to contain newlines. Think about what happens if someone does a mkdir -p devel/$'\n'/etc and then writes to devel/$'\n'/etc/passwd; you'd better hope at this point that your script doesn't have permission to write to /etc/passwd on the remote machine. — Charles Duffy
– Charles Duffy, Commented Mar 3, 2015 at 17:18
Also, [ -d $i ] will misbehave if $i contains spaces or a wildcard expression, or is empty. Either use [[ -d $i ]] (if your shell is bash or another ksh derivative) or [ -d "$i" ], with the quotes (for POSIX compatibility). — Charles Duffy
– Charles Duffy, Commented Mar 3, 2015 at 17:20
I just now found this problem, and just wanted to note, that the error condition only occurs when ssh successfully makes a connection with the remote machine. It does not happen when the connection is refused. — Pysis
– Pysis, Commented Aug 9, 2017 at 19:23

choroba · Accepted Answer · 2016-03-08 09:45:23Z

322

The problem is that ssh reads from standard input, therefore it eats all your remaining lines. You can just connect its standard input to nowhere:

ssh $USER@$SERVER "cd ${REMOTE_PATH}; mkdir -p $i" < /dev/null

You can also use ssh -n instead of the redirection.

edited Mar 8, 2016 at 9:45

answered Feb 22, 2012 at 10:34

choroba

245k27 gold badges221 silver badges304 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

jordanm Over a year ago

You can also use ssh -n to accomplish the same thing.

Charles Duffy Over a year ago

This is fairly buggy, by the way. What if $1 is a filename with spaces? You end up creating more than one directory. Similarly for $REMOTE_PATH. printf %q could be used to quote variables to be safely passed through the extra shell that ssh invokes.

wojciii Over a year ago

Thank you. :D It took me some debugging before I noticed that the issue was executing commands with ssh and not another part of script.

ethan Over a year ago

Could you elaborate what do you mean by ssh reads from stdin? I tried "ssh $USER@$SERVER "some commands"; echo "blabla"; and it works fine, so I don't understand what exactly it means?

choroba Over a year ago

@ethan: What do you mean by "works fine"? What did you expect? The echo is run after ssh is closed, so ssh can't read from it.

Charles Duffy · Accepted Answer · 2015-03-03 17:24:25Z

Another approach is to loop over a FD other than stdin:

while IFS= read -u 3 -r -d '' filename; do
  if [[ -d $filename ]]; then
    printf -v cmd_str 'cd %q; mkdir -p %q' "$REMOTE_PATH" "$filename"
    ssh "$USER@$SERVER" "$cmd_str"
  else
    printf -v remote_path_str '%q@%q:%q/%q' "$USER" "$SERVER" "$REMOTE_PATH" "$filename"
    scp -Cp "$filename" "$remote_path_str"
  fi
done 3< <(find devel/ -newer "$UPLOAD_FILE" -print0)

The -u 3 and 3< operators are critical here, using FD 3 rather than the default FD 0 (stdin).

The approach given here -- using -print0, a cleared IFS value, and the like -- is also less buggy than the original code and the existing answer, which can't handle interesting filenames correctly. (Glenn Jackman's answer is close, but even that can't deal with filenames with newlines or filenames with trailing whitespace).

The use of printf %q is critical to generate commands which can't be used to attack the remote machine. Consider what would happen with a file named devel/$(rm -rf /)/hello with code which didn't have this paranoia.

Collectives™ on Stack Overflow

ssh breaks out of while-loop in bash [duplicate]

2 Answers 2

5 Comments

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

5 Comments

Comments

Linked

Related