0

I am doing a massive 200TB transfer with rsync. I used a shell script I found off the internet to spawn a few processes but here is the relevant rsync command itself:

        rsync --recursive \
              --whole-file \
              --inplace \
              --sparse \
              --no-compress \
              --max-alloc=8GiB \
              --size-only \
              --human-readable \
              --info=progress2 \
              --log-file="$rsync_file_basename.log" \
              --log-file-format="%o=%-7'''b | total=%-7'''l [%i] => %f%L" \
           "/mnt/disk${disk_id}/$share_name/$share_subdir/" \
           "$target_path" \
        >> "$rsync_file_basename.out"

The transfer is going pretty well but in watching the logs, I see two kinds of errors sporadically pop up.

mostly this:

rsync: [receiver] ftruncate failed on "/mnt/remotes/TS140_stuff/pictures/family_1/IMG345.jpg": Resource temporarily unavailable (11)

But also a few of these:

rsync: [receiver] write failed on "/mnt/remotes/TS140_stuff/shows/ep1.mp4": Resource temporarily unavailable (11) rsync error: error in file IO (code 11) at receiver.c(380) [receiver=3.2.7] rsync: [sender] write error: Broken pipe (32) rsync error: error in file IO (code 11) at io.c(1700) [sender=3.2.7]

Now, the target directory is an SMB mount and this is a lot of data so I expected some things like this.

But the reason for this post is that I can see the files at the target destination and they look fine. Does rsync have some kind of retry logic that it's using based on the command?

Improve this question

1 Answer 1

Reset to default
1

rsync doesn't really do retries. However there's nothing stopping you wrapping it in a loop:

#!/bin/sh
k=1 ss=1
while [ "$k" -le 5 ] 
do
    echo "Attempt $k" >&2
    rsync -rt ... &&
        ss=0 &&
        break
    sleep 60
    k=$((k+1))
done

if [ "$ss" -ne 0 ]
then
    echo "ERROR: rsync repeatedly failed" >&2
    exit 1
fi

Make sure you use the --times (-t) flag, otherwise this loop will keep repeating until $k runs out. Do you understand what the flags you've chosen actually do? They're a strange set, and I'd recommend removing most of them:

rsync -rtS
    --info=progress2 \
    --log-file="$rsync_file_basename.log" \
    --log-file-format="%o=%-7'''b | total=%-7'''l [%i] => %f%L" \
    "/mnt/disk$disk_id/$share_name/$share_subdir/" \
    "$target_path"

Finally, if you can access the server that provides the SMB share directly, you will gain a considerable speed improvement for (at least) retries by using rsync in its network mode:

rsync -rtSz local_path/ remote_host:remote_path
Improve this answer
3
  • thanks for your response. but in this case the files have been transferred and are in intact despite the errors. and im trying to understand why Commented Mar 6, 2024 at 11:13
  • @johncs, oh. That's not actually what you asked. The errors are transient and may go away if you repeat the copy process for the errored files. The ftruncate issues are probably warnings that rsync couldn't complete its attempt to make the target file sparse. The write issues are more serious: you might have a file that looks to be the right size but doesn't contain the correct data Commented Mar 6, 2024 at 11:38
  • thanks, i think ill do some checksum comparisons on the write error files to be sure! Commented Mar 6, 2024 at 13:08

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.