2

I am transferring a file from source computer C1 to destination computer D1 [ not within the same lan] using rsync. File A in C1(source) is an updated version of the file B in D1(destination). file A is 850mb while file B is 530 mb.

command used: 

rsync -e "ssh -p 2222 -o StrictHostKeyChecking=no -o ConnectTimeout=10" -avvvz --stats --progress fileA.tar username@destIP:fileB.tar

here are the stats obtained:

 hash search b=25600 len=899737600  
 Number of files: 1  
 Number of files transferred: 1 
 Total file size: 899737600 bytes 
 Total transferred file size: 899737600 bytes 
 Literal data: 709324800 bytes 
 Matched data: 190412800 bytes       
 File list size: 38 
 File list generation time: 0.001 seconds 
 File list transfer time: 0.000 seconds  
 Total bytes sent: 617865859  
 Total bytes received: 153142     
 sent 617865859 bytes  received 153142 bytes  3501524.08 bytes/sec     
 total size is 899737600  speedup is 1.46
 data sent - 617865859 bytes (590mb approx)

According to rsync's delta algorithm only the difference of 320 mb should be transferred along with extra bytes for checksum and instructions to bring all the transferred data together. But a total of 590mb is being transferred. Why is there an additional 270mb transferred?

Is this additional data transfer because of extra data passed as rsync instruction or checksum , or is there transfer of additional data from file A apart from the difference of 320mb - which means the delta algorithm is not very efficient in this case ?

Improve this question
6
  • The algorithm used by rsync assumes that the files are blockwise identical. If you have many minor changes spread all over the file, the difference may be small, but the algorithm won't find matching blocks. You may optimize efficiency by a lower --block-sze in this case. Commented Oct 4, 2017 at 12:50
  • i tried for various block sizes, 100kb , 50kb , 25kb the best result was shown for the default block size chosen by the algorithm which was 29kb, there were only minor differences of few mb's but comparatively 29kb had minimum data transfer and comparatively higher speedup Commented Oct 4, 2017 at 13:37
  • You can extra debug info on the deltas with --debug=DELTASUM4. Commented Oct 4, 2017 at 18:20
  • I'm not 100% certain, but I believe the data sent value includes protocol overhead as well as metadata that's not accounted in total size, so it's an accurate representation of how much data was actually sent across the network, whereas the total size value only indicates the differences in the file data. Commented Oct 4, 2017 at 18:28
  • @AustinHemmelgarn - this is the doubt i have, if the extra data is a protocol overhead or as Philippos suggested if the extra data is due to the fact that the change in data is spread accross many blocks which are eventually sent. Commented Oct 5, 2017 at 7:39

1 Answer 1

Reset to default
1

You are transferring an updated tar archive. The old copy at the destination is about 530 Mb and the updated file is 850 Mb. The difference in size is 320 Mb, but I'm assuming that there are also differences in the first 530 Mb of the file that must be transferred.

If the updated archive only had things appended to it, then your concern would have been correct, but if you recreated the archive it only takes that two files in the first 530 Mb of the updated archive are added in a different order, or that the data that was added to the archive is actually spread across the whole archive in smaller files, for rsync to detect changes there too.

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.