27

I'm making a minor change to a very large file image file (just a few pixels difference) which takes a long time to transfer over the network.

Is there a way for rsync to identify the difference in the file and only send the small diff over the network?

Improve this question
2
  • 4
    When copying over a network, rsync behaves this way by default. As long as the minor change is a minor change to the file, and the file already exists on both sides of the link in near-identical versions, the data transferred should be much less than the total file size. From the first paragraph in the man page: "It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination." Commented May 22, 2017 at 16:23
  • Ah, ok, you should post that as an answer rather than a comment. It wasn't obvious that rsync was behaving this way, but with that expectation, I'll presume that my file changed more than I expected and look into why. The image compression process probably changes raw byte values throughout the file. Commented May 22, 2017 at 16:26

3 Answers 3

Reset to default
22

rsync delta-transfer algorithm does this by default. Quoting rsync manpage:

DESCRIPTION

Rsync is a fast and extraordinarily versatile file copying tool. It can copy locally, to/from another host over any remote shell, or to/from a remote rsync daemon. It offers a large number of options that control every aspect of its behavior and permit very flexible specification of the set of files to be copied. It is famous for its delta-transfer algorithm, which reduces the amount of data sent over the network by sending only the differences between the source files and the existing files in the destination. Rsync is widely used for backups and mirroring and as an improved copy command for everyday use.

If you want to disable it, you will have to use the -W or --whole-file option.

-W, --whole-file

This option disables rsync's delta-transfer algorithm, which causes all transferred files to be sent whole. The transfer may be faster if this option is used when the bandwidth between the source and destination machines is higher than the bandwidth to disk (especially when the "disk" is actually a networked filesystem). This is the default when both the source and destination are specified as local paths, but only if no batch-writing option is in effect.

If you really know how much your file have changed doing, you could even optimize this delta transfer behavior by tunning your delta block size:

-B, --block-size=BLOCKSIZE

This forces the block size used in rsync's delta-transfer algorithm to a fixed value. It is normally selected based on the size of each file being updated. See the technical report for details.

And if you want more information about the algorithm itself, you can find it here: The Rsync algorithm

Improve this answer
4
  • 3
    To be fair, the plural form of "files" in the manpage is ambiguous: "by sending only the differences between the source files and the existing files..." I took "files" to mean a collection of individual files, and that rsync would send (whole) files that were different. This should be clarified. Commented Nov 20, 2019 at 12:30
  • As a note, the algorithm isn't able to work well with compressed content so...if you upload a zip file with "a new file within it" you may end up re-uploading most of the file... Commented Nov 24, 2020 at 18:44
  • @rogerdpack I believe that this depends on how much of the file is changed by the compression if you change a small part of the original. If you have a 1 GB file, compress it, change 1 byte in the original file and compress it again, and the two compressed files also differ by only one byte, then it should not be a problem. However, there may be compression algorithms which change large parts of the compressed file even if you only change 1 byte in the source. Then it may be a problem. Commented Dec 15, 2020 at 16:29
  • Oddly, with a jar file, sometimes rsync works great. Which is also zipped but had some minor changes. So guess it depends on changes, thanks! Commented Jan 7, 2021 at 16:51
1

What you were looking for is the --partial and --inplace options. I found those yesterday as I need to update 100 GB to 300 GB single files over the network. It also works best with newer versions of rsync. The one in cygwin doesn't seem to work well at all. But from any semi-modern linux to linux, the limit isn't the network, but, rather, the speed that each end can read the already transferred files.

I use this specifically:

rsync -avPHx --inplace --partial src/foo remote_host:/target/path/

Edit - source was https://fedoramagazine.org/copying-large-files-with-rsync-and-some-misconceptions/ - note that --append, mentioned in the source article, can be dangerous in that if the file grows, AND has changes in the middle, --append hill not work as expected. Essentially only use --append for things like log files that are only ever appended to.

Improve this answer
0

What everyone says is true about how rsync operates, but the only formats where a small change to the image are likely to result in a small change to the file are raw bitmap formats (.bmp, .pnm, some types of .tif). The usual .png or JPEG, or Gimp or Photoshop .xcf or .psd files, those are already compressed so most likely a small image change would result in an almost totally different file on disk. That's why rsync's delta algorithm seems fairly ineffective.

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.