On 11/18/07, Jay R. Ashworth jra@baylink.com wrote:
Well, I gather the new version of rsync is *much* smarter that the old versions were about rilly, rilly big files, so perhaps this is worth revisiting.
Really big files aren't the issue, it's a really large number that's the issue. rsync < 3.0 will first create a list, in memory, of all files it's going to transfer. Only once it's made the list will it start the transfer. When I moved my server and used rsync to copy the entire contents of the old filesystem to the new server, it used several hundred MB of memory before it even started transferring files. The same happens for *image* files, since there are so many.
For a single large file, rsync's clever rolling diff algorithm might or might not be entirely optimal, but I haven't heard of either complaints against it or improvements in recent versions.