Moving files around
Three tools cover nearly every transfer, and the choice between them is usually obvious once you know what each is for.
rsync, for almost everything
rsync -aHAX --info=progress2 /source/ user@host:/target/
The flags that matter:
-a— recursive, keeps permissions, times, symlinks, ownership-H— preserves hard links, which matters for backup trees and some mail stores-A -X— ACLs and extended attributes, easy to forget and annoying to discover missing--numeric-ids— stop user ids being remapped when the accounts differ between machines--dry-run— see what it would do first, always, on anything with--deletein it
The trailing slash decides everything.
/source/ copies the contents; /source copies the directory
itself into the target. Half the rsync accidents in the world are this.
It resumes: run it again after an interruption and it transfers only what is
missing or changed. For a single huge file that was cut off part-way, add
--partial --append-verify.
scp, for one small file
scp bestand.tar.gz host:/tmp/
Fine for a config file. Do not use it for a directory tree or anything large: no resume, no delta transfer, and on a failure you start again.
rclone, for object storage
rclone sync /srv/data remote:bucket --progress
Speaks S3 and most other object protocols, parallelises properly, and is the right
tool for pushing to or pulling from your own object
store. Use copy rather than sync unless you have thought
hard about what gets deleted.
Between your own machines, at full speed
If both ends are with us, the transfer runs across our network rather than the internet — port speed, nothing metered. Over a private interface it is faster still, and over an interface with jumbo frames faster again. See connecting your own servers and jumbo frames.
For a very large one-off where encryption is the bottleneck rather than the link,
-e 'ssh -c aes128-gcm@openssh.com' uses a cheaper cipher. Only on a
private network, and only when you have measured that the CPU is the limit.
Verify, for anything that matters
rsync checksums as it goes, but a final independent check costs little:
find /target -type f -exec sha256sum {} + | sort -k2 > target.sha
and compare with the same on the source. For a migration of customer data, do this — see moving to another server.
Run it in tmux
Any transfer over a few minutes belongs in a session that survives your connection dropping. See long-running jobs.
Still stuck? Mail support@novogara.com — an engineer answers, at any hour. Back to the knowledge base