Thread by @devops_tech

Copying files and directories remotely.

Keeping Linux files and directories in sync.

Everything about "rsync"

A Thread 👇

What Is Rsync?

Rsync, or Remote Sync, is a fast, versatile and free command-line tool that lets you transfer and sync files and directories to local and remote destinations in an efficient and secure way. .

Why use Rsync?

When copying or moving large no. of files using tools like cp and mv may leave ur data in inconsistent state with part of it still in the original location and part of it in the target destination in case of any interruptions.

These tools takes more time as well. Even for a plain copy, restarting the copy is less than ideal as it will recopy everything that already got copied. This same issue applies to the SSH copy utility scp.

Whereas Rsync is very fast and efficient, copying only the changes from the source and offering customization options.

How to Install Rsync?

On Ubuntu -
apt install rsync

On RedHat -
yum install rsync

Rsync syntax -

rsync options SOURCE DESTINATION

-rAllows to sync data recursively but does not keep ownership for users and groups, permissions, timestamps, or symbolic links.
-aThe archive mode behaves like d recursive mode but keeps all file permissions, symbolic links etc.

-zUsed to compress data during transfers to save space.
-bPerforms a backup during data synchronization.
-hShows the numbers in the output in a human-readable format.
-nDoes a dry run. Used for testing before the actual synchronization takes place.

-eInstructs the rsync to use the SSH protocol for remote transfers.
-progressDisplays the transfer progress during synchronization.
-vVerbose output. Displays the details of the transfer.
-qUsed to suppress the output for the rsync command and options.

SRC - define the Source directory.
DEST - define the Destination directory.
USER - define the Remote username.
HOST - define the Remote hostname or IP Address.

Synchronize Directories locally -

To copy one file to another directory on a local machine, type in the source file's full path, followed by the target destination.

# rsync -r test1/ test2

Plz Note that there is a trailing slash (/) at d end of the first argument (test1/) in d above commands.

rsync -r test1/ test2

This is necessary to mean “d contents of test1”. The alternative, w/o d trailing slash, would place test1, including d directory, within test2.

Always run rsync command in a dry run mode first.

Rsync provides a method for doing this by passing the -n or --dry-run options. The -v flag (for verbose) is also necessary to get the appropriate output.

# rsync -rnv test1/ test2

# rsync -rnv test1 test2

Copy Multiple Files Locally -

To copy multiple files with rsync, add full paths of the source files:

# rsync -v test1/abc.txt test1/def.txt test2/

Copying a File or Directory from Local to Remote Machine -

Add the IP address & d destination after d source directory. Remember to put a colon (:) after d remote host's IP address, with no spaces before the destination.

# rsync -avzP test1 root@lco-linux-worker2:/root/rsync

Please note that the target directory on the remote host was not created before. The rsync tool takes care of that and create a directory for you and copy the data.

Copy a File or Directory from a Remote to a Local Machine -

Rsync supports transferring files from a remote server to your local machine.

To pull a directory from a server, specify its IP address before the source.

# rsync -avzP root@lco-linux-worker2:/root/rsync test2/

Delete a Nonexistent Source File or Directory from Destination -

Use the --delete option to keep the source and the target in sync.

This option tells rsync to delete any file or directory at the destination if the source does not have it.

Delete Source Files After Transfer -

sometimes you may want to delete d source files after d transfer. For e.g. we no longer need weekly backups at d source after d move to backup server.

# rsync -vP --remove-source-files test1/abc.txt root@lco-linux-worker2:/root/rsync/test1

Exclude Files and Directories -

We can exclude files or directories with rsync.

To exclude a file while transferring the contents of a folder with rsync, specify the file and the relative path.

# rsync -avP --exclude 'jkl.txt' test1/ root@lco-linux-worker2:/root/rsync/test1

Exclude a Specific Directory -

Specify a directory you want to exclude (instead of a file name):

# rsync -avP --exclude 'exclude_demo' test1/ root@lco-linux-worker2:/root/rsync/test1

Exclude Files or Directories Based on a Pattern -

Use an asterisk * (wildcard) when defining a file or directory name to exclude everything that matches the pattern.

To exclude files that start with demo, run this command:

rsync -av --exclude 'demo*' sourcedir/ destinationdir/

Exclude a Specific File Type -

To exclude a specific file type, in this case .mpv, run this command:

rsync -av --exclude '*.mpv' sourcedir/ destinationdir/

Exclude Files by Size-

To exclude all files larger than 800MB, run this command:

rsync -av --max-size=800m sourcedir/ destinationdir/

Set Maximum File Size for Transfer -

To transfer files no larger than 200KB, use this command:

rsync -av --max-size=200k /home/test/dev test@lco-linux-worker2:/home/test/dev/rsync/

Set Minimum File Size for Transfer -

To skip any file smaller than 50KB, run this command:

# rsync -av --min-size=50k /home/test/dev test@lco-linux-worker2:/home/test/dev/rsync/

Define rsync Bandwidth Limit -

To set the maximum transfer speed to 100KB/s, enter:

# rsync -av --bwlimit=100 --progress /home/test/dev test@lco-linux-worker2:/home/test/dev/rsync/

Avoid Copying Source File if d Same Destination File is Modified -

Sometimes it may happen that you modify a file at d destination and do not want to let rsync overwrite it.

Use the -u option for this.

rsync -avu /home/test/dev test@lco-linux-worker2:/home/test/dev/rsync/

Retweet if you like the thread and follow me for more content.