Using Rsync¶

This Howto explains how you can use SSH and Rsync to copy files from a remote server to your local machine fast and comfortably.

Updated on May 08, 2017

The Code ¶

I’m showing part of a script that I usually use to copy a web installation from a remote server to my local machine. Let’s start with the code right away:

#! /bin/bash

# settings
useratdomain=user@remote.com

# with / at the end of srcdir !!!
srcdir=/home/user/public_html/

# with / at the end of destdir !!!
destdir=/home/user/projectname/public_html/

if ((1)); then true;
    dryrun=--dry-run
else false;
    dryrun=
fi

# NO slash / at the end of workdir!
workdir=get-`date '+%Y-%m-%d'`

if ((1)); then true;
    echo "`date '+%Y-%m-%d %H:%M:%S'` Started."
fi

if ((1)); then true;
    echo "`date '+%Y-%m-%d %H:%M:%S'` Prepare log files in '$workdir'."
    log_v_file=$workdir/rsync_temp_log_v_file.txt
    log_v_stdout=$workdir/rsync_temp_log_v_stdout.txt
    rm -rf $workdir
    mkdir $workdir         &> /dev/null
fi

if ((1)); then true;
    echo "`date '+%Y-%m-%d %H:%M:%S'` Get public files from remote"
    rsync \
        $dryrun \
        -rlt  -v --safe-links --delete --log-file=$log_v_file \
        --exclude=/.htaccess \
        --exclude=*NOT_VERSIONED* \
        --exclude=/typo3conf/localconf.php \
        --exclude=/typo3conf/LocalConfiguration.php \
        --exclude=/typo3temp/ \
        -e 'ssh' $useratdomain:$srcdir $destdir \
        &>$log_v_stdout
fi

if ((1)); then true;
    echo "`date '+%Y-%m-%d %H:%M:%S'` Finished."
fi

In this example we mirror the remote folder srcdir to the local folder destdir. Beforehand I have uploaded my public key to the remote server so it will accept the SSH connection for useratdomain, which is user@remote.com. The slash at the end of both srcdir and destdir is important as otherwise you may end up with srcdir being copyied as a folder into destdir, In that case you see something like .../public_html/public_html/.

You can turn on and off each little part of the script by either writing ((1)) or ((0)).

The Rsync Command ¶

I prefer splitting the command into several lines. A backslash at the end tells the shell that there will be a continuation line. Add more exclude lines if you have more things to exclude. We are using these options:

`-r`	Copy recursively, with subdirectories.
`-l`	Copy symlinks as symlinks.
`-p`	Preserve permissions.
`-t`	Preserve modification times.
`-v`	Increase verbosity.
`--safe-links`	Ignore symlinks that point outside the source tree.
`--delete`	Delete extraneous files from destination dirs. Warning: Pay attention to provide correct paths! The `--delete` option turns on mirroring which means that files and folders in your destination directory can be deleted.
`--log-file`	Have rsync “log what we’re doing to the specified FILE”.
`--exclude`	Exclude files matching PATTERN.
`-e`	Specify the remote shell to use.

Updated on Nov 17, 2014: More options recommended:

`-z`	Compress file data during the transfer.
`--progress`	Show progress during transfer.
`-e`	Isn’t necessary sometimes.

Explaining ‘true’ and ‘false’¶

Writing:

if ((1)); then
    dryrun=--dry-run
else
    dryrun=
fi

would do the job. But I prefer to add ‘true’ or ‘false’ after ‘then’ and ‘else’ as in:

if ((1)); then true;
    # do something
else false;
    # or do this
fi

The sole reason for doing so is that there won’t be a syntax error if there isn’t a payload in the block and all lines are commented out – as in this example. Any command would do. But ‘true’ and ‘false’ seem perfectly suited and do not disturb at all.

Rsync Examples ¶

# recurse, perms, links, times (better use -a!)
rsync -rplt -v --safe-links --delete \
   /media/marble/LinuxData200        \
   marble@192.168.1.203:/media/marble/BackupThinkbook

Enjoy!¶

HTH - I hope this helps. What “tricks” are you using?

More ¶

Compare Directories ¶

Use rsync for a comparison between local and remote: Do a simulation (-n, –dry-run) only, as if you were archiving (-a, –archive), comparison based checksum (-c. –checksum), output a change-summary for all updates ( -i, –itemize-changes), as if you were mirroring to destination (–delete):

# compare, dry-run
rsync -ain --delete /local/path/  user@domain.de:/remote/path/

# compare by checksum, dry-run
rsync -acin --delete /local/path/  user@domain.de:/remote/path/

# compare by checksum, dry-run, with special params for ssh
rsync -acin --delete /local/path/  -e "ssh -i /home/user/.ssh/sshkey" user@domain.de:/remote/path/

#

As blueprint:

# ==========================
# compare
# --------------------------

useratdomain=me@server-domain.com

# with / at the end!
srcdir=/home/me/local/path/

# with / at the end!
destdir=/home/there/remote/path/

rsync -ain --delete "${srcdir}" "${useratdomain}:${destdir}"

# change direction:

rsync -ain --delete "$useratdomain:$destdir" "$srcdir"

#

Exclude some:

rsync -ain --delete "${srcdir}" "${useratdomain}:${destdir}" \
   --exclude typo3temp/ \
   --exclude typo3/ \
   --exclude t3lib/
#

Show lines for non identical items only:

rsync -ain --delete "${srcdir}" "${useratdomain}:${destdir}" \
   --exclude typo3temp/ \
   --exclude typo3/ \
   --exclude t3lib/ \
   | grep -v "\.\.\.\.\.\.\.\.\."
#

From the rsync manpage: -i, –itemize-changes ¶

Search the manual:

man rsync | less -p "--itemize-changes"

Find this:

Requests a simple itemized list of the changes that are being made to each file, including attribute changes. This is exactly the same as specifying --out-format='%i %n%L'. If you repeat the option, unchanged files will also be output, but only if the receiving rsync is at least version 2.6.7 (you can use -vv with older versions of rsync, but that also turns on the output of other verbose messages).

11 letters cryptic output ¶

The “%i” escape has a cryptic output that is 11 letters long. The general format is like the string YXcstpoguax, where Y is replaced by the type of update being done, X is replaced by the file-type, and the other letters represent attributes that may be output if they are being modified.

Type of update ¶

The update types that replace the Y are as follows:

<: A < means that a file is being transferred to the remote host (sent).

>: A > means that a file is being transferred to the local host (received).
c: A c means that a local change/creation is occurring for the item (such as the creation of a directory or the changing of a symlink, etc.).
h: A h means that the item is a hard link to another item (requires –hard-links).

.: A . means that the item is not being updated (though it might have attributes that are being modified).
*: A * means that the rest of the itemized-output area contains a message (e.g. “deleting”).

File type ¶

The file-types that replace the X are:

f  for a file, a
d  for a directory, an
L  for a symlink, a
D  for a device,  and a
S  for a special file (e.g. named sockets and fifos).

Attributes ¶

The other letters in the string above are the actual letters that will be output if the associated attribute for the item is being updated or a ”.” for no change. Three exceptions to this are:

(1) +: a newly created item replaces each letter with a “+”,
(2) spaces: an identical item replaces the dots with spaces, and
(3) ?: an unknown attribute replaces each letter with a ”?” (this can happen when talking to an older rsync).

The attribute that is associated with each letter is as follows:

.: A . means no change.
c: A c means either that a regular file has a different checksum (requires --checksum) or that a symlink, device, or special file has a changed value. Note that if you are sending files to an rsync prior to 3.0.1, this change flag will be present only for checksum-differing regular files.
s: A s means the size of a regular file is different and will be updated by the file transfer.
t: A t means the modification time is different and is being updated to the sender’s value (requires --times). An alternate value of T means that the modification time will be set to the transfer time, which happens when a file/symlink/device is updated without --times and when a symlink is changed and the receiver can’t set its time. (Note: when using an rsync 3.0.0 client, you might see the s flag combined with t instead of the proper T flag for this time-setting failure.)
p: A p means the permissions are different and are being updated to the sender’s value (requires --perms).
o: An o means the owner is different and is being updated to the sender’s value (requires --owner and super-user privileges).
g: A g means the group is different and is being updated to the sender’s value (requires --group and the author‐ ity to set the group).
u: The u slot is reserved for future use.
a: The a means that the ACL information changed.
x: The x means that the extended attribute information changed.

One other output is possible: when deleting files, the “%i” will output the string “*deleting” for each item that is being removed (assuming that you are talking to a recent enough rsync that it logs deletions instead of out‐ putting them as a verbose message).

Moving files ¶

# demo: move /tmp/demosource to my home dir at me@192.168.7.33

# settings 1:
srcpath=/tmp
movedfolder=demosource

# settings 2:
dest=me@192.168.7.33:$movedfolder

# if you don't trust the (binary) integrity of dest:
# use --checksum to force a binary compare
# BOTH source and destination have / at the end!
rsync -av --checksum $srcpath/$movedfolder/ $dest/

# to remove extraneous files from $dest
rsync -av --delete $srcpath/$movedfolder/ $dest/

# remove copied source files
rsync -av --remove-source-files $srcpath/$movedfolder/ $dest/

# remove empty source dirs:
# now WITHOUT /!
find $srcpath/movedfolder -type d -empty -delete

Using Rsync¶

The Code ¶

How does it work?¶

The Rsync Command ¶

Explaining ‘true’ and ‘false’¶

Rsync Examples ¶

Enjoy!¶

More ¶

Compare Directories ¶

From the rsync manpage: -i, –itemize-changes ¶

11 letters cryptic output ¶

Type of update ¶

File type ¶

Attributes ¶

Moving files ¶

Previous topic

Next topic

Tags

Archives

Languages

Recent Posts

This Page

Using Rsync¶

Quick search

Previous topic

Next topic

This Page