Perpetual rsync


Automatically resume rsync over SSH after broken connection
Create a file in /bin/prsync


#!/bin/bash
### COPY IN /bin/prsync - chmod 777 ###
# GitHub https://gist.github.com/pwenzel/4182838
# Extended by Gulshan Beejan - Persistent RSYNC# Make sure remote system has ssh key pairsif [[ -z $1 ]] ; then
echo "Persistent RSYNC Usage: prsync SOURCE DESTINATION"
exit
fi
if [[ -z $2 ]] ; then
echo "Persistent RSYNC Usage: prsync SOURCE DESTINATION"
exit
fiwhile [ 1 ]
do
rsync -avz --partial $1 $2
if [ "$?" = "0" ] ; then
echo "prsync completed normally"
exit
else
echo "==== prsync failure. ==== Backing off and retrying in 30 seconds..."
sleep 30
fi
done


chmod +x /bin/prsync

then you can run it as:

prsync mysource_file.tar user@destination_ip:/path/to/file

Read More

How-to: Local Backup with rsync


A backup using the rsync command can be very useful because it:

Takes less time to replicate

rsync will replicate the whole content between the source and the destination directory only once. Consecutive rsync’s will transfer only the changed blocks or bytes, which makes it very fast

uses Less bandwidth usage

rsync will use compression of data block by block at the sending end and decompress at the receiving end

is Secure

rsync uses ssh protocol during transfers and hence allows encryption of data.

The script broken down

I will create a script which will backup my current web directory /data to the same server on a disk mounted at /backup. The script if broken down, can be viewed as:

rsync -azvu –progress /data/ /backup

wher options

  • z is for compress mode
  • v is for verbose mode
  • a is to preserve symbolic links, permissions, timestamp and to be recursive
  • u is to preserve unmodified files at the destination
  • progress is to show the progress during transfer

An extract of the transfer start is:

sending incremental file list
data/
data/log/
data/log/gulshan.beejan.log
8713 100%    0.00kB/s    0:00:00 (xfer#1, to-check=1004/1010)
data/lost+found/

ending with:

sent 284122142 bytes  received 230478 bytes  1914832.46 bytes/sec
total size is 379003084  speedup is 1.33

Now, if I perform a second transfer, the result at the end will be much faster. Here’s a extract of that transfer:

[root@sdb backup]# rsync -avzu –progress /data /backup
sending incremental file list

sent 278200 bytes  received 1778 bytes  50905.09 bytes/sec
total size is 379003084  speedup is 1353.69

 The script

Let’s convert that into a nice and clean script, shall we?

Create a file in

vi /scripts/localrsync.sh

Paste the following:

#!/bin/bash

# declare variables

SOURCE_DIR=/data

DESTINATION_DIR=/backup

#rsync command, make sure the user you run this file as has the required permissions to the source/dest folder

#we dont need progress or verbose, this runs in background mode

rsync -azu  $SOURCE_DIR $DESTINATION_DIR

Chmod 777 the sh file

chmod 777 /scripts/localrsync.sh

Test the file

sh /scripts/localrsync.sh

[root@sdb ~]# sh /scripts/localrsync.sh
sending incremental file list

sent 278200 bytes  received 1778 bytes  62217.33 bytes/sec
total size is 379003084  speedup is 1353.69

Add it to crontab,

 crontab -e

Add this line at the end to run it daily, every 2AM

*    2    *    *    *    sh /scripts/localrsync.sh

 

Read More