Linode.com Forum

[caker]

http://ask.slashdot.org/askslashdot/04/02/10/2157236.shtml?tid=126&tid=95

-Chris

[Cool_blade]

I said my bit that linode is cool!

[bji]

Cool_blade wrote: I said my bit that linode is cool!

Me too ... I hope this Ask Slashdot generates some nice business for Linode.com!

[mcowger]

It just did...my previous provider, JVDS.com, just had a disk failure, and hadn't been keeping backups like they said they were! So, I demanded a refund of the last 6 months, got it, checked out the Slashdot article, and just signed up here.

How long does it take to get 'validated'?

[blahrus]

did right away for me, but I was talking with Chris . . . . just jump on the irc channel, you can find him there most of the time.

http://www.linode.com/cgi-bin/irc/irc.cgi

[bji]

mcowger wrote: It just did...my previous provider, JVDS.com, just had a disk failure, and hadn't been keeping backups like they said they were! So, I demanded a refund of the last 6 months, got it, checked out the Slashdot article, and just signed up here.


Wow, that is just terrible. I can't believe that they would say they were making backups and yet not do it.

Of course, Linode.com doesn't offer backups but at least they don't claim to do them and then not do them. It's pretty easy to do rsync backups to a home system anyway, if you need an example script for doing this let me know, mine has been working great for the past 6 months (I keep two weeks worth of incremental backups and the nightly backups take less than 5 minutes).

mcowger wrote:
How long does it take to get 'validated'?

Shouldn't take very long, especially if you find Chris on the IRC channel. Welcome to Linode.com, and best wishes!

[mcowger]

Im all setup now....would love to see that rsync script...

[mikegrb]

I use rdiff-backup to do my backups nightly. Uses the rsync library so equally quick on the transfer. I have a how-to up at http://thegrebs.com/ in the docs section.

[blahrus]

if you want to back up to a windows machine then I would take a look at unison:

http://www.cis.upenn.edu/~bcpierce/unison/

I have yet to set it up . . . I know a few people that have and read the doc's, and look just as good as any others, one plus for the windows users it does a windows build for those that want to back it up and a windows box. (others might I just don't know)

Welcome to Linode!

[bji]

mcowger wrote: Im all setup now....would love to see that rsync script...

Happy to oblige. A few caveats:

* This isn't "productized" in any way. You have to know what you are doing and you may need to modify this to work the way you want.

* This script is run from the system which is to be backed up to and it contacts the system which is to be backed up.

* In order to run this script unattended, you will need to set up keys so that the system which is doing the backup can ssh into the system which is being backed up, and as root no less. This could be a security concern because it means that having root on the backed up to machine is as good as having root on the backed up machine.

* I run this script on both systems, so that they back each other up. It is important to "exclude" the backups directory of one system from being backed up on the other, otherwise you will have "recursive" backups that will grow exponentially.

* Usage is as follows:

rsync_backup.sh [remote_root] [local_root] [excludes_file]

[remote_root] is the directory from the remote system to be backed up. I use '/' to back up everything.
[local_root] is the local directory under which the backups will be written. They are rotated so that if you are keeping 14 backups, after 14 days you will have backup.01, backup.02, ..., backup.14, with backup.01 being the oldest.
[excludes_file] is a file on the local system which lists, with one file or directory per line, the files and directories from the remote system which should not be backed up.

You can set the following environment variable also:

NUM_BACKUPS is the number of backups to keep (I back up once per day and use 14 to have two weeks' worth of backups at all times).

This script uses hard linking to eliminate redundancy of files, so each incremental backup only actually uses disk space for those files which have changed since the last backup, although each backup.XX directory will look like a complete backup.

Here is what my crontab entry for doing the backups looks like:

Code:
# At 5:13 every morning, back up mitya.ischo.com
13 5 * * *      /data/backup/rsync_backup.sh mitya.ischo.com / /data/backup /data/backup/excludes.txt


And here is what the /data/backup/excludes.txt file looks like:

Code:
/proc
/data/backup/eva.ischo.com
/data/rsync/modules
/data/share
/data/tmp


Have fun!

NOTE: I'm having some problems with the way that this forums system formats the code lines; it's wrapping some even though it doesn't show them as wrapped in the "preview". It looks like it's only some of the comment lines though that have alot of dashes in them. Be careful when/if you copy the script text into a file to fix that before trying to run it.

Code:
#!/bin/bash

echo "rsync_backup.sh started at " `date` "."

# ------------------------------- Constants ----------------------------------

NUM_BACKUPS=3

# -------------------------------- Commands ----------------------------------

MKDIR=/bin/mkdir
MV=/bin/mv
RM=/bin/rm
RSYNC=/usr/bin/rsync
SEQ=/usr/bin/seq

# --------------------------------- Paths ------------------------------------

BACKUP_ROOT="$3"
BACKUP_SYSTEM=$1
EXCLUDES_FILE="$4"
EXCLUDES_ARG=""
REMOTE_ROOT="$2"
TMP_OUT=$BACKUP_ROOT/rsync.out
TMP_TOUCH=$BACKUP_ROOT/rsync.touch

# --------------------------- Check Requirements -----------------------------

if [ -z "$BACKUP_ROOT" ]; then
    echo "Usage: rsync_backup.sh [remote_system] [remote_root] [local_root] [excludes file]"
    exit -1;
fi

if [ -z "$BACKUP_SYSTEM" ]; then
    echo "Usage: rsync_backup.sh [remote_system] [remote_root] [local_root] [excludes file]"
    exit -2;
fi

if [ -z "$REMOTE_ROOT" ]; then
    echo "Usage: rsync_backup.sh [remote_system] [remote_root] [local_root] [excludes file]"
    exit -3;
fi

BACKUP_DIR="$BACKUP_ROOT/$BACKUP_SYSTEM"
echo "Back up local root: $BACKUP_DIR"

if [ \! -d "$BACKUP_DIR" ]; then
    echo "Making directory: $BACKUP_DIR"
    $MKDIR -p "$BACKUP_DIR"
fi

# Figure out which is the newest backup
NEWEST_EXISTING=0
for i in `$SEQ 1 $NUM_BACKUPS`; do
    if [ $i -lt 10 ]; then
        if [ -d "$BACKUP_DIR/backup.0$i" ]; then
            NEWEST_EXISTING=$i
        fi
    else
        if [ -d "$BACKUP_DIR/backup.$i" ]; then
            NEWEST_EXISTING=$i
        fi
    fi
done

if [ $NEWEST_EXISTING -gt 0 ]; then
    if [ $NEWEST_EXISTING -lt 10 ]; then
        BACKUP_PREV="$BACKUP_DIR/backup.0$NEWEST_EXISTING"
    else
        BACKUP_PREV="$BACKUP_DIR/backup.$NEWEST_EXISTING"
    fi
    echo "Newest backup: $BACKUP_PREV"
else
    echo "No previous backups"
fi

NEW_NUM=$[NEWEST_EXISTING + 1]

if [ $NEW_NUM -lt 10 ]; then
    BACKUP_DEST="$BACKUP_DIR/backup.0$NEW_NUM"
else
    BACKUP_DEST="$BACKUP_DIR/backup.$NEW_NUM"
fi

echo "New backup dir: $BACKUP_DEST"

# ------------ Compose rsync args ------------

# Copy all files recursively including all file attributes, verbosely,
# and use compression over the wire, also delete any deleted files
RSYNC_ARGS="-avz --delete"

# Use ssh to the remote system
RSYNC_ARGS="$RSYNC_ARGS -e ssh"

# Exclude file, if present
if [ -n "$EXCLUDES_FILE" ]; then
    RSYNC_ARGS="$RSYNC_ARGS --exclude-from=$EXCLUDES_FILE"
fi

# Link dest, if we already have a previous rsync
if [ $NEWEST_EXISTING -gt 0 ]; then
    RSYNC_ARGS="$RSYNC_ARGS --link-dest=$BACKUP_PREV/"
fi

# Source location
RSYNC_ARGS="$RSYNC_ARGS root@$BACKUP_SYSTEM:$REMOTE_ROOT/"

# Destination
RSYNC_ARGS="$RSYNC_ARGS $BACKUP_DEST/"

# Do the rsync
$RM -f $TMP_OUT
echo "rsync command: $RSYNC $RSYNC_ARGS"
$RSYNC $RSYNC_ARGS > $TMP_OUT

# Touch the rsync directory to set the backup time
touch "$BACKUP_DEST"

tail -2 $TMP_OUT

# --------------------------------- Rotate -----------------------------------

# This function is unfortunately necessary because Linux is messing with the
# modification times of the directories on mv
function correct_mv() {
    $RM -f $TMP_TOUCH;
    touch -r $1 $TMP_TOUCH;
    $MV $1 $2;
    touch -r $TMP_TOUCH $2;
    $RM -f $TMP_TOUCH;
}

if [ $NEW_NUM -gt $NUM_BACKUPS ]; then
    # Remove number 1
    $RM -rf "$BACKUP_DIR/backup.01"
    # Renumber the others
    for i in `seq 2 $NEW_NUM`; do
        if [ $i -lt 10 ]; then
            FROM=0$i
        else
            FROM=$i
        fi
        j=$[i - 1]
        if [ $j -lt 10 ]; then
            TO=0$j
        else
            TO=$j
        fi
        correct_mv "$BACKUP_DIR/backup.$FROM" "$BACKUP_DIR/backup.$TO"
    done
fi

echo "rsync_backup.sh finished at " `date` "."
[/b]

[bji]

By the way, this is what the output of the script looks like; I get a mail like this every day and check it briefly to make sure there were no errors:

Code:
rsync_backup.sh started at  Thu Feb 12 05:13:00 EST 2004 .
Back up local root: /data/backup/mitya.ischo.com
Newest backup: /data/backup/mitya.ischo.com/backup.03
New backup dir: /data/backup/mitya.ischo.com/backup.04
rsync command: /usr/bin/rsync -avz --delete -e ssh
--exclude-from=/data/backup/excludes.txt
--link-dest=/data/backup/mitya.ischo.com/backup.03/ root@mitya.ischo.com://
/data/backup/mitya.ischo.com/backup.04/
wrote 207352 bytes  read 1092446 bytes  18178.99 bytes/sec
total size is 841411632  speedup is 647.34
rsync_backup.sh finished at  Thu Feb 12 05:14:24 EST 2004 .