Linode.com Forum

[ryantate]

I am having a good experience backing up my key Linode files (including all of /usr/local, /etc/, /var/log and /home) using s3sync, a ruby script that uploads files to Amazon S3, where you pay, I believe, 15 cents per gigabyte per month for storage and 20 cents per gigabyte for transfer bandwidth.

I already had an s3 account for JungleDisk, a Windows app I use to back up my home PC. You can sign up for one at Amazon.

I got s3sync set up using this tutorial:

http://blog.eberly.org/2006/10/09/how-automate-your-backup-to-amazon-s3-using-s3sync/

and the README:

http://s3.amazonaws.com/ServEdge_pub/s3sync/README.txt

Note: If you use the verbatim SSL cert provided in the README (and referenced, I believe, in the tutorial), you need to name it "f73e89fd.0", which is not mentioned anywhere but the main Amazon website thread on s3sync:

http://developer.amazonwebservices.com/connect/thread.jspa?threadID=11975&tstart=0

I have a shell script that calls s3sync on various directories, which I then invoke via cron, weekly for now. For my public web directory, I add the --public-read option, which lets me access those files from an URL that looks like:

mybucketname.s3.amazonaws.com

but this can be aliased to, for example, www2.mydomain.com, according to Amazon docs, though I have not tried this.

The only downside is that s3sync does not support putting items in at root level, there always has to be a prefix, if only a slash, so you can't make a perfect mirror of your site, www.mydomain.com/foo.mov would be mybucket.s3.amazonaws.com//foo.mov (or www2.mydomain.com//foo.mov). Hopefully this will be fixed in the future.

I still backup to my home pc using rdiff-backup as detailed elsewhere on this board. But it's nice to have redundant backups and the option of easily Webserving large files like video from Amazon's servers rather than my Linode, should I ever have the need.

[Jay]

I got mine working using s3 + fuse.

http://code.google.com/p/s3fs/wiki/FuseOverAmazon

Works like a champ on Ubuntu Hardy!

[Jay]

Okay, the fuse plugin is slow, so I rewrote myself some backup foo to work with s3.

I used the s3cmd packaged with ubuntu hardy. This script assumes you already have s3cmd installed and configured properly.

A couple of TODOs:
* Incremental backup
* Auto-pruning

I'm using this on my linode to great success. Just note you have to have enough free space for "scratch" to compress each chunk you're backing up.

I just print everything to stdout, and since I run the script in cron, it gets emailed to me.

Code:
#!/bin/bash
NOW=`date +%d-%m-%Y`
echo "====Beginning Backup for $NOW===="
echo ""
echo "Creating bucket viagraoldos-$NOW"
s3cmd mb s3://viagraoldos-$NOW

for i in home etc root; do
        echo "==Beginning Backup of /$i=="
        echo ""
        echo "Making a tarball of /$i..."
        tar cfj /tmp/$i.tar.bz2 /$i
        echo "Done!"
        echo ""
        echo "Uploading tarball of /$i to s3..."
        s3cmd put /tmp/$i.tar.bz2 s3://viagraoldos-$NOW/
        echo "Done!"
        echo ""
        echo "Deleting local copy of tarball of /$i..."
        rm -r /tmp/$i.tar.bz2
        echo "Done!"
        echo ""
        echo "==Backup of /$i completed=="
        echo ""
done

for i in backups www; do
        echo "==Beginning Backup of /var/$i=="
        echo ""
        echo "Making a tarball of /var/$i..."
        tar cfj /tmp/var-$i.tar.bz2 /var/$i
        echo "Done!"
        echo ""
        echo "Uploading tarball of /var/$i to s3..."
        s3cmd put /tmp/var-$i.tar.bz2 s3://viagraoldos-$NOW/
        echo "Done!"
        echo ""
        echo "Deleting local copy of tarball of /var/$i..."
        rm -r /tmp/var-$i.tar.bz2
        echo "Done!"
        echo ""
        echo "==Backup of /var/$i completed=="
        echo ""
done

echo "====Backup completed. Objects currently in s3 listed below.===="
s3cmd la

[egatenby]

Code:
s3cmd mb s3://viagraoldos-$NOW


You should use "s3cmd --ssl" or your AWS secret key is sent in plain text.

I have a script that uses s3sync to sync certain configured directories to a S3 bucket, then uses s3cmd to upload a tar/bz2 of other directories. It uses the --exclude features to not sync certain directories and it also auto-prunes the tar/bz2 files that are older than some number of days.

If you are interested, I will upload it. I just need to spend 5 minutes removing ugly code and important info like my access ID and secret key :)

--Eric

[tronic]

ryantate wrote: Note: If you use the verbatim SSL cert provided in the README (and referenced, I believe, in the tutorial), you need to name it "f73e89fd.0", which is not mentioned anywhere but the main Amazon website thread on s3sync:

http://developer.amazonwebservices.com/connect/thread.jspa?threadID=11975&tstart=0


You can do this to generate appropriate <hex digits>.0 for any cert by doing, via sh/bash:

Code: $ HASH=`openssl x509 -noout -hash -in ${CERT}`
$ ln -s ${CERT} ${HASH}.0
Where CERT is filename of the SSL certificate itself.