Backups with s3sync

From LinodeWiki

Jump to: navigation, search

Contents

[edit] Create an amazon s3 account

  • Visit Amazon's S3 Site
  • Create an Account
  • Enable S3
  • Hover over the "Your Web Services button," go to "AWS Access Identifiers"
  • Grab your Access ID and Secret Key, store them somewhere

[edit] Get s3sync

  • Install ruby >= 1.8.3 (I think, maybe 1.8) if you don't already have it 
 sudo apt-get install ruby
  • Install openssl for ruby 
 sudo apt-get install libopenssl-ruby
  • Download s3sync 
 cd ~
 wget http://s3.amazonaws.com/ServEdge_pub/s3sync/s3sync.tar.gz
  • Put it somewhere (I chose /usr/local/s3sync) 
 cd /usr/local
 sudo tar xzfv ~/s3sync.tar.gz

[edit] Configure s3sync

  • Copy example config to config file 
 cd /usr/local/s3sync
 cp s3config.yml.example s3config.yml
  • Edit s3config.yml, put your own Keys in there
  • Set the ssl_cert_dir to /etc/ssl/certs

[edit] Get SSL certs

  • I followed the instructions in README.txt
  • download CA certificates 
 sudo apt-get install ca-certificates

[edit] Create bucket

  • pick a name for your bucket that you'll remember
  • cd /usr/local/s3sync 
 ./s3cmd.rb -s createbucket bucketname

[edit] Now you can sync

  • Do a dry run: 
 cd /usr/local/s3sync;
 ./s3sync.rb -snrv --delete /etc/ bucketname:etc
  • If that doesn't work, try this: 
 s3sync -snrv --delete /etc/ bucketname:etc
  • If it looks good, remove the -n (dry run) 
 ./s3sync.rb -srv --delete /etc/ bucketname:etc
  • You can use the the JeS3t applet to verify what you uploaded: 
 http://jets3t.s3.amazonaws.com/applets-jets3t-0.5.0/jets3t-cockpit.html  It's a java app.
 Go to the "Direct Login" tab and put in your access id and secret key.
  • Rinse and repeat with other directories you wish to backup.

[edit] Cron it

I created a simple script and threw it in /etc/cron.weekly/s3backup 

#!/bin/bash

EXCMDS="-vs"  # you can add -n in here for testing, remove s for non-ssl, remove verbosity, etc
SENDMAIL=/usr/sbin/sendmail
EMAIL=myuser@mydomain

cd /usr/local/s3sync

echo -e "To: ${EMAIL}\nSubject: s3backup results\nContent-type: text/plain\n\n" > /tmp/s3backup.log

pg_dumpall -Ucdmoyer | gzip > /opt/www/db/pg_dumpall.gz  #/root/.pgpass is needed for this
./s3sync.rb  ${EXCMDS} --delete -r /etc/ bucketname:etc >> /tmp/s3backup.log
./s3sync.rb  ${EXCMDS} --delete -r /opt/ bucketname:opt >> /tmp/s3backup.log
./s3sync.rb  ${EXCMDS} --delete -r /var/backups/ bucketname:var/backups >> /tmp/s3backup.log

 cat /tmp/s3backup.log | ${SENDMAIL} "${EMAIL}"

[edit] Notes & Cost

This was originally done on Ubuntu Gutsy Gibon, so your mileage may vary.

Things I did:

  • backed up /etc, deleted it, did it again
  • backed up 700M in /opt
  • backed up a couple Megs in /var/backups
  • deleted stuff from my home directory, taking /opt back to 300M
  • ran s3backup several times for testing.

Total Cost: $0.29 transfer, and it will cost me $0.15 at the end of the month. It seems to cost a bit less than a penny each time I sync.

Retrieved from "http://www.linode.com/wiki/index.php/Backups_with_s3sync"
Personal tools