Backup strategies ...

My backups are simple:

1. Make daily incremental backups using rsync + hardlinks, delete any that is over 7 days old

2. Make a seperate copy of the latest backup every Sunday, (i.e. so you have a weekly as well as daily copy), and keep these for a month

This means that any time, I can restore my system to it's state in the last 7 days, or in the last 4 Sundays. The use of hardlinks means that little space is wasted (of course, it still uses more space than proper incremental systems, rdiff-backup, etc)

I use duplicity with a custom wrapper script. I do encrypted backups to an Amazon S3 store and do a full backup every 4 weeks with an incremental backup every day. I maintain the last 4 backup chains so at any given time I have 16-20 weeks of daily backups.

I keep the root GPG key that's used to encrypt the backups in an encrypted KeePass file which is then automatically distributed to all my PCs using Dropbox.

I do:

1. Minutely backup of data files to another local server and two remote ones

2. Daily full-disk backups to remote server

3. Daily beta backups to local backup device

@nfn:

What are your backup strategies? What do you suggest as a replacement for duplicity incremental (off server) backups?
While not necessarily for the faint of heart to get set up, I use bacula (www.bacula.org) as my primary system.

It's extremely flexible and powerful, though with that comes complexity, and it's UI options are somewhat limited. A minimum setup is a director on some central management host that is in charge of controlling the backups, one or more storage daemons that manage storage (tape or disk), and a file daemon on each node to be backed up. But you can operate with multiple directories, each with their own or shared storage daemons and independent or overlapping use of file daemons.

You can configure virtually any backup scheme you want (very flexible scheduling, including stuff like "first monday each month", in any combination of incremental, differential, full) with any desired retention times, reuse times for media, etc… Restorations can select any files from any given date in time that is still available on the media, and can restore to any file daemon and location on that system, even if not the original backup host.

Personally I have a NAS at home with my primary director and storage daemon, and have jobs to back up my 4-5 home machines, a business machine at home, 4 business machines at our sites, and 2 business Linodes. Oh, and two of my Dad's machines from his house in case he ends up calling me with a problem :-)

The home v. business machines have different schedules and retention periods, but in general it's daily incrementals for some period (between a week and a month), with differentials for a longer period (e.g., each week or every other week), and generally full once a month.

You do have to allow connections from the director to file daemons (though you could run your director on a Linode and just use the internal network) and connections from file daemons to the storage daemon being used to write the backup media. Bacula v3 adds TLS authentication but with 2.x I just used very limited firewall rules for the specific traffic needed.

-- David

For my linodes (mysql, web only -no mail etc)

I use rdiff-backup daily to a remote server. This in effect leaves a mirror on the backup server. I then archive these mirrors.

1st run takes a while but after that it runs very smoothly.

Hello,

How are you doing full and incremental backups? With Tar?

My scheme is:

4h rsync to external server (this is ok)

daily incremental backup to an external server (duplicity)

daily incremental backup to s3 (duplicity)

What do you suggest to replace duplicity?

What do you think about rdiff-backup or rsnapshot?

Do you suggest any script to automate the process?

Thanks

Using an rsync script to send my system off to rsync.net:

http://fukawi2.nl/index.php?page=code-rsync-backup

This script creates:

daily.0 (the most recent backup)

daily.1

daily.2

daily.3

daily.4

daily.5

daily.6

weekly.0

weekly.1

weekly.2

weekly.3

monthly.200911

monthly.200910

monthly.200909

etc etc etc

It uses hard-links so is very space efficient on the remote end.

I have websites hosted with 4 different hosting providers spread across multiple datacenters. Dozens of them and the total disk storage is approximately 30GB with nearly a million files (many sites have like hundreds of thousands of images). Plus, they are almost all dynamic websites storing data in MySQL databases, it's really hard to make a full backup.

I tar and download them to my local computer and then store them to USB hard drives. The last backup took me several full days.

:(

Anybody get any good ideas regarding my situation?

MySQL backup? Hmm. Unless you have reliable low-level means for the VM technology used (e.g. VMware), your best bet is probably to do a mysqldump to dump your MySQL databases to one or more text files (and gzip it in the process) then copy/backup that, along with whatever else you can safely backup.

Otherwise, you may not get a reliably recoverable consistent point-in-time restore by restoring the raw MySQL DB files.

@ichsie2036:

I have websites hosted with 4 different hosting providers spread across multiple datacenters. Dozens of them and the total disk storage is approximately 30GB with nearly a million files (many sites have like hundreds of thousands of images). Plus, they are almost all dynamic websites storing data in MySQL databases, it's really hard to make a full backup.

I tar and download them to my local computer and then store them to USB hard drives. The last backup took me several full days.

:(

Anybody get any good ideas regarding my situation?

For MySQL, just have a cron job that fires as part of your backup that does a mysqldump with the proper locking to maintain consistency.

Then, rather than taking several full days to pull down 30GB, just use rsync. If the files are the same (same filesize and last-modified date), nothing is sent and the file is skipped. If the files are different, in one round trip it figures out exactly what parts of the file have changed (by sending a list of checksums of parts and a separate rolling checksum for each block) and only sends the parts of the file that have changed.

The end result? If not much has changed since the last search, the time it takes to rsync is pretty close to the time it takes to do an "ls" of all the files (crazy fast).

There is also trickery you can do to create incremental backups using rsync, but in the basic operational mode it simply makes two directories the same.

It's pretty simple to use in that mode too. The arguments for the basic operation are quite similar to scp. Something like this:

rsync /home/guspaz/mydirectory guspaz@mydestinationhost:/home/guspaz/mybackupdirectory

That would take the local "mydirectory" and copy it over to "mybackupdirectory" on the remote machine; in other words it'd make it the same.

You can do the same direction from the other host, too. "source" doesn't HAVE to be a local directory. The only thing you can't do, IIRC, is have both source and destination as remote hosts.

I want to have rsync's babies.