Linode.com Forum

[lcannell]

Very powerful techniques posted at http://www.linode.com/forums/viewtopic.php?t=582 for managing disk images. Thank you Bill Clinton for writing this. These techniques along with the quality service I have seen to-date are really selling me on the value of Linode.

To complete this tutorial could you please post instructions for restoring this image? Thanks again.

[Bill Clinton]

Sure

But I will provide you with the instructions in a message. This way i will not be mindlessly flamed again.

Bill Clinton

[sweh]

Hmm, I'm not sure why that original thread devolved into a flame war. Odd. FWIW, Bill's "dd" solution is a valid method of backing up a partition, but there are some important notes.

Firstly, the filesystem must be quiescant; at best umounted, otherwise mounted read-only (as per Bill's examples), worst case mounted read/write but with no activity. Any writes to a filesystem being copied in this way can result in a corrupted backup. There is an important consequence to this: your root filesystem is not a good candidate for dd backups :-)

Secondaly, 'dd' style backups may not be as small as other types, since they copy the whole partition - even unused space - is copied.

Being aware of those two points, "dd" is a pretty simple backup solution.

To restore a dd backup, you simply need to do the reverse.
1) ensure the partition is UNMOUNTED.
2) ensure the partition is the SAME SIZE (or bigger) than the original backup
3) do the "dd" in reverse:
ssh user@host "dd if=ubda2.img" | dd of=/dev/ubda2
4) fsck /dev/ubda2 (just for safety)
5) mount the filesystem

Make sure you get the "if" and "of" options the right way round :-)

For what it's worth, if you plan on doing 'dd' style backups, and can afford "down time" then I would create a second OS image in some spare disk space, which has access to your original partitions (but doesn't mount them). Then you can boot into this image and "dd" your data with full confidence that the filesystems are quiescent (and can also backup your root disk safely!). In the event of a catastrophe, you can boot the alternate image again, and restore your data (as described above), including the root disk.

If you can not afford down time, then I'd recommend doing "dump" backups, since these are designed to handle hot filesystems (mounted read/write).

Personally, I use 'dd' backups to generate "gold images" of a distribution that may be rolled out to a bunch of identical machines. Especially with Solaris this is a lot quicker than an fresh install! Sun finally picked up on this and did something similar with "flash archives", which are essentially a cpio with wrappers, but in Solaris 2.5 and 2.6 days the "dd" image was a great solution :-)

Hope this helps,

[sufehmi]

sweh wrote: Secondaly, 'dd' style backups may not be as small as other types, since they copy the whole partition - even unused space - is copied.

g4u (http://www.feyrer.de/g4u/)compresses the disk image, so it arrived on the FTP server in a much smaller size.

But don't ask me how to use it to backup a Linode server, it'd be nice to be able to use it, but I have no idea how. :oops:


cheers, HS

[anderiv]

I've been using rsnapshot to keep incremental backups of my linode, and I'd highly recommend it. It keeps hourly/daily/weekly/monthly incremental backups using hard links for files that haven't changed since the last backup. This keeps the relative size of the backups very low. You can check out rsnapshot here:
http://www.rsnapshot.org/

Case study - I have a linode 128 with the default disk size:
Code: host root # df -m
Filesystem           1M-blocks      Used Available Use% Mounted on
/dev/ubd/0                5801      1194      4607  21% /
none                        61         0        61   0% /dev/shm

here's a listing of my rsnapshot folder on a server at home:
Code: [root@host2 snapshots]# ls -la
total 64
drwx------   16 root     root         4096 May 17 16:00 .
drwxrwxr-x    3 root     mp3rw        4096 May  6 17:22 ..
drwxr-xr-x    3 root     root         4096 May 17 05:00 daily.0
drwxr-xr-x    3 root     root         4096 May 17 05:00 daily.1
drwxr-xr-x    3 root     root         4096 May 17 05:00 daily.2
drwxr-xr-x    3 root     root         4096 May 17 05:00 daily.3
drwxr-xr-x    3 root     root         4096 May 17 05:00 daily.4
drwxr-xr-x    3 root     root         4096 May 17 05:00 daily.5
drwxr-xr-x    3 root     root         4096 May 17 05:00 daily.6
drwxr-xr-x    3 root     root         4096 May 17 12:04 hourly.0
drwxr-xr-x    3 root     root         4096 May 17 16:00 hourly.1
drwxr-xr-x    3 root     root         4096 May 17 16:00 hourly.2
drwxr-xr-x    3 root     root         4096 May 17 16:00 hourly.3
drwxr-xr-x    3 root     root         4096 May 17 16:00 hourly.4
drwxr-xr-x    3 root     root         4096 May 17 16:00 hourly.5
drwxr-xr-x    3 root     root         4096 May 16 00:00 weekly.0

and the disk usage of that folder: Code: [root@host2 snapshots]# du -sm .

269     .

So those 6 hourlies, 7 dailies, and one weekly are only taking up 269 megs....not bad.

Granted, I'm only backing up /home, /var, and /etc, but that's the guts of the system. I know this is completely different than backing up and restoring a disk image, but it's been effective for me, so I thought I'd share.

[adamgent]

anderiv wrote: I've been using rsnapshot to keep incremental backups of my linode, and I'd highly recommend it. It keeps hourly/daily/weekly/monthly incremental backups using hard links for files that haven't changed since the last backup. This keeps the relative size of the backups very low. You can check out rsnapshot here:
http://www.rsnapshot.org/


Looks to be similar to rdiff-backup, but packaged a bit nicer.

Will be worth looking at.

Adam

[anderiv]

adamgent wrote: Looks to be similar to rdiff-backup, but packaged a bit nicer.

Will be worth looking at.
Yeah - definitely. It only took me about 10 minutes to get it all set up and working flawlessly.

[dswartz]

I am already using it for two servers physically onsite, so it was relatively easy to install/configure the amanda client on my linode, add it to the amanda client list on my amanda server here, and have the backup run over the openvpn tunnel i had already set up. if i have to do a "bare metal" restore, i can restore the standard centos 5 image, repeat the amanda client install, and the do a restore.

[yejun]

Linode support raw disk image, I guess lvm snapshot will be option too.

[Jay]

adamgent wrote: anderiv wrote: I've been using rsnapshot to keep incremental backups of my linode, and I'd highly recommend it. It keeps hourly/daily/weekly/monthly incremental backups using hard links for files that haven't changed since the last backup. This keeps the relative size of the backups very low. You can check out rsnapshot here:
http://www.rsnapshot.org/


Looks to be similar to rdiff-backup, but packaged a bit nicer.

Will be worth looking at.

Adam
There is one SIGNIFICANT difference between rsnapshot & rdiff-backup -- rsnapshot backups 100% of every changed file -- rdiff-backup takes a binary diff of the changed files.

In the case of my employer, this makes a huge difference as we backup three-digit-gigs worth of SQL dumps where the files are ALWAYS different, but usually are similar data-wise.

--Jay

[dswartz]

i guess there is no perfect solution for everyone - we all have to find what works best for our situation. what i liked about using amanda is that i get a level-0 (full) dump initially, and then after that, level-1 incrementals (with a level-0 periodically). amanda pipes the tar or dump output through the appropriate compression program, so the backups don't take up that much space...