Linode.com Forum

chrisnolan · Senior Newbie Joined: 02 Jun 2006 Posts: 17

MaineCoon, many thanks for this. This is the S3 backup solution that I've been searching for.

Works very very nicely here on CentOS 5 - the only slightly tricky bit is getting all the dependencies installed for duplicity as the CentOS duplicity RPM is way out of date.

memenode · Senior Newbie Joined: 20 Nov 2008 Posts: 17

I followed the above guide, but running the backup script results in this:

freedom_is_chaos · Senior Member Joined: 12 Sep 2008 Posts: 166

Wild Guess: If your /boot is empty it is because it probably unmounted after a successful boot.

I found this in a wild google search:
http://osdir.com/ml/sysutils.backup.duplicity.general/2008-08/msg00036.html
https://bugs.launchpad.net/ubuntu/+source/backupninja/+bug/126417

Common thing is there seems to be a problem with duplicity.

poetics5 · Senior Newbie Joined: 23 Oct 2008 Posts: 13

this tut should get you up and running with s3 + rsync, no gui needed

http://el-studio.com/article/jungledisk-linux-backups

memenode · Senior Newbie Joined: 20 Nov 2008 Posts: 17

Thanks guys. Sorry for not responding earlier. I kinda just dropped this case for the time being and went on with other things, but I appreciate your help.

If duplicity is indeed buggy I'm tending towards not using it at this point. Jungledisk option seems good, but I dislike the fact that it's proprietary.. I guess though if no better option is available I might go with it.

Thanks again.

mnordhoff · Senior Member Joined: 03 May 2008 Posts: 179

Kev@hearSAY · Newbie Joined: 25 Feb 2008 Posts: 2

So I noticed that my alerts have been going haywire for the last week, and then I looked at my AWS account. Far too many GB used. Wow. I took a look, and apparently last week sometime duplicity started making full backups every time it runs on BOTH my linodes. I have no idea what I changed that caused this, but it must have been something I've changed on both linodes, and I'm racking my brain trying to figure it out. Does anyone have an idea what might be causing this?

Kev@hearSAY · Newbie Joined: 25 Feb 2008 Posts: 2

Update: It's not random. More than half the storage space used in my AWS account for duplicity backups was filled today. Today is the first of a new month, the first time a new month has occurred since I installed duplicity. Hmm... I know I've got it set to make a full backup on the first of every month, but I didn't realize it was going to do it during ANY hour of that first day that backups are scheduled. Oh well.... I suppose I'll delete all of today's backups except for the most recent.

memenode · Senior Newbie Joined: 20 Nov 2008 Posts: 17

cboshuizen · Senior Newbie Joined: 07 Jan 2009 Posts: 10

I had the same problems as mentioned by memenode, and I fixed them by checking three things: install location, permissions, and packages installed.

For install location, I picked a location for the root or backup user to run the script from. I chose /etc/backup, and performed all of the steps there.

For permissions, I made sure that I created all the necessary files, including the gpg key, with the correct user. This seemed to be an important step.

I also chown'd everything to the correct user.

Then, I made sure I installed the correct packages. It seemed to me that missing dependencies were to blame, so I followed these directions:

cboshuizen · Senior Newbie Joined: 07 Jan 2009 Posts: 10

I just mentioned some problem with the report in the log, stating that 0 files were changed/added when in fact there had been.

To check what is going on, I created some more scripts to do listings and status reports, and the listings do match my changes to the file system.

I did some test backups, adding/changing/removing files in between, and could successfully restore any file from any point in time. Thus, it is all working despite the incorrect messages in the log.

*shrug*

poetics5 · Senior Newbie Joined: 23 Oct 2008 Posts: 13

hybinet · Senior Member Joined: 02 May 2008 Posts: 445

phvt · Senior Member Joined: 28 Feb 2005 Posts: 74

I believe the 'FuseOverAmazon' s3fs is the most advanced of the s3fs' out there, but I haven't tested any of them. s3fs is quite proprietary in file format because it creates a virtual block device in the bucket.

I've been testing JungleDisk (JD) on my Linode and the two main issues are:
(1) lots of little files cause a performance bottleneck because each must be posted in its own transaction with S3. JD does this file-based uploading because that's the only way S3 will guarantee "consistency".
(2) The JD cache isn't yet available offline, so files in the JD mount are only visible if there is connectivity to S3. It was designed this way to ensure cache validity when the bucket is shared between multiple machines..

Another downside of JD is that for Linux CLI use, you'll need to download the USB version and create the XML configuration file using it (on a Mac or PC), then paste the config to your Linode. There's no published spec for the XML file but easy to edit by hand after creating the initial file.

JungleDisk can create and use compatibility buckets which are compatible with other software (but S3 doesn't have any "standard" for this, anyway). The biggest advantage of JD is that it can encrypt everything before it leaves your machine, which obviously makes those buckets incompatible with other software. There is an open source tool to access your JD encrypted bucket data.

We need both fast and cheaper storage, and backups that are easy to verify and restore. Handling both with the same system might not be optimal (see how Amazon separates EBS from snapshot backups to S3). I really look forward to seeing what Linode comes up with because everything else here rocks!