S3 /Cloud Connections

There's been quite a few posts about it, use the search to dig up a few.

The Linode Wiki has a few posts from other people that hooked S3 to their Linode:

http://www.linode.com/wiki/index.php/Ba … ith_s3sync">http://www.linode.com/wiki/index.php/Backupswiths3sync

http://www.linode.com/wiki/index.php/S3fs

I haven't used it myself, though.

@jamiedolan:

HI;

Has anyone used S3 or other cloud solutions in combination with your linode, either for live (less critical) storage or backup?

Are any of the linode datacenters close enough to one of the cloud networks that latency isn't a major issue?

Thanks

Jamie

I back up all of my photos and videos from my digital cameras to S3, using my own custom program that I wrote. It works fine for this.

I mount an S3 bucket using s3fs to serve up my photo gallery images. It does not work very well for this; mostly because s3fs is a really poorly implemented filesystem, and also because the photo gallery software I use (gallery2) is not very frugal with file reads and it ends up hitting files that it really doesn't need which results in s3 fetches that slow my photo gallery site down tremendously. There are more sophisticated ways to combine gallery2 with S3 but for reasons I won't go into they won't work for me.

In short: I recommend S3 for file storage; I don't recommend it for 'live files'. Also if you have an application that can just link to files already stored on S3, then you get the best of both worlds; but your application has to support that. Mounting an S3 bucket as a filesystem is possible and would provide the ultimate in flexibility, but current options for that mostly suck.

I don't use S3 in combination with my Linodes, but I do use cloudfront to serve static content. I have a lot of images on pages and the average page load time is now 1/3 what it was before I began using cloudfront. I use my Linode as the origin rather than S3. Cloudfront is fantastic.

@bji:

@jamiedolan:

I recommend S3 for file storage; I don't recommend it for 'live files'. Also if you have an application that can just link to files already stored on S3, then you get the best of both worlds; but your application has to support that. Mounting an S3 bucket as a filesystem is possible and would provide the ultimate in flexibility, but current options for that mostly suck.

Thanks everyone for the responses. You confirmed what I had gathered from other posts, that there is a lack of a really clean /stable means of mounting S3. I'm not sure if Amazon even has any cleaner way of mounting it on S2.

I'm going to need a lot of space for jpeg's for a upcoming project, I'll have to see if we can build the program so that it put the images right onto a CDN when it imports them(importing data and jpegs from several api's). Then I'll need to have the site link directly to the images on the CDN, which I suppose really make much more sense anyway than trying to mount the CDN and serve it off a linode; I just figured the traffic would be low enough in the beginning that it wouldn't matter too much.

I'm pretty sure that the size (of the jpegs) is just going to be prohibitive for a linode, especially as the collection grows. The database it's self will end up being big enough; I'm sure it will do just fine on the lindoe.

I'm going to play around with that back-up script. With the amazon backup scripts I looked at before, it didn't seem like they were supporting incremental back-ups, which is obviously a huge waste of resources. I see from the link posted how you can run it with rsync, so that should solve the problem. I really like keeping good backups of everything, and I've heard only good things about using the S3 network for reliable backups.

Thanks for the help.

Jamie