failover between data centres

It is by no means trivial to do what you desire, but I can offer some tips to point you in the right direction.

1. You'll need to use DNS-based failover. When the Japan datacenter goes down, you'll need to update your DNS records so that your hostname resolves to the US IP address instead. Since you want the failover to happen within 10 minutes, you'll need to set a TTL on your DNS record no greater than 10 minutes. For this to happen automatically, you'll need the DNS to be updated dynamically. Conventional DNS servers like BIND support dynamic DNS; newer DNS services like Amazon's Route 53 have an API for making updates to DNS records. You'll need a monitoring process running somewhere that monitors the Japan datacenter and updates the DNS when it goes down. This is rather tricky to do correctly, since you want to avoid failing over unnecessarily if there's only a momentary outage or if you have a network partition (where both the Japan data center and the monitoring process are running but can't communicate with each other).

2. You'll need to continuously replicate your database from your Japan server to your US server. MySQL and PostgreSQL both have replication support builtin. To make life easier, you should put everything in a single database. For example, if you're saving stuff to the filesystem (such as images), use a database table instead, since it will be far easier to do replication if there's only one thing to replicate. You should use asynchronous replication instead of synchronous replication, since this provides better performance, with the downside that you'll lose any data that was being written to the database in the moments before Japan goes down (which you've said is OK). NOTE: replication between data centers is only feasible for databases which aren't changed too often. If your webapp is constantly modifying your database, the bandwidth requirements will be too great. It will saturate your bandwidth, causing the replication to fall behind, and/or run up your bandwidth bill from Linode.

3. When Japan goes down, you'll need to promote your US database to master so you can start writing to it. It is VERY VERY hard to do this automatically and correctly. You do not want to accidentally promote your US database to master while Japan is still up; this will cause the two databases to diverge. Remember, because of network partitions, just because your US server can't contact your Japan server doesn't mean that Japan is down.

4. In order to spin up additional Linodes on the fly, you should use configuration management (Puppet, Chef, etc.) to provision your servers.

I would strongly recommend against trying to do failover automatically, since it's very difficult for a computer program to correctly decide whether failover is appropriate or not. It's better to "wear a pager" (i.e. set up monitoring that will wake you up at night if necessary) and initiate the failover yourself if you decide it's necessary. Write a script to do the failover so it's quick. But, I could see why this wouldn't work for you, since a major disaster in Japan could prevent you from initiating the failover.

All that said, maybe you should just put your Linodes in the US. Latency between Fremont and Tokyo is ~127ms. For most types of webapps that's not going to be noticable. You can always serve your static files from a Linode in Japan (or a CDN) to get a slight speed boost for Japanese users.

With any failover plan, you need to decide how much time you can allow between SITE1 going down and SITE2 coming up, and how often SITE1 updates the data on SITE2.

Those factors will determine the best method to use.

Hard to offer more details without knowing what site you're trying to setup the failover (and what Distro/Apps/etc it uses, how often it's content/userbase changes, etc).

Easiest (and cheapest) is set Site2 to be independent. Setup rsync to mirror SITE1 to SITE2 nightly. Advertise on SITE1 the URL for SITE2 with the notice that if SITE1 goes down, your users are to check SITE2. Although easy, this method has several pros and cons - once again, hard to tell without knowing your SITE1 details.

I use this on a few sites, using DOMAIN.COM for SITE1 in DC1 and DOMAIN.NET for the backup SITE2 in DC2. Easy for the users to remember, plus I can do maintenance on SITE1 with little or no warning (I just wait for the latest sync to SITE2 to happen then pull the plug on SITE1 while I work on it). Main downside is that new users/vistors might be confused, but for the sites I use it on, we're willing to risk it for the ease of maintaining a Duct Tape and Lego Failover system.

Thank you to both of you for your extensive comments and input.

I do take on board the suggestion that one not try to over-automate, but let a human make the final decision. I also hear what you are saying about keeping it as simple as possible and only replicating one thing. (In fact I probably won't keep images in the database, but the basic point you make is a good one.). I currently write a lot of logging stuff to the database that would not need to be replicated, so I really need to start thinking about putting that into a separate database and just syncing the necessary stuff. (I have not got into MYSQL replication yet - I wonder if it is possible to exclude certain tables when replicating…)

The latency thing is - mmm - vanity in part, I suppose. A highly ajaxy web-app just feels so much better with latency at 20ms than it does with latency at 120ms.

Just FYI I am running Centos 6.5. Just a basic LEMP stack, though most of the app runs client side in JS. Wondering whether I should get up to speed with Ubuntu, as Docker looks potentially useful, but seems slightly more mature (or slightly less bleeding edge) in Ubuntu then Centos.

Once again, thanks.

Nick