What's best way for maintenance message when server is down?

If you're going to be running another set of servers to fail over to so you can show a maintenance page, why not have them serve your actual site. People who want high availability generally use load balancing and health checks to set it up. I'm biased, but I'd recommend NodeBalancers:

https://library.linode.com/nodebalancers

• Les

On Linode, I've got about 11 servers, and on each of those ranges from several to say 40 websites. I assumed for Load Balancing I would need a second server for each server I want to mirror? So I dont want to invest in that at the moment, it will triple my costs I think.

Can IP Failover be used for this? Without using Heartbeat and Pacemaker which this page implies is being used:

https://library.linode.com/remote-acces … p-failover">https://library.linode.com/remote-access#sph_configuring-ip-failover

I cant find much info on what IP Failover is, only this configuration page.

I would not want a load balancer/mirror, I would just want a second server to serve a default page only when the first server is down, and never any other time.

Doing that requires having a second server, in which case you might as well take the aforementioned course and actually serve your sites from the second server.

The IP failover documentation is short because it's very simple: you enable for the IPs you want, on the Linodes that should also be able to use those IPs. After that, you can bring up those IPs on the Linodes you chose. We don't fail over the IPs automatically, and traffic only goes to a single Linode (whichever one you've brought up the IP on).

• Les

I would think mirroring multiple sites from multiple servers on one server may be a bit more tricky that just having a default maintenance page served for all sites, so I wanted to start at just a default maintenance page but will certainly look into having a mirrored site once I get this first step working.

How does one "bring the ip up" on a server? If Linode does not automatically repoint the main Ip to the failover server how do I do it?

Thanks

amityweb,

If you're running 11 Linodes, just redirect the DNS for all the sites on the Linode you're going to be performing maintenance on to a 'Maintenance' page on one of your other servers. If you're using Apache just set up a vhost file that has all the domains in it and point them to a generic 'Currently Unavailable Due To Maintenance' page (I'm not familiar with Nginx, but I'm sure it has the same option).

> just redirect the DNS for all the sites
Well, that can be done yes, but I personally don't think thats a good way of doing because DNS changes can take a while, there will be no control over who sees what site on what server when. TTL can be reduced but that requires at least 24 to 48 hours planning. As stated one server has 40 domains, that will take ages. Some customers control their own DNS records which would be a pain to manage. So I don't think DNS changes is a good way to control switches to different servers, so ideally I am looking for an easier way.

The Apache/Nginx vhosts wont work, as that server would be down. Any IP repointing/redirection would need to be before traffic hits the server thats going down.

Failover environments don't necessarily need to be as powerful (or diverse) as the primary. As long as storage space isn't the concern, the failover environments can be configured for lower performance and greater consolidation.

Due to the complexity with database and file syncing (i.e. if the database changes on the failover server we need that synced back) then I am fine with a maintenance page stating site is down for maintenance and to checkback soon (just SOMETHING other that page not found).

What I dont understand at the moment though, due to lack of documentation, is if I do choose a failover server, how do I get that to kick in instead of the main server, because its stated above failover does not automatically work when the main server goes down. Sounds like I just need to switch the main servers IP addresses to point to a new server I have with he maintenance pages.

Thanks

I set up DNS Made Easy's failover option, and I fail over my Linode in Dallas to one in Fremont. DNSME will allow extremely short TTLs (IIRC I use 5 minutes and never had a problem) and they monitor very quickly, though local DNS caching is a pain I can't get around with this particular setup. It's not perfect, but as akerl noted I can mirror the site with a maintenance page that looks official and seamless even though it's on a completely different server, and I can even offer some limited services to my visitors. For simplicity I replicate MySQL as master/slave, so the slave linode is always current, though I don't allow DB writes so when the main server comes back online DNSME switches back automatically. Been doing this for years, and it's great piece of mind. Does wonders for the stress level, particularly when you've experienced an extended outage (not at Linode in my case).

I really wont be pursuing DNS changes for maintenance (the server upgrade took 30 minutes)… I control most domains but not all, I would have to go through customers to arrange DNS changes (they set the A record), and even then some recent upgrades involved over 50 domains, I'm really not going to change DNS for 50 domains many of which I have to arrange with customers. It really is in my opinion the worse way for a sort of failover (unless you know a server will be down for a very long time). I'm looking for a quick/simple way to point the original server Ip to a different one. Amazon have elastic IPs you can reassign easily.

I would like to look into Linodes IP Failover, I just cant see how to to get the "down" server IP to point to the failover server. I tried it, by choosing a failover server, then shutting down the main server, and restarting the failover one even, and added the IP as per static networking guide, but it didn't work. I cant see where to point the downed server IP to the failover one. If anyone knows how please could they let me know, documentation says nothing about this.

Thanks

@amityweb:

I would like to look into Linodes IP Failover, I just cant see how to to get the "down" server IP to point to the failover server. I tried it, by choosing a failover server, then shutting down the main server, and restarting the failover one even, and added the IP as per static networking guide, but it didn't work. I cant see where to point the downed server IP to the failover one. If anyone knows how please could they let me know, documentation says nothing about this.
Did Linode Support have any answers regarding the failover not working?

@amityweb:

I control most domains but not all, I would have to go through customers to arrange DNS changes (they set the A record), and even then some recent upgrades involved over 50 domains, I'm really not going to change DNS for 50 domains many of which I have to arrange with customers. It really is in my opinion the worse way for a sort of failover (unless you know a server will be down for a very long time). I'm looking for a quick/simple way to point the original server Ip to a different one. Amazon have elastic IPs you can reassign easily.

I understand the part about you not controlling all domains (and sorry I missed that), but with respect to it being the [worst] method to fail over, I think you're missing that with DNSME the failover is automatic. You define the failover IP address in advance, and your server is monitored 24x7. If it goes down, the DNS records switch within a minute or two with no action on your part. When your server comes back up, the DNS switches back, and you don't have to do anything. (or you can tell them to continue pointing to the failover server if you want.)

Just clarifying that for anyone else who reads your post and assumes that someone would have to change A records manually in advance of an outage. That is not how this system works.

You define the failover IP address in advance, and your server is monitored 24x7\. If it goes down, the DNS records switch within a minute or two with no action on your part
I was hoping it was something like this, but it didn't work for me. I chose a Failover IP/server for another server, then I shut down the server. A domain which was pointing to the down server never did start pointing to the failover one. So I must have missed something somewhere.

Oh, but you mention DNS records change. So do you mean the DNS records in Linode DNS Manager? So therefore the domains need to be in there? My domains are not. I thought the failiver IP means the IP address would point to the failover server, not update DNS.

akerl said above that Linode doesn't failover IP's automatically, so I think that's the piece you are missing. In my case, DNS for the site that requires failover is hosted at DNS Made Easy, not at Linode, and DNS Made Easy does the IP switching on their end. You pay per monitored/failover IP, though, and it sounds like you've got a huge number of A records that would get expensive.

The interesting thing about the nodebalancer solution is that if you set up a single nodebalander and it went down, you'd at least get a 503 error, which is better than a timeout. So if you don't want to serve sites from two nodebalancers for true HA, you could just set up one and live with "Service Unavailable". Maybe that's your answer to "anything other than page/server not found"?

@hous, so your IP failover is controlled by your DNS provider? But how does the Linode one work if its not automated? Apologies if I misunderstand, just trying to get to grips with it all.

We mainly use the domain company to manage DNS records. We used to use PointHQ but they kept going down from denial of service attacks, so we moved all our domains DNS back to the domain registration companies, and some domains use Rackspace as DNS backup. Eitherway, our DNS management system does not support failover at the DNS level. LCN, the domain company, provide basic DNS management (like most registration companies). And as you guessed, using a DNS management system for all our domains (~400) would be very costly so its not something we wont pursue at the moment.

About nodebalancers, I am assuming I would need one per server (and then a second server), which means with 11 servers we would need 22 servers and 11 nodebalancers, and again, due to investment its not something I want to pursue.

What you suggest about the generic message is exactly what I want, but I dont know how to reassign an IP to the standby server… I just want one server with a "default" apache vhost setup for all incoming requests, and then when a main server goes down (planned or not) that main servers IP can be pointed to the standby server, thus showing a generic message for all the domains on the main server. For my premium hosting customers I can even add the domain to the standby server so as to customise the message with their logo and contact details. Amazon have elastic IPs I can move to a new server, I wondered why cant we do that with Linode. In my head its quite simple!

Still dont understand how Linodes failover IP work.

Here's how IP failover works, given two servers:

You have server A with IP address 2001:db8::1

You have server B with IP address 2001:db8::2

Server A removes its IP address, either by shutting down or by "ip addr del 2001:db8::1" or whatnot.

Server B adds the 2001:db8::1 IP address by "ip addr add 2001:db8::1 dev eth0" or whatnot.

When Server A comes back, Server B gives up the IP and Server A picks it back up.

Ok thanks, so the IP failover in Linode IS automated?

I did try this… for the main Linode I added a failover to a new server, I added the main server IP to eth0:0 on the standby server and rebooted, I then I shutdown the main Linode. The domains pointing to the main linode did not point to the standby linode (so the IP was not pointing to the standby linode), unless I made some mistake in setting up the IP on the standby linode. I even rebooted that after the main one shutdown.

I will go through it again some time and if that doesnt work still contact Linode support.

Edit: I didnt run this command though "ip addr add 2001:db8::1 dev eth0". I followed the static netorking guide to add a new Ip to the server.

It is not automated, because it requires that your servers perform specific, coordinated actions under conditions of your choosing. Linode gives you the ability to do it, you just have to do it when the time is right. (Something like Heartbeat and Pacemaker can automate this nicely.)

Also, you'd want to shut the "main server" down before bringing it up on the standby server, since all hell breaks loose if the same IP is on two servers at once. Also, it can help to originate a packet or two from the failed-over IP address, since routers tend to cache IP-to-MAC mappings for some time.

I mean the re-pointing of the IP address to the new server is automatic, once the main server shuts down? Or is that why I run the command "ip addr add"? I didn't run this. Or is adding it to eth0:0 like this OK and rebooting? https://library.linode.com/networking/c … interfaces">https://library.linode.com/networking/configuring-static-ip-interfaces

And to confirm, I assume when the main one comes back up I should remove the Ip address from the standby server again? As you say, you cant have multiple servers using the same IP? If this is true then thats a shame, I was hoping I could leave them in there. I was hoping a failover could work if the server goes down for unplanned maintenance without me doing anything.

Thanks

It's just like having two servers on a traditional Ethernet network: they can't both have the same IP address at the same time. An IP address maps to exactly one MAC address.

If you want more than one server sharing an IP address… well, that's what Nodebalancers are for. If it were as easy as just setting up the same IP address on two different servers, then Nodebalancers wouldn't need to exist…

No I understand thats fine. Setting up the failover will still be much quicker than messing with DNS manually. I shall give it a go next time I think.

May I ask one more question… do you know if I need to reboot the standby server in question? So when I shutdown the main linode, and add the IP address to the standby one, do I need to reboot that one? Just trying to estimate downtime as it sounds like there will be downtime between the IP address being re-assigned.

Thanks

No reboot is necessary – any Linode that is set up to bring that IP up can bring it up at any time, as long as no more than one Linode has it at any time.

I'd recommend firing up a couple Linodes and practicing…