Fremont RAM upgrade

On the IRC channel it has been said that the post will be coming up within the next few days, and they didn't get it done before due to some delays that will make much more sense once the blog post comes up…

This is wild speculation on my part, but I suspect some sort of migration of old servers housed in Hurricane Electric's Fremont 1 facility to HE's Fremont 2 facility (where I would guess their newer Fremont servers are) is behind the delay. I've been to both facilities and Fremont 1 is very low on space, and it would make sense to consolidate in one data center, even if they are just down the road from one another…

At least, I hope that's what it is, and not something that involves any sort of bad news for those of us who like where we are…

No bad news, all good news. We're working on the delays as best we can. Stay tuned!

@Valley:

This is wild speculation on my part, but I suspect some sort of migration of old servers housed in Hurricane Electric's Fremont 1 facility to HE's Fremont 2 facility (where I would guess their newer Fremont servers are) is behind the delay. I've been to both facilities and Fremont 1 is very low on space, and it would make sense to consolidate in one data center, even if they are just down the road from one another…

At least, I hope that's what it is, and not something that involves any sort of bad news for those of us who like where we are…

This has been the speculation (and hope) in other threads on the forum.

Can we get a less vague status update with some kind of reasonable timeframe? I've been planning an upgrade for my client's website based around your initial estimates (1-2 weeks. It has been nearly a month). My client wants to know when the site will be upgraded to faster servers and I don't have an answer for him.

I waiting too for my memory upgrade !

I am still hoping the delay is due to the Ameristralia servers being shipped from Fremont across the pacific to Australia where they belong. Meanwhile still loving my new improved Tokyo instance, at least until it gets wiped out by earthquake/godzilla.

Unforeseen technical difficulties resulting in the rollout taking longer than anticipated are fine.

Complete lack of any official update regarding the progress after nearly four weeks, when the rollout (or, at the very least, more information) was initially promised within two, is not.

@carterparks:

Can we get a less vague status update with some kind of reasonable timeframe? I've been planning an upgrade for my client's website based around your initial estimates (1-2 weeks. It has been nearly a month). My client wants to know when the site will be upgraded to faster servers and I don't have an answer for him.
Solution: Move your Linode to another DC for a few weeks more while they get the last quirks worked out.

Deleted

Sorry I don't have any more information, other than we're working as hard as we can on it and hope to enable the upgrades very soon.

-Chris

Deleted

Yes, of course we realize. You can work around that by opening a ticket and we'll get you fixed up.

-Chris

@lakridserne:

Solution: Move your Linode to another DC for a few weeks more while they get the last quirks worked out.
Many of us are geographically tied to Fremont.

Sigh.

Remember when Linode used to be the cool nerdy VPS provider that was transparent about this sort of thing?

Starting to make me uncomfortable / wonder what it is, exactly, that Linode doesn't wan't us to know.

@Valley:

@lakridserne:

Solution: Move your Linode to another DC for a few weeks more while they get the last quirks worked out.
Many of us are geographically tied to Fremont.

Sigh.

Remember when Linode used to be the cool nerdy VPS provider that was transparent about this sort of thing?

Starting to make me uncomfortable / wonder what it is, exactly, that Linode doesn't wan't us to know.
1) It's not that much more time it takes when we're talking about the internet

2) They're the same cool and nerdy VPS provider that are transparent about things. They have never, to my knowlegde, preannounced things.

3) They will announce it as soon as possible.

Strictly speaking, they did kinda pre-announce that Fremont would be getting the upgrades soon, and are not saying anything about why it hasn't happened yet. So, I would say that this isn't a great example of either transparency nor of avoiding premature announcements.

Here is the straight dope: We were getting everything ready for a move to Hurricane Electric's FMT2 facility, which is like two miles down the road from FMT1. This was all to be ready about a week after the upgrade announcement. FMT2 is a modern build-out, which we consider to be a great improvement over FMT1's aging power and space restrictions. The strategy involved us getting a 10G wave on fiber they have between buildings, which places our two distinct networks on the same layer2 network. Meaning no IP swaps, no disruption in IP failover support, private networking working as expected, etc. Essentially this has created one large LAN despite hardware and Linodes existing in both facilities. Over the course of 3 months or so, we would perform a few phases of migration queues + physically moving and upgrading hardware from FMT1 into FMT2, and then continuing with the next phase, rinse, repeat. Doing it in phases was largely a host-resource game.

This was all staged and ready to go, but we discovered this link wasn't as redundant as we had believed. We asked questions, and got answers from HE. There are scenarios where the wave between the two facilities could break, and that news put a stop to everything while we investigated how to implement absolute redundancy between the two facilities in the event the link failed. We have a solution, but want to greatly shorten the time we're relying on this bridged network, which means altering the FMT1 to FMT2 transition timeline. This is where we stand now.

Things could change, but we're now thinking this will be: a few weeks heads-up during which the upgrade-button would simultaneously upgrade you AND migrate you to FMT2. Those who don't perform the upgrade by the deadline(s) would have their host shut down, the host de-racked and physically moved to FMT2 and then powered back on. Of course we would have everything prepared in FMT2 to receive these machines and essentially it would be: a shutdown, unplug host, transport it to FMT2, rack it and plug it into already-ran cables, and then power it on. Logistics so far indicate the downtime for that scenario is minimal, 60-120 minutes, performed in the middle of the west-coast night. Over the course of a few days we'd be out of FMT1 completely.

We've already deployed an impressive amount of capacity in FMT2, however this shortened timeline and elimination of phases means upgrade+migrate-to-fmt2 availability will be on a first-come-first-served basis. We've also been working on redirecting host builds from other facilities to fmt2 to augment fmt2's host resources. However, we can't guarantee everyone that wants to move/upgrade early will be able to, under this scenario.

There are still a few loose ends, and of course this could all change, but that's where we are. The logistics of this type of endeavor are quite complicated as you can imagine, and we want to do this with the least amount of stress, downtime, and risk as possible. We hope to nail down the timeline and enable the upgrade button very soon.

-Chris

Thank you so much for the update.

I have hated being on Fremont (FMT1) all this time, and I cannot wait until I get to FMT2. It's helpful to know what's going on. You're the best!

Thanks!

Bruce

Thanks Chris,

This is exactly the kind of answer I was hoping we'd get (it's obvious there was some sort of delay) so knowing what's happening and having appropriate expectations is the next best thing. Also, your response shows that you/the Linode team truly are concerned about making this a smooth transition for us, although I know that the world wouldn't come to an end if my very small $20/mo Linode went down, it's nice to know that minimizing downtime and inconvenience is a priority.

I really wish that you guys at Linode would post this sort of thing more often. Not necessarily all the time, or in that much detail, but it's nice to see. It's kind of like UI feedback; for a task that takes a given number of seconds, compare the user experience to having a progress bar versus just locking up the UI. The user who gets the progress bar is going to feel like the machine is faster and more response, while the user who just gets a frozen screen for the duration is going to feel the machine is slow and unresponsive.

It's like that for this; getting some insight into the technical stuff behind the scenes may not be something that we customers need, but it's definitely appreciated and reinforces that Linode (the company) is doing something. It also tends to make customers more tolerant of stuff too.

So, whenever you guys can afford to share some insight into goings-on, it's appreciated.

Thank you for the update, Chris, as well as your apparent attention to the details. It's great to see the move to FMT2. Does the current plan still mean no IP swaps?

Thanks for the update! Would love to see more detail like this in the future as has been in the past.

If we create a new linode at Fremont, which facility is the linode being created in?

FMT1, until we enable the upgrade button - then all new Linodes will go to FMT2.

-Chris

Thank you SO MUCH, Chris. :D

That's the kinda thing I would have been happy to hear earlier on – as I am now. Good luck with the move, glad to hear about servers there already.

That all makes perfect sense. FMT2 is awesome. I like their "dark room" -- it's like putting an amphitheater over a small town then shutting off all the lights.

@lakridserne:

@Valley:

@lakridserne:

Solution: Move your Linode to another DC for a few weeks more while they get the last quirks worked out.
Many of us are geographically tied to Fremont.
1) It's not that much more time it takes when we're talking about the internet

Yes, we are taking about the Internet – and not the web, which seems to be your perspective since you do shared hosting. For other applications where latency is critical (let's use a game server as an example), a difference of even 10ms ping could make all the difference.

@Guspaz:

I really wish that you guys at Linode would post this sort of thing more often. Not necessarily all the time, or in that much detail, but it's nice to see. It's kind of like UI feedback; for a task that takes a given number of seconds, compare the user experience to having a progress bar versus just locking up the UI. The user who gets the progress bar is going to feel like the machine is faster and more response, while the user who just gets a frozen screen for the duration is going to feel the machine is slow and unresponsive.

The problem is that situations like this are often fluid. There's a tough balance to strike between enough information and too much. Had Chris been putting out the updates along the way, I could easily see people misinterpreting pieces of it and getting unrealistic expectations or simply not understanding pieces of it and jumping to conclusions. There's a reason that linode usually doesn't communicate things out until they're done, because then you don't have to deal with threads like this where people are saying "it's been two weeks, where is it!?!?". My day job has plenty of this kind of stuff, so I'm very sympathetic to Chris' position (of course, my linodes are in NJ, so it helps that I'm not personally affected as well).

I'm pleased to have some info on the Fremont RAM upgrade. I will admit to a little frustration at being in the dark for about a month. AS my company's CEO likes to say, "Our customers can handle good news. They can handle bad news. What they can't handle is no news." I kind of felt like that.

Thanks for the forthcoming upgrade and the news.

Thanks for the update, it's really very much appreciated.

Chris… great post.

So how can I keep up to date with how the move is progressing? Are you going to email everyone who has a host in Fremont when you open the migration queue… or do I need to check the blog and/or this thread every day to ensure I get a good spot in the line… really looking forward to to the upgrade.

It will be a blog post first, and soon after: tickets (emails) to everyone yet to be moved.

-Chris

I am currently on an 768 plan - I would get upgraded to an 1536, but really I just need the lowest plan.

Would a resize to the 1 gig also upgrade me or should I upgrade then resize?

@mosdl:

I am currently on an 768 plan - I would get upgraded to an 1536, but really I just need the lowest plan.

Would a resize to the 1 gig also upgrade me or should I upgrade then resize?

First Upgrade then resize, you can't resize first

In the queue now! 37 out of 40.

I am in the queue too. Finally!

I got in the queue 1 of 1 at 7:00 p.m. local time… Gotta love Alaska Daylight Time!

Thanks Linode!

My Fremont Linode has been upgraded. Yay!