In dieser Woche wurden mehrere schwerwiegende Sicherheitslücken, die viele CPU-Architekturen betreffen(CVE-2017-5753, CVE-2017-5715 und CVE-2017-5754), vom Project Zero-Team von Google und anderen bekannt gegeben. Unser Team arbeitet mit Anbietern und unseren eigenen Ingenieuren zusammen, um die Auswirkungen auf unsere Plattform zu bestimmen, aber die Erwartung ist, dass ein flottenweiter Neustart notwendig sein wird, um sich gegen diese Probleme zu schützen.
Während wir unseren Reaktionsplan durcharbeiten, haben Sie bitte Verständnis dafür, dass aufgrund der Art und Schwere dieser Probleme eine schnelle Reaktion erforderlich sein kann. Wie immer werden wir so viel Vorankündigung wie möglich geben. Wir werden Sie direkt mit Informationen zur Terminplanung informieren, wenn ein Neustart Ihrer Linodes erforderlich ist.
Wir werden Sie an dieser Stelle weiterhin auf dem Laufenden halten, sobald mehr Informationen verfügbar sind.
Informationen zu diesen Sicherheitslücken finden Sie auf den folgenden Seiten:
- Googles Project Zero Beitrag
- meltdownattack.com
- Dokumentation/x86/kaiser.txt
- LWN: KAISER: Verstecken des Kernels vor dem Userspace
Update: 4. Januar 2018
Wir untersuchen dieses Problem weiterhin und wollten Ihnen ein kurzes Update über den aktuellen Stand geben:
- Wir verschieben alle nicht damit verbundenen Wartungsarbeiten, um unsere Bemühungen und Ressourcen auf die Behebung dieses Problems zu konzentrieren.
- Wie das Scaleway-Team heute bereits erläutert hat, haben wir uns aufgrund der unvollständigen Informationen der Hardware-Hersteller mit anderen potenziell betroffenen Cloud-Hosting-Anbietern zusammengetan, darunter Scaleway, Packetund OVH. Wir haben einen speziellen Kommunikationskanal eingerichtet, um Informationen auszutauschen und gemeinsam an der Behebung der Schwachstellen von Meltdown und Spectre zu arbeiten.
- Wir setzen die interne Bewertung und Prüfung von Abhilfemaßnahmen fort.
- Für morgen haben wir Gespräche mit den Hardware-Anbietern angesetzt, um tiefer in die Materie einzutauchen.
Wir werden an dieser Stelle weiterhin Updates zur Verfügung stellen, wenn es angebracht ist.
Update: 5. Januar 2018
Wir machen weiterhin Fortschritte und wollten Sie über das Neueste informieren:
- Die neuesten stabilen und langfristigen Linux-Kernel wurden heute mit den KPTI / Meltdown-Patches veröffentlicht. Daher haben wir Ihnen den Kernel 4.14.12 zur Verfügung gestellt und ihn als den neuesten eingestellt. Wenn Sie einen Linode-Kernel verwenden, wird Ihre Linode bei Ihrem nächsten Neustart auf diese Version aktualisiert. Dies schützt Sie nicht vollständig vor den Meltdown- und Spectre-Schwachstellen, bietet uns aber eine gute Grundlage, mit der wir arbeiten können, während wir eine vollständige Behebung planen.
- Wir hatten Planungssitzungen mit unseren Hardwareanbietern und haben an Implementierungsplänen für Kernel-, Hypervisor- und Firmware-Updates gearbeitet. Diese werden alle benötigt, um uns in einen behobenen Zustand zu bringen, aber nicht alle sind verfügbar.
Wir erwarten nicht viel Bewegung in dieser Sache über das Wochenende, während wir auf externe Abhängigkeiten warten, werden aber sicherlich hier Updates bereitstellen, wenn es welche gibt. Wenn nicht, werden wir am Montag nächster Woche hier weitere Updates veröffentlichen.
Update: 8. Januar 2018
Wir machen weiterhin Fortschritte bei unseren internen Tests, warten aber immer noch auf Microcode-Updates von unseren Hardware-Anbietern. Sowohl das Microcode-Update als auch das Kernel-Update sind erforderlich, um eine angemessene Entschärfung der drei Varianten von Meltdown und Spectre zu gewährleisten.
Update: 9. Januar 2018
Wir haben den heutigen Tag damit verbracht, den Plan für den Einsatz der Meltdown-Abschwächungen in der gesamten Linode-Flotte vorzubereiten. Im Laufe des nächsten Tages werden wir die Korrekturen für eine Teilmenge der Flotte implementieren, die Auswirkungen überwachen und dann mit dem Rollout für den Rest fortfahren. Die Meltdown-Entschärfung erfordert einen Neustart unserer physischen Hardware, der die darauf gehosteten Linodes neu starten wird. Eine Teilmenge der Linodes in den Rechenzentren in Tokio 1, Frankfurt und Singapur wird als Teil dieser ersten Gruppe neu gestartet. Die Betroffenen erhalten ein Support-Ticket und eine E-Mail mit Informationen zur Terminplanung.
Die Reboots für diese Woche betreffen nur Meltdown. Parallel dazu finden Tests und Planungen für Spectre statt. Weitere Reboots in den kommenden Wochen werden erforderlich sein, um alle Spectre-Varianten ordnungsgemäß zu entschärfen.
Update: 10. Januar 2018
Der Rollout für die Teilmenge unserer Flotte ist bisher gut verlaufen, was die Meltdown-Abschwächung angeht. Wir fahren mit diesem Plan fort und werden im Laufe der nächsten Tage Reboots für den Rest der Flotte durchführen. Betroffene Kunden erhalten Support-Tickets und E-Mails mit dem Reboot-Fenster für ihre Linodes mit einer Vorankündigung von mindestens 24 Stunden.
- Aufgrund des anhaltenden Problems wurde die folgende Statusseite erstellt.
- Wir veröffentlichen bald ein Dokument, das eingehender über Meltdown und Spectre spricht, um zu zeigen, was es für dich bedeutet und was du tun kannst, um dich darauf auf deinen Linoden vorzubereiten. Wir werden einen Link dazu in einem kommenden Blogbeitrag veröffentlichen.
Update: 11. Januar 2018
Die Entschärfung der Meltdown-Sicherheitslücken läuft weiter, wir machen jeden Tag Fortschritte über die gesamte Rechnerflotte hinweg. Es gibt einen neuen Leitfaden mit weiteren Informationen zu diesen Schwachstellen und wie Sie Ihr Linode schützen können: Was Sie tun müssen, um von Meltdown und Spectre verschont zu werden.
Update: 12. Januar 2018
Wir arbeiten weiter an der Meltdown-Entschärfung und haben für das Wochenende Neustarts geplant. Unser Zeitplan läuft bis zum 18. Januar. Wir werden unsere täglichen Blog-Updates pausieren, bis dies abgeschlossen ist, es sei denn, es werden weitere verwertbare Neuigkeiten bekannt.
Update: 8. Februar 2018
Zur Erinnerung: Alle unsere KVM -Hosts sind jetzt ordnungsgemäß für Meltdown entschärft. Wir arbeiten weiterhin an einer angemessenen Entschärfung für die Spectre-Schwachstelle und werden einen aktualisierten Plan in unserem Blog bereitstellen, sobald dieser verfügbar ist.
Weitere Informationen über diese Schwachstellen, den Status unserer Flotte und wie Sie Ihre Linode schützen können, finden Sie im Leitfaden zu Meltdown & Spectre.
Kommentare (77)
Wishlist: alternate-CPU-architecture Linode hosts 😉
-Eugene
Sadly, portions of this affect both AMD and Intel, and likely others, fwiw.
My wishlist would be that all VPS providers could get the same early notifications and patches that a certain trendy yet slower provider received two weeks ago.
Yup, ARM’s response is https://developer.arm.com/support/security-update
fwiw, Scaleway is patching & mass rebooting hypervisors 1/4/2018
Sadly, it appears that even ARM CPUs are affected (to some degree) by this design flaw – so even if other CPU architectures were available, a reboot would still likely be necessary.
I could be wrong, but I think Eugene is referring to an alternative architecture like RISC-V…?
https://www.codasip.com/2016/09/22/what-is-risc-vwhy-do-we-care-and-why-you-should-too/
Thanks for the update. One more question, the patch says 5-30% performance hit depending on workload. Do we need to add more VMs to deal with the load?
@Scott, @Eugene – it seems that “all CPUs” may very well be most modern CPUs; RedHat advisory claims that POWER and even SystemZ (as used in IBM mainframes) may be impacted by Spectre.
https://access.redhat.com/security/vulnerabilities/speculativeexecution
Basically if your core does speculative execution for performance gain then it may be vulnerable. The BOOM RISC-V core ( https://www2.eecs.berkeley.edu/Pubs/TechRpts/2017/EECS-2017-157.html ) can do out of order execution and so _may_ be vulnerable. It would require a deeper look at other implementations of the spec to see if they are vulnerable or not.
Will this require patching the OS in our Linodes?
We’re still planning our full mitigation strategy, but Linodes will need an updated kernel, and we’re working on providing one from the Linode Manager. If you currently use a distribution-supplied or custom-compiled kernel, you will need to take separate actions to update it.
@Ruben Yes.
It appears that patching the guest is required to mitigate meltdown on Xen VM’s. If KVM, it should just be the hypervisor that needs the patches. Am I reading this correctly?
https://www.theregister.co.uk/2018/01/04/intel_amd_arm_cpu_vulnerability/
Correction, all VM’s have to be patched because of Meltdown, not just those on Xen.
Thanks Linode for your quick response and adapting 🙂
Related tweet by WikiLeaks at https://twitter.com/wikileaks/status/948723793324838914
Official website: https://meltdownattack.com
Article: https://www.theregister.co.uk/2018/01/04/intel_amd_arm_cpu_vulnerability/
A severe design flaws that allow stealing of sensitive data from memory has been discovered in Intel chipsets, affecting Xen, KVM, and more.
Linode will fix this, I trust in them. So no worries here.
Linode, can you please provide the details of patching hypervisors?
I guess mitigation of memory reads between different VMs is often even more important than within a single VM.
I am relatively new to Linode, and have a low volume site. When critical issues like this are discovered and eventually fixed, it would be awesome to get an email with directions on what to do, if anything. Thanks !!
Could you clarify how “[upgrading your VM kernel] provides us a good foundation to work with while planning for full remediation”? I mean, less attack surface is great and all, but, how does it factor into your planning? I ass-u-me you’re going to be rebooting my host at some point anyway.
It’s great to see that you are on top of it. As long as we are informed ahead of time, we are fine with the reboot and security patches. As you know our customers don’t like downtimes. Also, please try to minimize fleet-wide reboot, which makes services completely unavailable.
Thanks Linode for the updates.
I guess being on a shared (virtualized) server, all linodes on that (physical) server have to apply the new kernel, for the protection to be really effective.
That’s a good start anyway.
Thanks for updated Kernel. Looks like it doesn’t support Redhat/CentOS KPTI tunables to be able to control KPTI and related patch operations https://community.centminmod.com/posts/57936/. Would be nice to have though – details https://access.redhat.com/articles/3311301
cat /sys/kernel/debug/x86/pti_enabled
cat: /sys/kernel/debug/x86/pti_enabled: No such file or directory
cat /sys/kernel/debug/x86/ibpb_enabled
cat: /sys/kernel/debug/x86/ibpb_enabled: No such file or directory
cat /sys/kernel/debug/x86/ibrs_enabled
cat: /sys/kernel/debug/x86/ibrs_enabled: No such file or directory
were any Spectre fixes added to the kernel ? PoC at https://github.com/crozone/SpectrePoC successfully runs = not fixed on updated linode with 4.14.12-x86_64-linode92 on centos 7.4 64bit
but on dedicated elsewhere with centos 7.4 64bit and 3.10.0-693.11.6.el7.x86_64 the PoC fails to read = fixed
@George: One of the Spectre vulns requires you either recompile EVERYTHING with mitigations or a microcode patch. The other Spectre vuln isn’t fixable without newly architecture hardware (that doesn’t exist yet)
Meltdown is the one you apply the KPTI patch for (Intel only).
So Linode will probably have to issue a second round of reboots when their motherboard/OEM’s get around to issuing a CPU microcode patch. (Or they recompile everything)
Re: previous comment, ah upstream linual kernel hasn’t tackled Spectre yet according to http://kroah.com/log/blog/2018/01/06/meltdown-status/ but some distro backported kernels have i.e. Redhat/CentOS
you say “If you are leveraging a Linode kernel, upon your next reboot your Linode will be upgraded to this version.”.
Is the easiest way to tell that by doing a `uname -a` and seeing if the string contains `-linode` e.g. “4.9.50-x86_64-linode86”
@Patrick,
If `uname -a` is showing “somethingsomething-linode”, your kernel is coming from Linode. The particular version is assigned each time the instance boots, so if it is out of date, just reboot your Linode.
If you aren’t sure about the kernel source or want to change it, you can log into the Linode Manager and click the “edit” link for your Linode’s Configuration Profile. The Kernel option under Boot is where this particular setting is stored.
Some of the OS patches are requiring microcode changes. But I think the linux ones work without and differently once the microcode has changed.
The microcode will come in the form of a firmware update from the vendor.
Is there a reason why the Intel microcode update cannot be used directly rather than waiting for vendors to repackage it?
Ie, https://downloadcenter.intel.com/download/27337/Linux-Processor-Microcode-Data-File
Apparently the earlier mentioned link is not the latest version, not sure of the direct Intel link but it’s apparently included in eg https://launchpad.net/ubuntu/+source/intel-microcode/3.20171215.1
The question remains the same regardless.
excelent
Hi,
Any idea how long the reboots will take once they are scheduled?
Thanks!
Neil
We’ve allocated a two hour window for maintenance, however in many cases the actual downtime will be less. That being said, we would still recommend preparing for a full two hours of downtime.
The link given in “Jan 11 update” gives misleading information.
“Spectre targets the way modern CPUs work, regardless of speculative execution” is incorrect.
Both Spectre and Meltdown take advantage of “speculative execution”.
While Meltdown exploits a race condition based on the code after an exception is triggered, Spectre relies on the code speculatively run after a ‘if’ branch (uncached) condition that “usually” goes through happens to be false (since the next code accesses an out of bounds array).
I used to use linode vps, very good network speed.
> Our schedule runs through January 18th.
Are you kidding me.. Why so long to patch Meltdown?
Rather let your customers stay vulnerable to the exploit than lose a few customers because of insufficient server capacity?
Hey Krian!
Due to the scope of this vulnerability, we are rolling out the patch in waves to balance downtime for customers as well as ensure the patches work effectively across the entire fleet. With the hasty release of the kernel patches, we are making sure the patches don’t cause more issues for our customers than they fix.
So, my service is spread accross 5 linodes. Bringing down my VMs, one at a time, at unknown intervals spread over the next several days, is going to cause me to have possibly *five* outages in the worst case (if my VMs are all on different physical hosts, which I have no way of knowing), rather than one.
@Neil Ticktin
It took one minute to fix.
According to Uptime Robot, the monitor (my linode VM, Linode 2048) is back UP (Keyword Exists) (It was down for 0 minutes and 49 seconds).
I am relatively new to Linode, and have a low two volume site. Today I got the email with this subject “Linode Support Ticket 9678973 – Critical Maintenance for CPU Vulnerabilities (Meltdown) “. it would be great, If you provide an email what to do exactly, or anything.
Thanks !!
Hey Maneesh, first of all, welcome to Linode! For these Critical Maintenance for CPU Vulnerabilities (Meltdown) tickets, there is no action required on your end. That being said, we do recommend making sure your Linode is set to the latest kernel, which you can read more about how to do here. We would also recommend taking a look at the Reboot Survival Guide to ensure these reboots and migrations have as little impact on your Linode as possible.
I’m still seeing “Maintenance is not yet scheduled” on my dashboard.
Warning would be good it sounds like there is a schedule.
Hi Adrian! Wo don’t have a full schedule of exactly what host will experience the mitigation at what time just yet, however once we do set the schedule for a host your Linode is on you will be alerted with a ticket and via the dashboard.
How long reboots will take, once it is scheduled? Thanks in advance!
The maintenance window for hosts is 2 hours, however we expect the reboots to not take the full two hours. Beyond that, I’m afraid I can’t really give a more accurate assessment of how long the reboot will take. Hope this helps!
Oh. It is serious issue that we need to mitigate.
The whole operation, from shutdown to server back up and running took 11 minutes.
Mine reboot was around 45 minutes.. I moved my servers for now in DigitalOcean. I do not know what DO is doing but as of the moment the do not promise down time..
@Romel
That is hilarious. DO hasn’t done their reboots yet. So you leave a hosting provider for a serious, unavoidable reboot to another that has to do the same thing!
Needed that laugh this morning.
You’ve schedule 2/3 of my cluster hosts for the same window, and given me no way to reschedule that. Support has not responded to my message about preventing downtime on my cluster by either rescheduling or migrating one of my nodes to another host. That means in 11 hours my cluster will be broken.
What happened linode? You used to be good about handling these outages, but last summer you started to suck. Please improve, I’d hate to end a 4 year relationship over this.
Hey Zach, I’m terribly sorry we haven’t gotten back to your ticket yet. Can you let us know the ticket number so I can take a look and see what we can do for you? Please also feel free to call our support line for more immediate assistance as well. Thank you for being patient with us during this process!
reaaaally wish you’d had a pool of servers that we could migrate onto, on our own schedule instead of this middle-of-the-night stuff. You’ve done it before.
I apologize, due to the scope and severity of these vulnerabilities, we are unable to be as flexible as we would like with the mitigation process. If you open a ticket or call the support help line, however, we will be happy to see what we can do for you.
@Zach: Ask support for server migration for one of the cluster nodes to another hypervisor before reboot…
Hmm.
The mugration og my first linode scheduled, could be started when it fit my schedule. Why not the next? (Both nodes are located in london)
/Henning
I have a question about notifications. In the past, when downtime was scheduled for one of our systems, we’d receive an e-mail giving us plenty of advance notice. With the reboots for Meltdown/Spectre, however, we were left to polling our Linode Manager page to see what, if anything is scheduled.
From reading an article at Ars Technica, it appears that Linode learned of these vulnerabilities like the rest of us — with no advance notice. I cannot imagine the scrambling that must have caused!
As you would have appreciated advance notice, so too would we.
We’ve been hosted on Linode coming up on 4 years and currently have 9 servers here. One of the reboots took down 3 of our nodes at once. As one of these handled DNS, the lack of sufficient notice prevented us from redirecting to our redundant system (TLS propagation issues). Fortunately, that reboot happened quickly and our loss of service resulted in limited down time for our site (less than 15 minutes).
As further phases of remediation are planned, we expect that each of our systems will see at least one more reboot.
How can we get email notification of upcoming server reboots?
I completely understand where you’re coming from. For this round of reboots we aimed for at least 24 hours notice via ticket. For future reboots we’ll be able to provide notice further in advance, which will allow you more time to plan for any maintenance.
You’ll be notified of any ticket updates via the contact email address on file. If you’re not receiving notices via email let us know in a ticket and we’ll be happy to take a look.
Thank you for the prompt response!
Best wishes to you as you try and deal with rolling out fixes to thousands(?) of systems and deal with anxious customers.
We have continued to receive e-mail messages informing of system reboots… *after* they occurred. The last e-mail message *predicting* a reboot was sent on Jan 11, 2018. We had 7 systems reboot after that, and only learned of those ahead of time by constant scanning of our Linode Manager page.
This is both error-prone and time-consuming.
Our Linode Manager Notifications tab ( https://manager.linode.com/profile/notifications ) currently shows:
Linode Events Email
Events Email Notification
Notifications are currently: Enabled
So we *should* be receiving e-mail notifications in *advance* from now on? Or is there something else that must be done?
Yes, you should receive an email notification before any reboots from here on out. Email notifications should be much more timely moving forward. Even if there is a delay, the increased notice we’ll be providing for future maintenance will allow ample time for email notifications to reach you before any reboots take place.
That is a tremendous relief — Thank You!
That is one thing Linode has made a name for itself — being open and forthright in dealings with its clients. Thanks for holding your standards high!
This is unbearable. Our linodes go down with no notice. Went down twice in the last 12 hours for about an hour each time. The scheduled downtimes have been VEEERRRYYY slowish, also up to 1 hour which is unheard of in professional hosting… Lots of our services affected, our users outraged… What are u going to do about it folks??
Hello George, we are terribly sorry for the inconvenience. Due to the scope and and serious nature of the vulnerabilities, we had to perform the maintenance in an expeditious manner. You should have had tickets open informing you of the downtime in at least 24 hours notice. The maintenance window to address the vulnerability was 2 hours, however often the maintenance itself was shorter within about the hour mark. Do you currently have a ticket open so we can take a look at your account and confirm exactly what happened? I thank you for being patient through this process.
Hello,my maintenance status show phase 1 complete and future maintenance is pending,i want to know when will the migration plan come to an end。
now i try to ping my linode IP aways show request timeout, thank you!
This round of maintenance has completed, and your Linode should have returned to its previously booted state. I’m sorry to see the pings are timing out when you attempt to connect to the Linode. Do you currently have a ticket open with us to we can take a deeper look at this?
i had open a ticket to you,please help me to solve the problem,thanks!
Can you please provide the ticket number? Thank you.
Hello,my maintenance status show phase 1 complete and future maintenance is pending,i want to know when will the migration plan come to an end。
now i try to ping my linode IP aways show request timeout,
my ticket number 9869831
thank you!
We don’t currently have an ETA on the next maintenance window but tickets will be sent out once that is determined. We are currently waiting for patches to come from our hardware vendors.
We have a medium size fleet of Linodes and are seeing random loss of connectivity (requiring a reboot from Linode control panel to fix) across this fleet. This is causing major problems; last one to go was a primary database server which took out most of the fleet, and the nature of the networking problems is not playing nice with our cluster failover (eg the failures are inconsistent so the cluster might think the master it elected is fine, but outside the cluster its not visible – so the whole application is broken).
Linode support is aware of networking issues and advised us to reboot onto latest kernel versions which, unfortunately, do not seem to have resolved these issues.
Hi
I’ve set passwordauthentication NO
But now i can’t log in with my public key any more.
the public key remind the same and nothing change.
It show:
PuTTY Fatal Error
Disconnected: No supported authentication methods available (server sent: pulickey,gssapi-keyex,gssapi-with-mic)
Should I wait until the maintenance done so I can log in?
We’d like to look into this with you. Could you open a ticket for us and let us know the ticket number so we can locate it on our end?
You may be able to use Lish to log into your Linode if it’s up and running, even if SSH isn’t working:
https://www.linode.com/docs/guides/using-the-lish-console/
ticket 9859656
@paul, what would be a medium sized fleet?
Linode, can you please provide the details of patching hypervisors?
I guess mitigation of memory reads between different VMs is often even more important than within a single VM.
We just completed fleet-wide reboots for Spectre v1/v2 mitigation and are now working through a few rounds of migrations in order to close this out. More information can be found on the status page here: https://status.linode.com/incidents/8dbtk37dwm67/.
None of your servers need need BIOS/firmware updates for any of the recent CPU vulnerabilities? Would you consider providing at least some hardware with all AMD & Intel “management” features disabled? That seems like it would be 100% unique offering for a cloud host.
Also, I would like to know if you use Intel ME for any management done in the datacenters – or you use other tools (which I think are more suited to managing datacenters).
Hi Wayne – In addition to switching to our latest patched kernel (5.1.5), we are addressing these vulnerabilities at the host level during scheduled maintenance windows. This guide has additional detailed information on these vulnerabilities as well as their mitigation.
As far as providing “hardware with all AMD & Intel ‘management’ features disabled,” I have added your suggestion to our internal tracker.
Regarding your last question about using Intel ME or other tools, we aren’t able to discuss specific information like this. Though if you have any other questions, let us know and we’ll be happy to provide as much information as we’re able to.