本周早些时候，英特尔公开披露了一类新的处理器漏洞，称为L1 Terminal Fault（L1TF）。L1TF的变种影响了许多单用户和多用户环境，包括一些Linode的基础设施和Linodes本身。
我们已经开始了缓解工作，并预计在未来几周内对我们的车队进行全面缓解。我们相信我们可以在不中断你们的运行系统的情况下实现这一目标，而且不需要你们的任何协调。然而，这仍在发展中，我们会在发展中了解更多。我们的缓解措施的早期结果是令人鼓舞的。
虽然这保护了我们这边的事情，但你应该确保你运行的是有缓解措施的Linux内核。请看我们的升级内核指南。
随着我们在未来几周内推进减灾工作，我们将继续在我们的博客上提供更多信息。敬请关注!
评论 (8)
Thanks for the hard work in dealing with this. Though I am not sure it’s enough to just update the Kernel on OS side, need microcode updates – I guess from host node OS level too https://www.linode.com/community/questions/17120/how-is-linode-handling-l1tf-what-actions-can-we-take-to-mitigate#answer-66869 ?
Wouldn’t the microcode updates require host node level reboots ?
You are correct! We’re able to transparently move VMs to patched infrastructure using live migrations.
Ah sweet – live migration feature is awesome. One of many reasons I have stuck with Linode for 4+ yrs now 🙂
What are your plans regarding HyperThreading?
One of the things that has me shocked about L1TF is that there does not yet appear to be any publicly-available, complete mitigation to either of the major open-source hypervisors (KVM and Xen) that does not require HyperThreading to be disabled.
L1TF is not fully mitigated if unrelated guests can run as hyper-siblings (or if an untrusted guest–which is all guests for a cloud VM provider–can run as a hyper-sibling of a hypervisor thread). Technically, this could be enforced by a scheduler, but the most unequivocal statement of a scheduler that will do so comes from, of all places, Microsoft, and therefore Azure (https://blogs.technet.microsoft.com/virtualization/2018/08/14/hyper-v-hyperclear/).
Google also indicates that individual cores are never concurrently shared between VMs (https://cloud.google.com/blog/products/gcp/protecting-against-the-new-l1tf-speculative-vulnerabilities). Certainly, they have the wherewithal to pull this off with custom internal kernel changes, so there’s no particular reason to doubt them. (I didn’t find any clear statement from AWS on shared cores, but they already have their custom Nitro hypervisor, so plausibly they have a custom modification.)
Unfortunately, the current docs applicable to KVM don’t provide any good solution for a cloud VM provider other than disabling HyperThreading: https://www.kernel.org/doc/html/latest/admin-guide/l1tf.html
Am I wrong about this?
I too would like to know more about the hyperthreading story. We have multiple internal deployments of openstack and vmware that would suffer if we have to disable HT. Did Linode disable HT?
I am very happy with Linode being able to live migrate things with no downtime to customers. That is a massive improvement over the past migration queues.
Thanks for implementing these security fixes.
The new “live” migrations is certainly interesting – is this a new feature that you’re now able to use? It’s certainly much less painful than existing migration queues and forced downtime.
Futhermore, will live migration be introduced for other server moves, such as upgrades and downgrades?
Our current plan for L1TF mitigation is to disable HyperThreading.
Yes, live migrations are a feature that we are now able to use. We are evaluating the different use cases for this one, but currently it cannot be used for upgrades/downgrades with plan resizing.
Thanks for keeping us informed and patching the hosts. We appreciate the effort and due diligence. I’m sure these projects at large scale are never fun.