A principios de esta semana Intel reveló públicamente una nueva clase de vulnerabilidades en los procesadores conocida como L1 Terminal Fault (L1TF). Las variantes de L1TF afectan a muchos entornos de uno o varios inquilinos, incluyendo algunas de las infraestructuras de Linodey los propios Linodes.
Hemos comenzado los esfuerzos de mitigación y prevemos la mitigación completa de nuestra flota en las próximas semanas. Creemos que podemos lograrlo sin ninguna interrupción de sus sistemas en funcionamiento y sin requerir ninguna coordinación por su parte. Sin embargo, esto todavía está evolucionando y sabremos más a medida que avancemos. Los primeros resultados de nuestras mitigaciones han sido alentadores.
Mientras que esto protege nuestro lado de las cosas, debe asegurarse de que está ejecutando un kernel de Linux con las mitigaciones en su lugar. Echa un vistazo a nuestra guía sobre cómo actualizar tu kernel.
A medida que avancemos en nuestros esfuerzos de mitigación durante las próximas semanas, seguiremos proporcionando más información aquí en nuestro blog. Esté atento.
Comentarios (8)
Thanks for the hard work in dealing with this. Though I am not sure it’s enough to just update the Kernel on OS side, need microcode updates – I guess from host node OS level too https://www.linode.com/community/questions/17120/how-is-linode-handling-l1tf-what-actions-can-we-take-to-mitigate#answer-66869 ?
Wouldn’t the microcode updates require host node level reboots ?
You are correct! We’re able to transparently move VMs to patched infrastructure using live migrations.
Ah sweet – live migration feature is awesome. One of many reasons I have stuck with Linode for 4+ yrs now 🙂
What are your plans regarding HyperThreading?
One of the things that has me shocked about L1TF is that there does not yet appear to be any publicly-available, complete mitigation to either of the major open-source hypervisors (KVM and Xen) that does not require HyperThreading to be disabled.
L1TF is not fully mitigated if unrelated guests can run as hyper-siblings (or if an untrusted guest–which is all guests for a cloud VM provider–can run as a hyper-sibling of a hypervisor thread). Technically, this could be enforced by a scheduler, but the most unequivocal statement of a scheduler that will do so comes from, of all places, Microsoft, and therefore Azure (https://blogs.technet.microsoft.com/virtualization/2018/08/14/hyper-v-hyperclear/).
Google also indicates that individual cores are never concurrently shared between VMs (https://cloud.google.com/blog/products/gcp/protecting-against-the-new-l1tf-speculative-vulnerabilities). Certainly, they have the wherewithal to pull this off with custom internal kernel changes, so there’s no particular reason to doubt them. (I didn’t find any clear statement from AWS on shared cores, but they already have their custom Nitro hypervisor, so plausibly they have a custom modification.)
Unfortunately, the current docs applicable to KVM don’t provide any good solution for a cloud VM provider other than disabling HyperThreading: https://www.kernel.org/doc/html/latest/admin-guide/l1tf.html
Am I wrong about this?
I too would like to know more about the hyperthreading story. We have multiple internal deployments of openstack and vmware that would suffer if we have to disable HT. Did Linode disable HT?
I am very happy with Linode being able to live migrate things with no downtime to customers. That is a massive improvement over the past migration queues.
Thanks for implementing these security fixes.
The new “live” migrations is certainly interesting – is this a new feature that you’re now able to use? It’s certainly much less painful than existing migration queues and forced downtime.
Futhermore, will live migration be introduced for other server moves, such as upgrades and downgrades?
Our current plan for L1TF mitigation is to disable HyperThreading.
Yes, live migrations are a feature that we are now able to use. We are evaluating the different use cases for this one, but currently it cannot be used for upgrades/downgrades with plan resizing.
Thanks for keeping us informed and patching the hosts. We appreciate the effort and due diligence. I’m sure these projects at large scale are never fun.