2016年1月5日，我们 发布了一个密码重置在我们对三个客户账户的未授权访问进行调查期间，我们向所有Linode客户发出了密码重置。我们一直在与联邦当局合作处理这些问题，他们的刑事调查正在进行。今天，我们将分享我们的调查结果和我们聘请的第三方安全公司的调查结果，以协助我们进行这项调查。
在深入了解之前，我们想向你保证，你的账户信息是安全的。我们发现只有三个客户受到这一事件的影响，并且已经直接与他们解决了这些问题。
发生了什么
这是对两项独立调查的复杂回顾，一项在7月，另一项在12月。虽然这两起案件有相似之处，但我们没有证据支持这两起案件有关联。尽管如此，以下是我们了解到的事件的完整时间表。
7月9日，一位客户通知我们，他们的Linode账户受到了未经授权的访问。客户了解到，一个入侵者在收到确认他们的一个Linode的根密码重置的电子邮件通知后，获得了对他们账户的访问。我们的初步调查显示，未经授权的登录在第一次尝试时就成功了，并且与正常活动相似。
7月12日，由于预计到执法部门的介入，客户跟进了一个保存请求，要求保存一个与被认为涉及未经授权的访问的IP地址相对应的Linode。我们尊重该请求，并要求客户向我们提供任何额外的证据（例如，日志文件），以支持Linode是恶意活动的来源。客户和执法部门都没有跟进，由于我们在没有可能的情况下不会检查客户数据，所以我们没有分析保存的图像。
在同一天，客户报告说，账户被访问的用户在几周前丢失了一个移动设备，里面有访问账户所需的2FA凭证，并解释说，该设备的所有者在一段时间后试图远程擦除该设备。此外，该用户使用了一个弱密码。考虑到这些信息，并且没有证据支持凭证是从Linode获得的，我们没有进一步调查。
12月9日，一位独立的安全研究人员与我们联系。该研究人员声称正在追踪一个从许多其他服务提供商那里盗取凭证的人。该研究人员想让我们知道，这个人可能已经试图使用这些被盗的凭证来登录我们一些客户的账户。
我们初步调查的结论是，所提供的IP事实上被用来在第一次尝试时登录到三个账户。换句话说，用户来到Linode Manager的登录页面，拥有必要的凭证来登录，就像任何普通用户那样。同一天，我们联系了这些客户，并从他们那里得到确认，这些活动是可疑的。我们还确认，这些账户都没有启用多因素认证，而且都使用了弱密码。
12月13日，我们开始了必要的舰队范围内的Xen 安全咨询（XSA）维护，在当地夜间24小时重新启动服务器。虽然与调查无关，但这种情况一直持续到12月18日，是一个重要的资源限制。
12月14日，尽管我们没有发现任何入侵我们基础设施的证据，但我们开始采访第三方安全公司并与多个执法机构联系。我们还将所有可用的内部资源用于这项工作，并开始仔细检查我们的环境，以确定任何滥用或误用的证据。
12月17日，由于本案与7月份的案件有相似之处，我们重新审理了7月份的案件，并认为我们现在有足够的理由审查该调查中保留的图像。
Linode使用 TOTP来提供双因素认证。这是一种算法，使用存储在我们的服务器和客户的双因素认证应用程序（如谷歌认证器）之间共享的秘密密钥。该算法生成一个对时间敏感的代码，用户在登录时提供这个代码作为额外的认证组件。我们在将这些密匙存储在我们的数据库中时对其进行加密。
在检查了我们7月调查的图像后，我们发现如果提供TOTP密钥，软件能够生成TOTP代码。我们发现了实现我们用来保护TOTP密钥的解密方法的软件，以及我们用来加密的秘密密钥。我们还在bash 历史中发现了成功生成一次性代码的命令。虽然发现的证书与12月的任何未经授权的Linode Manager登录无关，但这一信息的发现大大改变了我们调查的严重性。
12月21日，我们的第三方安全合作伙伴加入了调查。该团队进行了取证分析，以确定任何可能允许入侵者访问我们数据库中的客户凭证的未经授权的系统级活动。该团队还搜索了网络应用程序滥用的证据，这将提供从一个Linode经理账户到另一个账户的横向移动。此外，该团队启动了一个有针对性的漏洞评估，目的是确定一个可能的攻击载体，以获得对Linode数据库的访问。
12月25日 针对我们基础设施的DDoS攻击开始了.虽然我们没有任何证据支持这些攻击与未经授权的访问事件有关，但这些攻击使得我们必须从调查中抽出资源。这一点，再加上员工外出度假，给我们的支持和运营团队带来了额外的挑战。
1月5日，我们的安全合作伙伴结束了调查，我们发布了密码重置。在接下来的几周里，我们的内部安全团队继续审查我们的基础设施，并制定了一个详细的计划来改善我们的整体安全。
研究结果
我们的安全合作伙伴的调查结论是，没有证据表明滥用或误用Linode的基础设施会导致客户凭证的泄露。此外，安全合作伙伴对我们的基础设施和应用程序的评估没有产生一个可以提供这种级别访问的载体。
Linode的安全团队确实在Lish的SSH网关中发现了一个漏洞，该漏洞有可能被用来获取12月17日发现的信息，尽管我们没有证据支持这一推测。我们立即修复了该漏洞。
其他被认为可以解释未经授权的访问的理论包括外部妥协，如之前提到的弱口令被用于其他在线服务，或针对这些用户的网络钓鱼攻击。
我们正在做的事情
我们正在利用我们所学到的知识对我们的基础设施进行全面改进，包括与事件无关的领域。以下是我们一直在努力的几件事：
认证微服务：我们的一些应用程序（如Linode Manager和Lish）执行用户认证。以前，这些应用程序通过直接访问我们数据库中的证书信息来执行这一功能，然后自己进行比较。我们已经开发了一种新的方法，涉及到一个仔细的安全和监控的微服务，维护所有客户凭证的所有权。在这种方法下，当一个应用程序需要用户认证时，微服务能够通过返回一个简单的 "是 "或 "否 "来验证凭证。应用程序将无法访问凭证信息。事实上，当这个微服务推广完成后，为我们的基础设施提供动力的数据库将完全不包含凭证信息。此外，客户的密码，以前是以盐化的SHA-256哈希值存储的，有数千轮，将使用bcrypt存储，并在随后的登录中无缝升级。
Linode管理器的通知：我们将努力加强客户收到的有关其各自账户活动的通知，包括对来自新IP地址的登录尝试和登录失败的提醒。
CC标记化：虽然我们的调查没有发现信用卡信息被访问的证据，但我们正在利用我们的支付处理器的标记化功能来消除与存储信用卡信息有关的风险。
政策：我们一直在制定源自NIST框架的多种政策，主题包括清洁桌面和密码标准。一项重要的新政策是为基础设施的敏感元素创建 "安全区"，如我们的数据库和认证服务器。这些努力的结果是大大减少了能够接触敏感系统和数据的员工数量。
雇用：除上述变化外，我们正在 聘请一位高级别的安全专家来加入我们公司，领导一个更大的全职安全工程师团队。这个团队不仅将确保我们遵循当前的最佳实践，还将扩大我们的书面政策，正式确定我们的供应程序，并从根本上确保我们的政策得到流程和问责制的支持。
新的 Linode API： 我们最重要的长期战略是重写传统的 ColdFusion 代码库，让我们有机会重新开始，并应用我们在过去 13 年中吸取的经验教训。为此，我们一直在构建一个新的 Linode API，它是无状态、RESTful 的，并在Python 中实现。在过去的几个月中，我们一直在为此而努力，并将在未来几周内公布新 API 的公开 alpha 版。
开源 Linode 管理器： 这一新的 API 将成为未来所有功能的基础，包括将取代当前管理器的开源 Linode 管理器。
展望未来
我们认识到，在沟通和透明度方面，我们还有改进的余地。尽管有XSAs和整个12月的持续DDoS攻击，我们应该更早地将DDoS攻击和这一安全事件的性质和程度传达给我们的客户。说我们当时的资源有限是一个公平的评估。尽管如此，我们本可以做得更好，并在此后进行了程序上的调整，以确保在未来像这样的重要事件中指定一名团队成员，以促进与客户的频繁和透明的沟通。
我们非常感谢在这些活动中一直支持我们的客户。我们听到了你们的建议，感受到了你们在过去几个月中提供的支持。要知道，我们会继续倾听你们的反馈并采取行动。
最后，我们会说，如果我们让你失望了，我们非常抱歉。我们重视你对我们作为你的主机供应商的信任，并致力于每天赢得这种信任。我们希望这里提供的细节能澄清一些错误的信息，并表明我们愿意解决改进的机会，做正确的事情，并增加与你，我们的客户的沟通和透明度。
评论 (17)
*”…we are incredibly grateful for the customers who have supported us throughout these events… [we] felt the support you’ve provided over the past few months… We value the trust you’ve placed in us…”*
Nice sentiments, but words are cheap. You could have shown your appreciation by giving people some money off their bill for the period in question
If there is to be a new Linode Manager, please please PLEASE do not try and “modernize” or “streamline” the interface like Namecheap did. I like my utilitarian and functional Manager.
“We found software implementing the decryption method we use to secure TOTP keys, along with the secret key we use to encrypt them. We also found commands in the bash history that successfully generated a one-time code.”
How do you explain the presence of software using the decryption method you use to secure TOTP keys, along with the secret key you use to encrypt them?
There are always some bumps on the road, sometimes these are big and hard to pass thru, but I think that you’re on the right track.
Keep up the good work 🙂
The most significant take away for me is that Linode is becoming more transparent. Please continue down this path.
Good news for the new Linode manager, is there any ETA?
Long-term customer here. I appreciate all the effort made during the difficult time over the holidays. However, I have asked for 2FA by SMS rather than google authenticator (or in addition to), and was told it will not be happening. It seems that this breach could have been avoided at least in part if 2FA by SMS had been used. The user who lost his phone would have moved his phone number to his new phone as soon as it was replaced. The entire compromise of the TOTP algorithm would not have been possible. So my request, and recommendation still stands; give us 2FA by SMS if we want it.
I know how difficult these situations can be and wish you all the best in your continued growth and improvements. Unfortunately, I’ll be switching to another provider after reading this. I stuck around waiting for an explanation that I hoped would satisfy my concerns and this certainly doesn’t do that… There is reasonable evidence that your story regarding the cell phone is incorrect. In my opinion you should have had a senior security director years ago. Given the nature of the July incident, that image going unexamined is mind boggling and no explanation is given about how the 2FA crypto keys could have ended up on that system. It sounds entirely plausible your infrastructure got breached in July or earlier by a skilled attacker and the evidence is simply not there months later.
It sounds like you’re moving in the right direction, but those are some seriously poor decisions by those in charge and I’ve lost confidence that further mistakes won’t be made.
I like the current Linode manager. It’s simple and fast and clean. Please don’t change it too much. (Don’t go all Namecheap on us)
I’m a longtime Linode customer as well. After reading up on this latest incident I am seriously considering moving my infrastructure to another provider. I’ve been through a number of these cases with Linode and so far have been unaffected and given them the benefit of the doubt. I completely understand that threats are uncovered and exploited and that’s a risk you take with any provider. This isn’t about me taking a knee jerk reaction. I’m fully aware that other providers have been exploited as well but Linode have had multiple exploits and that is reason for concern.
In this post you state that you found software on a client’s VPS generating login credentials using a very private piece of your data and a piece of data that can be used to decrypt other customers data and you have no idea how it arrived on a clients machine and you don’t really seem bothered by that or at the very least you gloss over it. Have you audited every other instance to ensure they’re not also capable of generating keys/decrypting data? From my knowledge of this, the client in question was PagerDuty who alerted you after their own intrusion detection system picked up on the login. By your own account the initial illegal access looked like a legitimate login to your systems and didn’t raise any alarms. You also state that you don’t inspect customers instances without probable cause. So we have to assume that there might be other malicious software running in your infrastructure that you are not aware of and it’s allowing access that isn’t raising alerts.
You say that only three customers were accessed under suspicious circumstances. On that face of it that looks like a small number, considering you have a large client base. However, from what I can gather these were large and notable customers PagerDuty and WPEngine, I believe. That, to me sounds less like a percentage risk and more like plucking high value targets. I’m only a small customer so it’s probably more out of luck that I haven’t been affected rather than my details not actually being available.
Linode has offered a product that’s very valuable to me and my business. It’s provided reliable hosting and features. I chose them because I felt they would be able to support any application that I need to grow without the fuss of moving providers. I no longer have that confidence. The previous attack against Linode resulted in credit card loss and it’s only now that you are looking at tokenizing credit cards and moving to bcrypt for hashing?
I do appreciate that resources get stretched and trying to pick out relevant data and act on it is really difficult in a growing business. But the lack of security awareness at Linode has got to a point not where I feel I can’t trust it with my business which is sad because my experience with them has been positive, well if you put aside data breaches, credit card loss and their private keys ending up on clients’ VPSs and inability to communicate.
“We also found commands in the bash history that successfully generated a one-time code”
I don’t get how you saw this and the investigation concluded that “the security partner’s assessment of our infrastructure and applications did not yield a vector that would have provided this level of access.”
I mean, I don’t understand why someone have a bash on your systems, how he achieved that and what changed so he can’t do it in the future
Just FYI, I’ve been forced to spend considerable ongoing time and money investigating other services and finally choosing a viable non-Linode. This is money that could have been spent on better things, even if it had just been more Linode servers. Outside 3-4 work weeks redirected, my computing budget doubled even after whittling down my 4 Linodes to two.
You might consider joining forces with a cooperative and reputable industry counterpart in an effort to provide more viable business recovery scenarios for your mutual customers. In my case, now I have both a dedicated server vendor and Linode, a solution that is not neither pleasant nor economical. Each has complementary benefits.
I hope that your security efforts prove successful and I concur with the comments that should you change the inside workings of the Linode manager, it should remain simple and shun any feature that is only cosmetic.
I agree with the poster above that Linode appears to be going down a path of greater transparency. I’m extremely happy to see this.
I also believe that full disclosure and admitting fault where necessary is a very honorable thing, and you are to be commended for what you’ve said here.
Odd, all this transparency is making some people think there is a lack of security knowledge at Linode. Guys, you’d have this or worse at practically any other company.
Sounds like almost this entire thing boiled down to a few customers who have no clue about security to begin with. Weak passwords and a mobile device that doesn’t have a screen code to lock the phone. Sigh, sorry Linode had to go through all of that because of a few lazy people, but it does look like they’ve learned a lot and are improving and moving forward.
Kneejerk reacting will only put you with someone else, that probably isn’t doing the same thing as Linode.
Keep up the great works guys and I’m glad to see the improvements.
Well done, Linode!
If you are going to update the Linode Manager API, *please, please, please* give us the ability to do programmatic snapshots.
Thank you.
I greatly appreciate the explanation and transparency. So very different than other companies. This is the kind of thing that creates customer loyalty.