12月25日から1月5日までの12日間で、Linode はインフラ のすべての主要な部分に対して100以上のサービス妨害攻撃を受け、その中には何十万人ものLinode のお客様のサービスを著しく中断させるものもありました。前回の更新に続き、どのように攻撃を受けたのか、また再発防止のために何をしているのかについて、もう少し詳しくご説明したいと思います。
基本的に、攻撃者はスタックを大まかにこの順番で上に移動していきました：
- 一般向け公開側のウェブサイトに対するレイヤー7("400 Bad Request")攻撃
- 当社のウェブサイト、認証ネームサーバー、その他の公共サービスに対するボリューム攻撃
- Linode ネットワーク インフラに対するボリューム攻撃
- コロケーションプロバイダーのネットワークインフラに対するボリューム攻撃
攻撃のほとんどは単純なボリューム攻撃でした。ボリューム攻撃は分散型サービス拒否(DDoS) 攻撃の最も一般的なタイプで、ゴミのトラフィックが大量に IP アドレスに向けられ、意図した被害者をインターネットから一掃します。これはレンタカーを使って意図的にトラフィック渋滞を引き起こすのと仮想的に同等のものであり、この種の攻撃の普及により、世界的に数千億ドルの経済的損失をもたらしています。
通常、Linode は毎日数十件のボリューム攻撃を顧客に向けて受けています。しかし、これらの攻撃が広範囲のLinode ネットワークに影響を及ぼすことはほとんどありません。IP アドレスが"ブラックホーリング" されると、インターネットはその IP アドレスを宛先とするすべてのトラフィックを停止することに同意し、良いトラフィックも悪いトラフィックもそのアドレスに到達できなくなります。Linode のように何十万もの IP を持つコンテンツ・ネットワークにとって、ブラックホーリングは、"手を救うために指を切り落とす" 能力、つまり、他の人をオンラインに保つために攻撃されている顧客を犠牲にする能力を与えてくれる、鈍重ではありますが重要な武器です。
ブラックホーリングが失敗する場合があります。それは標的とされている IP、例えばインフラ の重要な部分が、他の IP をダウンさせずにオフラインにすることができない場合です。その例として通常思い浮かぶのは、API エンドポイントや DNS サーバーのような"サーバーのサーバー" で、他のインフラ の基盤を構成している場合です。攻撃の多くは当社の"サーバ・オブ・サーバ" に対するものでしたが、当社にとって最も緩和が難しいのは、当社およびコロケーション・プロバイダーのネットワークインフラに直接向けられた攻撃であることが判明しました 。
セカンダリ アドレス
当社のネットワークインフラ に対する攻撃は比較的単純でしたが、それに対処することはそうではありませんでした。歴史の産物として、当社では顧客を個々の/24サブネットに分割しています。つまり、当社のルーターは、顧客がネットワークゲートウェイとして使用するために、これらのサブネットのそれぞれの内部に "セカンダリ" IPアドレスを持たなければならないということです。
時間が経つにつれ、ルータは何百ものセカンダリアドレスを蓄積し、それぞれが攻撃の標的となる可能性があります。もちろん、当社のルータが直接攻撃を受けたのはこれが初めてではありません。通常、ブラックホール広告を送信するために特別な措置がとられており、コアにブラックホールを設けることなく、顧客のトラフィックを通常通り通過させながら攻撃を停止させています。しかし、何者かがルーター上の何十もの異なるセカンダリIPを迅速かつ予測不能に攻撃するというシナリオには、私たちは備えていませんでした。これにはいくつかの理由があります。第一に、ネットワーク機器への攻撃を緩和するには、ネットワークエンジニアによる手動の介入が必要で、時間がかかり、エラーが発生しやすいこと。第二に、当社のアップストリームプロバイダは、エラーが発生した場合の損害の可能性を制限するために、限られた数のブラックホール広告しか受け付けることができませんでした。
数日間、攻撃者との駆け引きを繰り返した結果、コロケーションプロバイダと協力してセカンダリアドレスをすべてブラックホール化するか、ブラックホール化が不可能なトランジットプロバイダのネットワークのエッジにトラフィックを落とすかのどちらかを選択することができました。
クロスコネクト
コロケーショ ン・プロバイダーを標的とした攻撃も同様に単純でしたが、それを軽減するのは想像以上に困難でした。ルータが直接攻撃できなくなると、コロケーション・パートナーとそのトランジット・プロバイダが次の論理的なターゲットとなりました。クロスコネクトは、一般的にインターネット上の2つのルーター間の物理的なリンクと考えることができます。この物理的なリンクのそれぞれの側には、2つのルーターが相互に通信できるようにIPアドレスが必要で、そのIPアドレスが標的となりました。
我々のインフラと同じように、この攻撃方法はそれ自体が目新しいものではありませんでした。この方法が非常に効果的だったのは、攻撃の迅速性と予測不可能性でした。多くのデータセンターでは、アップストリームネットワーク内の数十の異なる IP が攻撃を受け、コロケーションパートナーとそのトランジットプロバイダーの間で、維持するのが困難なレベルの集中と調整が必要となりました。これまで一番長時間のサービス中断 - アトランタで30時間以上 - はLinode のスタッフと、時には 4 度離れた場所にいる人々との間のコミュニケーションが頻繁に途切れたことが直接の原因となっています。最終的には、いくつかの頑固なトランジットプロバイダーがインフラ が攻撃を受けていることを認め、攻撃を阻止するための対策を講じることに成功したため、この攻撃のベクトルを完全に封じることができました。
教訓
個人的にはこのようなことが起きてしまったことは恥ずかしいのですが、辛い経験からいくつか学びました。
レッスン１：中間サービスに頼るな今思えば、IPトランジットのコロケーションパートナーに頼っていなければ、より長い時間の停止は避けられたかもしれないと考えています。その具体的な理由は 2 つあります。第一に、コロケーションプロバイダーが実際よりも多くの IP トランジット容量を持っていると思われていたケースがいくつかありました。Linode に向けられた攻撃トラフィックの量が非常に多かったため、コロケーション・プロバイダーは、攻撃が終了するまでLinode ネットワークとの一時的なピアリングを解除せざるを得なかったことが何度かありました。第二に、より複雑な攻撃のいくつかを緩和するためには、異なるティア 1 プロバイダの上級ネットワークエンジニアが直接関与しなければなりませんでした。休日の週末の午前 4 時には、コロケーション・パートナーは、私たちと問題を解決してくれる人たちとの間で冗長な障壁となっていました。
レッスン2：より大きな攻撃を吸収する Linodeの IPトランジットのキャパシティ管理戦略はシンプルです。ピーク時の1日の利用率が全体のキャパシティの50%に近づいてきた時がリンクを増やす時です。この戦略はキャリア・ネットワークでは標準的なものですが、当社のようなコンテンツ・ネットワークでは不十分であることがわかりました。実際の数字を出してみると、当社の小規模なデータセンター・ネットワークの総IPトランジット容量は40Gbpsです。これは多くの方には大容量に見えるかもしれませんが、80Gbps の DDoS の場合、ブラックホールをかけることができないため、20Gbps のヘッドルームしかないため、攻撃の間にパケットロスが発生してしまいます。
レッスン3：顧客に現状を知らせる失敗した時にそれを認めることが重要で、攻撃の初期段階での詳細なコミュニケーションの欠如は大きな失敗でした。いざという時に詳細な技術的な最新情報を提供するのは、現状に詳しい人でなければできません。通常、そのような人たちは、火消しの仕事をしている人たちでもあります。事態が落ち着き、広報活動を見直した結果、言葉遣いの悪さやパニックを起こすことを恐れて、必要以上に曖昧な表現をしてしまっていたという結論に至りました。これは間違っていました。今後は、このような大規模なイベントの際には、指定された技術的なキーパーソンが詳細なコミュニケーションを担当することになります。さらに、当社のステータスページでは、"更新情報を購読する" リンクを介して、お客様に電子メールやSMSのテキストメッセージでサービスの問題を通知することができるようになりました。
私たちの未来は過去よりも明るい
これらの教訓を念頭に当社がどのように改善しているかを知っていただきたいと思います。まず、簡単なことですが、DDoS 緩和対応を実施することで、当社の公開サーバーに対する攻撃の脅威を軽減しています。ネームサーバーは現在 Cloudflare によって保護されており、ウェブサイトは強力な商用トラフィックスクラビングアプライアンスによって保護されています。さらに、これらの休日の攻撃の際に導入された緊急時の緩和技術を恒久的なものにしました。
これらの対策により、連休中に起きたような攻撃は二度と起こらないと確信しています。しかし私たちはもっと多くのことを行う必要があります。そこで本日、Linode は当社のデータセンターの接続戦略全体をオーバーホールし、200 ギガビットのトランジットおよびピアリング容量を主要な地域の拠点から当社の各拠点にバックホールすることを発表できることをうれしく思います。
ここでは、最初に容量アップグレードを享受するニューアークのデータセンターに対する今後のインフラ の改善点の概要をご紹介します。
このアーキテクチャの主役は、すでに構築を開始している光トランスポート・ネットワークです。これらのネットワークにより、この地域で最も重要なPoPのいくつかに完全に多様なパスが提供され、Linode は何百もの異なるキャリアオプションと何千もの直接ピアリングパートナーにアクセスできるようになります。既存のアーキテクチャと比較して、このアップグレードのメリットは明らかです。当社は、インフラ 全体を、インターネットのエッジに至るまで制御することになります。これは IP トランジットを中間サービス業者に頼るのではなく、サービスを提供するために我々が依存している通信事業者と直接提携することを意味します。
さらに、Linode では現在利用可能な帯域幅が 5 倍になり、適切に緩和されるまで、非常に大規模な DDoS 攻撃を吸収することが可能になります。将来的に攻撃の規模が大きくなっても、このアーキテクチャ、新たに大規模な設備投資をすることなく、その需要を満たすために迅速に拡張することができます。
最後に
最後に、心からのお詫びを申し上げます。お客様のために重要なインフラをホストする企業として、私たちはそのインフラをオンライン状態に維持する責任を任されています。私たちは、この記事の透明性と先進的な考え方がその信頼の一部を取り戻すことを願っています。ご理解とご支援のご協力に感謝いたします。私たちの多くはこれらの容赦ない攻撃によって私たちの休日を台無しにし、それを私たちの愛する人たちに説明することは困難なことです。コミュニティからの支援は本当に助けになりました。以下に質問やコメントを投稿することをお勧めいたします。
コメント (67)
Thanks for your great work. My VPS was running well during these days.
Good postmortem analysis – thanks for being candid.
Thanks for being honest and forthcoming about this and the issues you addressed-both on the technical and PR sides-as well as the steps you are taking to better your company.
Kimo.
You people are awesome and have great stamina. We are satisfied customer from Pakistan.
I’ll never stop buying linodes!!
You guys are are rock stars in my book, and I appreciate the transparency. More tech companies need to live and breath that these days, or else find themselves losing the game to cheaper competitors.
While I haven’t been a fan of how some past incidents were handled, I still give Linode a 5-star rating. Good job!
Things happen. Those of us who network or sysadmin know that when youre fighting fires and figuring out what is going on and fielding calls from angry clients the last thing you have time for is updating everyone. Hell…you may not even know what all is going on for a couple days or more with huge attacks.
This is a good postmortem and your ability to learn and adapt and invest in your own infrastructure is why I love and continue to be a Linode fanatic.
Keep it up you guys. Sorry Christmas was such a bummer.
May the Network be with you!
Can’t thank the Linode team enough for your dedication. The livelyhood of thousands rest in your hands, I feel like this whole event further proves how well qualified you guys are to be doing what you’re doing.
The only part of this that really bothers me is the idea that if I get a DDOS, Linode is just going to blackhole me, and me alone. Doesn’t that mean that I have to give in to ransom demands from attackers?
I really appreciate this. We were waiting for this to take the decision if we will stay in linode or move away, and we are staying.
I strongly agree that being more transparent would have helped a LOT.
I’d like to know, though, when is scheduled the above change in the rest of the datacenters. I’m not using newark right now and would like to know when my datacenter will have it : )
Thanks a lot,
Rodrigo
@Mogden – for people who are attacked regularly, we suggest Cloudflare or others in the DDoS protection market. I’m not sure what the future holds on this subject, but rest assured that it really bothers us too.
Thanks for the update. Any time frame for other datacenters to be updated? My linodes are in Atlanta and we suffered almost three days of downtime.
Cheers
We had 2 linodes, one of them in Atlanta datacenter. We have not experience any issues during holidays, but I was worried though. Thanks for the explanation and amazing work. Honestly hope your family can understand the situation.
Amazing company!
Like Rodrigo, this is a huge thing to us. I was honestly feeling that it was going the usual corporate way with silence and deniability, just waiting for the furore to die down. It really makes a difference to hear not only the details of the response/mitigation activities, which we appreciate, but also acknowledgement of the position we were put into when communication was sparse.
It goes a long way.
Thanks again.
Mark.
Great to hear we could help you get protected.
swiner@cloudflare.com
@mogden – if your the one being ddos’d then you deserve to be blackholed. I dont pay for my linodes for you to be targetted with a ddos and mine linodes taken down!!
Thank you for the analysis and a break down of what took place, and most importantly, thank you for being honest with customers!
Cheers!
I’m obviously a huge fan of Linode, but I wonder if this attack will force them to re-evaluate their “3 strikes” policy towards hosted sites which come under DDoS attack. As this attack should have taught them, it’s indiscriminate, and there’s not a whole lot a small website owner can do to mitigate it. We rely on Linode to be able to deal with this, and punishing the victim is hardly a fair solution.
And attacks started minutes after posting updates. http://status.linode.com/incidents/mkcgnmjmnnln
I’ve a message for Linode especially Chris, please invest more and more on infrastructure if you want to stay in the game otherwise, you’ll be overtaken by heavily funded startups in this domain. We know you have innovative mind and excellent technology but this alone is not sufficient for you to win in this domain. I like performance and flexibility of Linode but moved to DO just because I needed to setup my stuff at Japan and Singapore data-centers and Japan DC is sold out. 3 out of 6 locations are sold out and you are not yet expanding? How will you compete?
Come out of your box and look at your neighbors. It was painful to move to Digital Ocean for me but I had to take this decision. I am still using Linode for some of my stuff will continue using it until I need redundancy or you expand.
There’s nothing that I love more than the amount of technical detail that you provide to us on these cases, and even with some minor updates.
I love being a Linode customer, no DDoS will get that away from me 😀
Thanks for this post, Alex. This was a rough period for everyone involved and affected but I am extremely impressed by Linode making the effort to hopefully prevent the same scenario from happening again.
There were many lessons to be learned from this – both for Linode and for customers.
Linode appears to have realized what they needed to do and that is fantastic. Instead of saying sh*t happens and going about business as usual you are actively working to make sure it doesn’t happen again. Well done.
We (customers) need to cover our own bases too. For anything critical or even slightly important you need to have a plan in place in the event of a Linode outage (regardless of the reason).
I have now split some of my services and are far better placed to recover quickly in the event something like this were to happen again. Linode had always been so reliable that I got complacent. Lesson definitely learned.
In my case my costs have now increased as I am now paying other providers in addition to what I have and will continue to pay Linode, but the ability to keep some important services online is worth it.
Thank you to everyone at Linode for your hard work and for looking out for your customers.
Some of our big clients suffered with the downtime on those days but, with several VPS and more online each day, we never accepted any offer from others players. This kind of behaviour make us confident with the team and give us peace of mind that we’re in good hands.
Thank you for the update and respect with your customers.
Hostcare Internet
Thank you for being open, good luck with your new defences and I hope that you catch up on your family time!
Linode user here. Thanks for the transparency. I wasn’t directly affected but I appreciate the openness on the issue. It’s a welcome change to most companies now. I plan to keep using Linode just because of how cool you all handled the situation. Keep up the good work!
Cloudflare will probably help with your DDoS but they aren’t infallible as any other vendor.. But what happens when they get hit really hard themselves? I’d recommend getting a second DNS provider.
See Also: https://blog.thousandeyes.com/ultradns-ddos-affects-major-web-services/
https://blog.thousandeyes.com/ultradns-outage-october-2015/
I was beginning to wonder if such a note would arrive. The explanation is useful and I’m feeling as though things are safer than before.
Thank you for being transparent about what happened. That was a truly hellish attack. Getting slammed with a sophisticated and highly targeted 80 Gbit DDoS is stressful for any network admin and I’m glad that Linode succeeded in weathering the storm.
I am really impressed with way you have handled this whole situation, your company’s honesty and explanation is more than anyone could have expected. I’m sure there were many hours invested, not only in locating and fixing the problem on top of adding the double protection; but even in your letter to your customers. I hope all your customers are as loyal to your company as you have been with them. Way to step up your game, keep up the good work. Wishes for much more success……
Thank you very much for the detailed breakdown of what went wrong and what you plan to do to prevent this in the future. I have to say though, technical reasons and justifications aside, Linode has a lot to learn in regards to communication. I know you acknowledge that in your blog post but for many people (myself included) it’s too little way too late. It’s taken you 30 days to write a blog post that could’ve been written in hours. For 30 days people have been sitting on the fence wondering exactly what you guys are doing and whether or not they should jump ship. For many people (myself included), the absence of this response and the overall feeling that it has been so long since you said you were going to provide an update, that honestly you were just going to push this to the side and hope it went away, has directly contributed to Linode losing a significant amount of business from us.
I don’t want my response to turn into some Linode bashing post, but I want you to be aware that your failure to provide sufficient information and responses is the biggest problem here – for me, at least. It hit your reputation hard and caused us to lose a significant amount of trust in your company and services. DDoS attacks happen, and we know you guys were working extremely hard to deal with those. You reminded us often enough in your status updates. What we really wanted to know was that the worst was over and that you identified your weaknesses and were addressing those. The longer we had to wait for this information, the less trust we had/have in you.
I’d like to end this on a more positive note. All of the above said, your services are fantastic overall and I’d love to come back to Linode in the future, once you’ve performed all of the changes you have mentioned here. Just please, improve on your communications!
Long-time Linode customer…I wasn’t affected by the outage, but I’m really glad you’ve taken the time to write up what happened. Thanks for being transparent and generally awesome.
Alex, this caught my attention: “… requiring a level of focus and coordination between our colocation partners and their transit providers which was difficult to maintain.”
How did you structure this communication? What tools / technologies did you use or tried to use?
This is a nicely put article. I only have amazing things to say about Linode and its staff. Awesome post!
As a long time customer and a fellow network administrator I just wanted to say that I do really appreciate all your hard work. Respect.
Sounds an exciting project Alex, good luck!
Any news on continued security farces at Linode? and ‘The Best Practices not invented here’ approach.. For example to reset 2FA
—
Should you need us to disable your Two-Factor Authentication, the following information is required:
An image of the front and back of the payment card on file, which clearly shows both the last 6 digits and owner of the card.
An image of the front and back of the matching government-issued photo ID.
—
A) Photoshop CC in 2 mins, you have no idea what my CC should look like.
B) You can’t verify government ID so say 5 minute photoshop.
Woohoo for 2FA, known as 2 f… alls
Thanks for the update, and letting us know that things will be better handled in the future. Both technically and on the communication front.
Any idea who attacked and why?
Linode – you are the best. Thanks for your service.
Thanks for the update. As a long time linode customer, it is appreciated.
For you guys complaining about being kicked out in case of a DDoS, I recommend getting DDoS protection for your linodes. There are a lot of cheap options right there that can be integrated easily.
Some one recommended CloudFlare and they are great. You can also look at Sucuri:
http://sucuri.net/website-firewall/
Or Incapsula:
https://incapsula.com
Both great products and solutions. Stay safe!
200g? this years ddos was 800gbps…
good postmortem. now can you explain what happened with the “leaked” credentials and the fact that we had to reset the passwords.
thank you
These attacks could happen to anyone and any provider. Keep up the good work!
Great article and the right way to handle these kinds of problems. Transparency and constructive retros are the way to go.
I think you did great job considering the size of the attack. That’s why continue to use Linode for my virtual machines. Thank you for your support and keep up the good work.
Thank you for the clear and concise explanation. I look forward to you rolling out your upgrades and continue to be a happy customer with Linode.
Cisco routers, seriously?
Juniper high end routers take a gigantic steaming dump all over Cisco.
@Jake that’s essentially what ASRs are 😉
If you want to do it on the cheap side and be safe, get some cheaper / best equipment from huawei (give them a call). You might think the Chinese cannot be better than Cisco, but Cisco is now also made in China. Also I’m sorry, but you need some Ddos protection (expensive). You cannot just nullroute your costumers… you have to protect them. If the cheap OVH company can do it, why can’t you…
Looks like you guys need to hire someone with real experience in network engineering (worked at ISP level), not just some cheap undergraduate out of university.
You need to rely more on anycast, have reserved capacity, etc.
After reading this, I would not host my sites on linode. You guys look amateur (sorry).
I appreciate this honest insight, but I’ve moved back to a local server since these attacks made access to my Linode difficult or impossible, and always-on, always-accessible was my main reason for moving to Linode in the first place. Sorry, and better luck in the future.
I like the transparency, even delayed. I like that you’re taking steps. I DON’T like that your “security appliances” block ALL ICMP packets including the “Packet Too Big” messages required for path MTU discovery and breaking my ability to access the Manager over my VPN.
Buying blended internet direct from your colo provider is a bad idea (as it seems you have learned the hardway)
You should be getting your transit direct from diverse carriers… this is networking 101
Love the armchair quarterbacks giving their input. Now, for you QBs, where is your massive company you are running and making decisions and learning lessons from? Oh you don’t have one and you don’t work for one? Sit back and let Linode do their job, they are by far the best provider out there. The cost of this type of infrastructure is gigantic and you wanna-be QBs have no idea what it takes to run a business.
Great job Linode. I know I’ve made the right choice by using you.
Excellent. I knew you guys were “on it”. I really appreciate the detail you provided.
Thank you for releasing this honest and detailed report
Regarding CloudFlare, did you shop around for any other DNS DDOS protection services? The reason I ask is because CloudFlare happily caches too many dodgy websites. Some sources that may be of interest:
http://news.netcraft.com/archives/2015/10/12/certificate-authorities-issue-hundreds-of-deceptive-ssl-certificates-to-fraudsters.html (large number of phishing certificates issued by CloudFlare)
http://www.crimeflare.com (non-profit that investigates CloudFlare and its customers)
I appreciate the update, but i find a bit late too.
Also i don’t really get why Mr. Forster signing this post?
And don’t get me wrong, i have nothing against him, i don’t doubt his intentions or knowledge.
But i expected a statement from someone from the top of the food chain . This was also one of my main problems when the events happened, its like nobody cares from the top management, until one of the engineers realized that they can’t be silent anymore.
I still have that feeling, and is pretty alarming .
It’s time to move to IPv6-only internet. Attacking a single address will become impractical if a host can have millions of them changed automatically in an unpredictable way.
Appreciate the info.
It is a minor point, I know, but status.linode.com should either be un-available over https, or have its own cert.
try this in chrome…
https://status.linode.com
Thanks Linode Team for acknowledging your challenges, and courageously taking adaptive actions 🙂
Great job! Didn’t know such a story ongoing since my site was on all the time. Really appreciate all the hard work of LINODE support team!
Thank you for the very interesting update. Best of luck for the future.
I’m also quite curious on who could benefit from such attacks in the first place.
I am using Cloud Flare to protect the blog from DDOS attack, is there any other best application available to replace cloudflare? Is there a way to stop the DDOS or brute force attack for wordpress sites?
Great write up & good to see such honesty and transparency. I think it is important for readers of this to understand that DDoS attacks can affect anyone at any time on any host. Obviously when you are on the receiving end of a nullroute it is not nice, but It’s important to note though that providers do not want for you to have downtime, but if a DDoS directed at you is affecting other customers and you don’t have some form of mitigation, there is seldom any other option than to take this action. As they said, ‘cut off a finger to save the hand’. I’m quite sure that if someone else is being DDoS’d that you would prefer to see them nullrouted than have your own service impacted, so that has to work both ways in my eyes.
It’s important to look at the issue objectively – DDoS attacks are not going to go away and really if you have concerns around protection then this does mean paying for a mitigation service, especially if outages will be more costly than the monthly sub.
@Srinivas – You’ll need a CloudFlare business plan for DDoS attack mitigation. Simply being behind CloudFlare on a free plan won’t give you this protection, and there isn’t another service that I am aware of that provides free DDoS protection without at least having some other paid service. Keep in mind that CloudFlare isn’t an application, but rather a service which is totally separate from your Wordpress sites. If you want to run something locally to stop a brute force attack then have a look at a plugin such as Wordfence, which is very effective. Another good plugin is iQ Block Country which uses GeoLocation – you can lock down your back end to whitelisted countries only. Plugins are not infallible, but they definitely add extra security. Another good way to stop brute force attacks is by not using obvious account names for the administration area of your site…lots of tools will try to brute force on usernames like ‘admin’ – as with any security approach, it’s all about the layers!
As a final note, I do always find it interesting when posts like this attract the critics who dish out ‘advice’ about how X and Y should have already been done, or that they are amateur, etc. I would like to know which fairytale jobs they have at companies that have everything 100% perfect with 100% uptime and 0% chance of outages or attacks…
Fair play Linode, tip of the cap.
Thank you for your honesty and transparency. Very very good post. Thank you for your hard work during the attacks even on holidays. Keep pushing Linode Team!
yeah thank you also for your transparency. I remember what happened, evthg gave tears and I think, as many people, we planned to move to another company. Even some days ago, I compared with AWS, reading their doc for RDS, EC2, ELB, S3 etc, but Linode, even with much less available options and possibilities if we compare to amazon, Linode stay for us a better company, with a great support and reactive, providing faster and cheaper solutions.
I started with Linode 4 years ago, I loved the service and I am not going to go away from you guys. I know how painful firefighting could be, thanks to your team for working so hard. And please do everything that could prevent this from repeating.
Hello,
on the article you said following
“our nameservers are now protected by Cloudflare, and our websites are now protected by powerful commercial traffic scrubbing appliances.”
but seems it is not anymore. did you moved away from cloudflare protection? if yes then why? many hosting giants now rely on cloudflare protection.
Thank you for this update and the recent additional high memory and $5 options.