12월 25일부터 1월 5일까지 12일 동안 Linode는 인프라의 모든 주요 부분에서 100건 이상의 서비스 거부 공격을 확인했으며, 일부는 수십만 명의 Linode 고객의 서비스에 심각한 피해를 입혔습니다. 저는 우리가 어떻게 공격을 받았는지, 재발을 방지하기 위해 무엇을 하고 있는지에 대해 추가로 인사이트를 제공함으로써 이전 업데이트의 후속 조치를 취하고 싶습니다.
기본적으로 공격자는 대략 다음과 같은 순서로 스택에 도달했습니다.
- 공용 웹사이트에 대한 Layer 7(“400 Bad Request”) 공격
- 당사 웹사이트, 권위 있는 네임서버 및 기타 공공 서비스에 대한 볼륨 기반의 공격
- Linode 네트워크 인프라에 대한 볼륨 기반의 공격
- 코로케이션 제공 업체의 네트워크 인프라에 대한 볼륨 기반의 공격
대부분 공격은 단순한 볼륨 기반의 공격이었습니다. 볼륨 기반의 공격은 가장 일반적인 유형의 분산 서비스 거부(DDoS) 공격으로, 대규모 가비지 트래픽이 IP 주소로 직접 유입되어 의도된 피해자를 인터넷에서 삭제하는 것입니다. 이는 렌트카를 사용해 의도적으로 교통 체증을 유발하는 것과 실질적으로 동일하며, 이러한 유형의 공격이 확산되면서 전 세계적으로 수천억 달러의 경제적 손실이 발생했습니다.
일반적으로 Linode는 매일 고객을 겨냥한 수십 건의 대량 공격을 확인하고 있습니다. 그러나 원격 구동 블랙홀링이라 불리는 당사에서 우리 자신을 보호하기 위해 사용하는 도구가 있기에 이러한 공격은 더 광범위한 Linode 네트워크에 거의 영향을 미치지 않습니다. IP 주소가 "블랙홀링"된 경우 인터넷 전체에서 해당 IP 주소로 향하는 모든 트래픽을 삭제하는 데 동의하여 양호한 트래픽과 나쁜 트래픽이 모두 해당 IP 주소에 도달하지 못하게 막습니다. 수십만 개의 IP를 보유한 Linode와 같은 콘텐츠 네트워크의 경우 블랙홀링은 우리 무기고에서 뭉툭하지만 중요한 무기이며, ‘손을 구하기 위해 손가락을 절단할’ 기능을 제공합니다. 즉, 다른 고객들을 온라인 상태로 유지하기 위해 공격받는 고객을 희생하는 것입니다.
블랙홀링은 명백하지만 중요한 하나의 상황에서 효과적인 완화 장치로서 실패합니다. 대상이 되는 IP(예: 인프라의 일부 중요한 부분)가 다른 IP를 중단하지 않고 오프라인 상태가 될 수 없는 경우입니다. 일반적으로 떠오르는 예시로는 다른 인프라의 기반을 구성하는 API 엔드포인트 또는 DNS 서버와 같은 "서버의 서버"가 있습니다. 많은 공격이 "서버의 서버"에 대한 것이었지만, 가장 완화하기 어려운 공격은 당사와 코로케이션 제공 업체의 네트워크 인프라를 직접 겨냥한 공격이었습니다.
보조 주소
네트워크 인프라에 대한 공격은 상대적으로 간단했지만 공격 완화는 그렇지 않았습니다. 과거의 아티팩트로서 우리는 고객을 개인/24 서브넷으로 분할합니다. 즉, 고객이 네트워크 게이트웨이로 사용할 수 있도록 라우터는 이러한 각 서브넷 내부에 "보조" IP 주소를 보유해야 합니다.
시간이 지남에 따라 라우터는 수백 개의 보조 주소를 수집했으며 각각은 잠재적 공격 대상이 되었습니다. 물론 라우터가 직접 공격을 받은 것은 이번이 처음은 아닙니다. 일반적으로 코어 블랙홀링 없이도 블랙홀 광고를 업스트림으로 전송하여 공격을 중지하고 고객 트래픽이 평소와 같이 전달될 수 있도록 특별한 조치가 시행됩니다. 그러나 누군가 라우터에서 수십 개의 서로 다른 보조 IP를 빠르고 예측할 수 없이 공격하는 시나리오에는 대비하지 못했습니다. 이것은 몇 가지 이유 때문입니다. 첫째, 네트워크 장비에 대한 공격을 완화하려면 느리고 오류에 취약한 네트워크 엔지니어의 수동 개입이 필요했습니다. 둘째, 업스트림 제공 업체는 오류 발생 시 손상 가능성을 제한하기 위해 제한된 수의 블랙홀 광고 만 수락할 수 있었습니다.
며칠간 공격자와 쫓고 쫓기는 추격전을 벌인 후 코로케이션 제공 업체와 협력하여 모든 보조 주소를 블랙홀링하거나 그 대신 블랙홀링이 불가능한 트래픽 제공 업체 네트워크의 가장자리에서 트래픽을 삭제할 수 있었습니다.
교차 연결
코로케이션 제공 업체를 표적으로 하는 공격은 간단하지만 완화하기 더욱 어렵습니다. 우리 라우터를 더 이상 직접 공격할 수 없게 되자 코로케이션 파트너 및 전송 제공 업체, 특히 그들의 교차 연결은 필연적으로 다음 목표가 되었습니다. 교차 연결은 일반적으로 인터넷의 두 라우터 사이의 물리적 링크로 생각될 수 있습니다. 이 물리적 링크의 각 측면에서 두 라우터가 서로 통신할 수 있도록 IP 주소를 필요로 하며, 이것이 표적이 된 IP 주소였습니다.
자체 인프라의 경우와 마찬가지로이 공격 방법 자체는 새로운 것이 아닙니다. 이 방법이 그토록 효과적인 이유는 공격의 신속성과 예측 불가능성 때문이었습니다. 많은 데이터 센터에서 업스트림 네트워크 내의 서로 다른 IP 수십 개가 공격을 받았기에 코로케이션 파트너 및 유지 관리가 어려웠던 트래픽 제공 업체 간의 집중 및 협력이 필요했습니다. 애틀랜타에서 30시간을 초과한 현재까지 당사의 가장 긴 중단은 Linode 직원과 때때로 우리에게서 4도 정도 떨어진 사람들 사이의 빈번한 커뮤니케이션 중단에서 직접적으로 기인할 수 있습니다. 완고한 트래픽 제공 업체가 마침내 인프라가 공격을 받고 있음을 인정하고 공격을 막기 위한 조치를 성공적으로 취한 후에야 결국 이 공격 벡터를 완전히 차단할 수 있었습니다.
교훈
개인적으로 저희는 이런 일이 일어날 수 있다는 사실에 부끄러움을 느끼며 경험을 통해 몇 가지 교훈을 어렵게 배웠습니다.
교훈 1: 중개자에게 의존하지 말 것. 돌이켜보면 IP 전송을 위해 코로케이션 파트너에 의존하지 않았다면 더 긴 중단을 피할 수 있었을 것입니다. 이에 대한 두 가지 구체적 이유가 있습니다. 첫째, 여러 인스턴스에서 우리는 코로케이션 제공 업체가 실제보다 더 많은 IP 전송 용량을 보유한다고 믿게 되었습니다. 여러 번 Linode로 향하는 공격 트래픽의 양이 너무 커서 코로케이션 제공 업체는 공격이 끝날 때까지 Linode 네트워크를 일시적으로 피어링을 취소할 수밖에 없었습니다. 둘째, 더 미묘한 공격을 성공적으로 완화하려면 여러 티어 1 제공 업체의 선임 네트워크 엔지니어가 직접 참여해야 했습니다. 연휴 주말 오전 4시경 코로케이션 파트너는 우리 자신과 문제를 해결할 수 있는 사람들 사이에 불필요한 장벽이 되었습니다.
교훈 2: 대규모 공격 흡수 IP 전송을 위한 Linode의 용량 관리 전략은 간단했습니다. 일일 최대 사용률이 전체 용량의 50%에 근접하기 시작하면 더 많은 링크를 확보하는 것입니다. 이 전략은 이동통신사 네트워크의 표준이지만 이제는 우리와 같은 콘텐츠 네트워크에 적합하지 않다는 것을 알고 있습니다. 실제 숫자를 살펴보자면 당사의 소규모 데이터 센터 네트워크가 보유한 총 IP 전송 용량은 40Gbps입니다. 이것은 많은 분들께 큰 용량으로 보일 수 있지만, 블랙홀링이 불가능한 80Gbps DDoS의 컨텍스트에서 20Gbps 정도의 헤드룸만 있을 경우 공격 지속 중 심각한 패킷 손실이 발생합니다.
교훈 3: 고객에게 상황을 알릴 것. 실패한 경우를 인정하는 것은 중요하며 공격 초기에 상세한 의사소통의 부재는 우리의 큰 실패였습니다. 위기 상황에서 자세한 기술 업데이트를 제공하는 것은 현재 상황을 자세히 알고 있는 사람만 수행할 수 있습니다. 보통 그 사람들도 긴급 수리 활동을 맡고 있습니다. 상황이 해결되고 공개적 소통을 검토한 후, 우리는 잘못된 발언이나 과도한 공포를 유발하는 것에 대한 두려움으로 인해 상태 업데이트 시 해야 했을 말보다 더 모호하게 말하게 되었다는 결론에 도달했습니다. 이는 잘못된 것이었고 앞으로 전담 기술 담당자가 이와 같은 주요 이벤트 중 자세한 의사소통을 담당할 것입니다. 또한 상태 페이지에서 “업데이트 구독” 링크를 통해 이메일 및 SMS 문자 메시지로 서비스 문제에 대한 알림을 받을 수 있습니다.
우리의 미래는 과거보다 밝습니다.
이러한 교훈을 염두에 두고 이를 실천하는 방법을 알려드리고자 합니다. 첫째, 쉬운 부분: DDoS 완화를 구현하여 공용 서버에 대한 공격의 위협을 완화했습니다. 우리의 네임 서버는 이제 Cloudflare로 보호받으며, 웹사이트는 강력한 상업용 트래픽 스크러빙 어플라이언스로 보호됩니다. 또한 연휴 시 발생한 공격 중 적용된 긴급 완화 기술이 영구적으로 적용되도록 조치했습니다.
이러한 조치만으로 연휴 중 발생한 공격 유형이 다시는 발생할 수 없다고 확신할 수 있습니다. 그래도 우리는 더 많은 일을 해야 합니다. 그래서 오늘 저는 Linode가 전체 데이터 센터 연결 전략을 재정비하여 주요 지역의 PoP에서 각 위치로 200Gb의 전송 및 피어링 용량을 백홀링할 것임을 기쁜 마음으로 발표합니다.
다음은 이러한 용량 업그레이드를 가장 먼저 받을 Newark 데이터 센터의 향후 인프라 개선 사항에 대한 개요입니다.
이 아키텍처의 핵심은 우리가 이미 구축하기 시작한 광전송 네트워크입니다. 이러한 네트워크는 지역에서 가장 중요한 PoP 중 일부에 매우 다양한 경로를 제공하여 Linode가 수백 개의 서로 다른 캐리어 옵션과 수천 개의 직접 피어링 파트너에 액세스할 수 있도록 합니다. 기존 아키텍처와 비교할 때 이 업그레이드의 이점은 분명합니다. 우리는 인터넷의 가장자리까지 전체 인프라를 제어할 것입니다. 즉, IP 전송을 위해 중개자에 의존하는 것이 아니라 서비스를 위해 의존하는 이동통신사와 직접 파트너십을 맺을 것입니다.
또한 Linode는 현재 사용 가능한 대역폭의 양을 5배로 늘려 적절히 완화될 때까지 매우 큰 DDoS 공격을 흡수할 수 있습니다. 향후 공격 규모가 증가하면 이 아키텍처는 새로운 주요 자본 투자 없이도 요구 사항을 충족하도록 빠르게 확장될 것입니다.
마무리
마지막으로 진심으로 사과드립니다. 고객을 위한 중요한 인프라를 호스팅하는 회사로서 우리는 인프라를 온라인 상태로 유지할 책임이 있습니다. 이 게시물의 투명성과 미래지향적 사고를 통해 신뢰를 회복할 수 있기를 바랍니다. 또한 여러분의 이해와 지지를 담은 친절한 말씀에 감사드립니다. 우리 중 다수는 끊임없는 이 공격으로 인해 연휴가 엉망이 되었으며, 이는 사랑하는 사람들에게 설명하기 어려운 일이었습니다. 커뮤니티의 지원이 정말 도움이 되었습니다. 아래에 질문이나 의견을 게시해 주십시오.
댓글 (67)
Thanks for your great work. My VPS was running well during these days.
Good postmortem analysis – thanks for being candid.
Thanks for being honest and forthcoming about this and the issues you addressed-both on the technical and PR sides-as well as the steps you are taking to better your company.
Kimo.
You people are awesome and have great stamina. We are satisfied customer from Pakistan.
I’ll never stop buying linodes!!
You guys are are rock stars in my book, and I appreciate the transparency. More tech companies need to live and breath that these days, or else find themselves losing the game to cheaper competitors.
While I haven’t been a fan of how some past incidents were handled, I still give Linode a 5-star rating. Good job!
Things happen. Those of us who network or sysadmin know that when youre fighting fires and figuring out what is going on and fielding calls from angry clients the last thing you have time for is updating everyone. Hell…you may not even know what all is going on for a couple days or more with huge attacks.
This is a good postmortem and your ability to learn and adapt and invest in your own infrastructure is why I love and continue to be a Linode fanatic.
Keep it up you guys. Sorry Christmas was such a bummer.
May the Network be with you!
Can’t thank the Linode team enough for your dedication. The livelyhood of thousands rest in your hands, I feel like this whole event further proves how well qualified you guys are to be doing what you’re doing.
The only part of this that really bothers me is the idea that if I get a DDOS, Linode is just going to blackhole me, and me alone. Doesn’t that mean that I have to give in to ransom demands from attackers?
I really appreciate this. We were waiting for this to take the decision if we will stay in linode or move away, and we are staying.
I strongly agree that being more transparent would have helped a LOT.
I’d like to know, though, when is scheduled the above change in the rest of the datacenters. I’m not using newark right now and would like to know when my datacenter will have it : )
Thanks a lot,
Rodrigo
@Mogden – for people who are attacked regularly, we suggest Cloudflare or others in the DDoS protection market. I’m not sure what the future holds on this subject, but rest assured that it really bothers us too.
Thanks for the update. Any time frame for other datacenters to be updated? My linodes are in Atlanta and we suffered almost three days of downtime.
Cheers
We had 2 linodes, one of them in Atlanta datacenter. We have not experience any issues during holidays, but I was worried though. Thanks for the explanation and amazing work. Honestly hope your family can understand the situation.
Amazing company!
Like Rodrigo, this is a huge thing to us. I was honestly feeling that it was going the usual corporate way with silence and deniability, just waiting for the furore to die down. It really makes a difference to hear not only the details of the response/mitigation activities, which we appreciate, but also acknowledgement of the position we were put into when communication was sparse.
It goes a long way.
Thanks again.
Mark.
Great to hear we could help you get protected.
swiner@cloudflare.com
@mogden – if your the one being ddos’d then you deserve to be blackholed. I dont pay for my linodes for you to be targetted with a ddos and mine linodes taken down!!
Thank you for the analysis and a break down of what took place, and most importantly, thank you for being honest with customers!
Cheers!
I’m obviously a huge fan of Linode, but I wonder if this attack will force them to re-evaluate their “3 strikes” policy towards hosted sites which come under DDoS attack. As this attack should have taught them, it’s indiscriminate, and there’s not a whole lot a small website owner can do to mitigate it. We rely on Linode to be able to deal with this, and punishing the victim is hardly a fair solution.
And attacks started minutes after posting updates. http://status.linode.com/incidents/mkcgnmjmnnln
I’ve a message for Linode especially Chris, please invest more and more on infrastructure if you want to stay in the game otherwise, you’ll be overtaken by heavily funded startups in this domain. We know you have innovative mind and excellent technology but this alone is not sufficient for you to win in this domain. I like performance and flexibility of Linode but moved to DO just because I needed to setup my stuff at Japan and Singapore data-centers and Japan DC is sold out. 3 out of 6 locations are sold out and you are not yet expanding? How will you compete?
Come out of your box and look at your neighbors. It was painful to move to Digital Ocean for me but I had to take this decision. I am still using Linode for some of my stuff will continue using it until I need redundancy or you expand.
There’s nothing that I love more than the amount of technical detail that you provide to us on these cases, and even with some minor updates.
I love being a Linode customer, no DDoS will get that away from me 😀
Thanks for this post, Alex. This was a rough period for everyone involved and affected but I am extremely impressed by Linode making the effort to hopefully prevent the same scenario from happening again.
There were many lessons to be learned from this – both for Linode and for customers.
Linode appears to have realized what they needed to do and that is fantastic. Instead of saying sh*t happens and going about business as usual you are actively working to make sure it doesn’t happen again. Well done.
We (customers) need to cover our own bases too. For anything critical or even slightly important you need to have a plan in place in the event of a Linode outage (regardless of the reason).
I have now split some of my services and are far better placed to recover quickly in the event something like this were to happen again. Linode had always been so reliable that I got complacent. Lesson definitely learned.
In my case my costs have now increased as I am now paying other providers in addition to what I have and will continue to pay Linode, but the ability to keep some important services online is worth it.
Thank you to everyone at Linode for your hard work and for looking out for your customers.
Some of our big clients suffered with the downtime on those days but, with several VPS and more online each day, we never accepted any offer from others players. This kind of behaviour make us confident with the team and give us peace of mind that we’re in good hands.
Thank you for the update and respect with your customers.
Hostcare Internet
Thank you for being open, good luck with your new defences and I hope that you catch up on your family time!
Linode user here. Thanks for the transparency. I wasn’t directly affected but I appreciate the openness on the issue. It’s a welcome change to most companies now. I plan to keep using Linode just because of how cool you all handled the situation. Keep up the good work!
Cloudflare will probably help with your DDoS but they aren’t infallible as any other vendor.. But what happens when they get hit really hard themselves? I’d recommend getting a second DNS provider.
See Also: https://blog.thousandeyes.com/ultradns-ddos-affects-major-web-services/
https://blog.thousandeyes.com/ultradns-outage-october-2015/
I was beginning to wonder if such a note would arrive. The explanation is useful and I’m feeling as though things are safer than before.
Thank you for being transparent about what happened. That was a truly hellish attack. Getting slammed with a sophisticated and highly targeted 80 Gbit DDoS is stressful for any network admin and I’m glad that Linode succeeded in weathering the storm.
I am really impressed with way you have handled this whole situation, your company’s honesty and explanation is more than anyone could have expected. I’m sure there were many hours invested, not only in locating and fixing the problem on top of adding the double protection; but even in your letter to your customers. I hope all your customers are as loyal to your company as you have been with them. Way to step up your game, keep up the good work. Wishes for much more success……
Thank you very much for the detailed breakdown of what went wrong and what you plan to do to prevent this in the future. I have to say though, technical reasons and justifications aside, Linode has a lot to learn in regards to communication. I know you acknowledge that in your blog post but for many people (myself included) it’s too little way too late. It’s taken you 30 days to write a blog post that could’ve been written in hours. For 30 days people have been sitting on the fence wondering exactly what you guys are doing and whether or not they should jump ship. For many people (myself included), the absence of this response and the overall feeling that it has been so long since you said you were going to provide an update, that honestly you were just going to push this to the side and hope it went away, has directly contributed to Linode losing a significant amount of business from us.
I don’t want my response to turn into some Linode bashing post, but I want you to be aware that your failure to provide sufficient information and responses is the biggest problem here – for me, at least. It hit your reputation hard and caused us to lose a significant amount of trust in your company and services. DDoS attacks happen, and we know you guys were working extremely hard to deal with those. You reminded us often enough in your status updates. What we really wanted to know was that the worst was over and that you identified your weaknesses and were addressing those. The longer we had to wait for this information, the less trust we had/have in you.
I’d like to end this on a more positive note. All of the above said, your services are fantastic overall and I’d love to come back to Linode in the future, once you’ve performed all of the changes you have mentioned here. Just please, improve on your communications!
Long-time Linode customer…I wasn’t affected by the outage, but I’m really glad you’ve taken the time to write up what happened. Thanks for being transparent and generally awesome.
Alex, this caught my attention: “… requiring a level of focus and coordination between our colocation partners and their transit providers which was difficult to maintain.”
How did you structure this communication? What tools / technologies did you use or tried to use?
This is a nicely put article. I only have amazing things to say about Linode and its staff. Awesome post!
As a long time customer and a fellow network administrator I just wanted to say that I do really appreciate all your hard work. Respect.
Sounds an exciting project Alex, good luck!
Any news on continued security farces at Linode? and ‘The Best Practices not invented here’ approach.. For example to reset 2FA
—
Should you need us to disable your Two-Factor Authentication, the following information is required:
An image of the front and back of the payment card on file, which clearly shows both the last 6 digits and owner of the card.
An image of the front and back of the matching government-issued photo ID.
—
A) Photoshop CC in 2 mins, you have no idea what my CC should look like.
B) You can’t verify government ID so say 5 minute photoshop.
Woohoo for 2FA, known as 2 f… alls
Thanks for the update, and letting us know that things will be better handled in the future. Both technically and on the communication front.
Any idea who attacked and why?
Linode – you are the best. Thanks for your service.
Thanks for the update. As a long time linode customer, it is appreciated.
For you guys complaining about being kicked out in case of a DDoS, I recommend getting DDoS protection for your linodes. There are a lot of cheap options right there that can be integrated easily.
Some one recommended CloudFlare and they are great. You can also look at Sucuri:
http://sucuri.net/website-firewall/
Or Incapsula:
https://incapsula.com
Both great products and solutions. Stay safe!
200g? this years ddos was 800gbps…
good postmortem. now can you explain what happened with the “leaked” credentials and the fact that we had to reset the passwords.
thank you
These attacks could happen to anyone and any provider. Keep up the good work!
Great article and the right way to handle these kinds of problems. Transparency and constructive retros are the way to go.
I think you did great job considering the size of the attack. That’s why continue to use Linode for my virtual machines. Thank you for your support and keep up the good work.
Thank you for the clear and concise explanation. I look forward to you rolling out your upgrades and continue to be a happy customer with Linode.
Cisco routers, seriously?
Juniper high end routers take a gigantic steaming dump all over Cisco.
@Jake that’s essentially what ASRs are 😉
If you want to do it on the cheap side and be safe, get some cheaper / best equipment from huawei (give them a call). You might think the Chinese cannot be better than Cisco, but Cisco is now also made in China. Also I’m sorry, but you need some Ddos protection (expensive). You cannot just nullroute your costumers… you have to protect them. If the cheap OVH company can do it, why can’t you…
Looks like you guys need to hire someone with real experience in network engineering (worked at ISP level), not just some cheap undergraduate out of university.
You need to rely more on anycast, have reserved capacity, etc.
After reading this, I would not host my sites on linode. You guys look amateur (sorry).
I appreciate this honest insight, but I’ve moved back to a local server since these attacks made access to my Linode difficult or impossible, and always-on, always-accessible was my main reason for moving to Linode in the first place. Sorry, and better luck in the future.
I like the transparency, even delayed. I like that you’re taking steps. I DON’T like that your “security appliances” block ALL ICMP packets including the “Packet Too Big” messages required for path MTU discovery and breaking my ability to access the Manager over my VPN.
Buying blended internet direct from your colo provider is a bad idea (as it seems you have learned the hardway)
You should be getting your transit direct from diverse carriers… this is networking 101
Love the armchair quarterbacks giving their input. Now, for you QBs, where is your massive company you are running and making decisions and learning lessons from? Oh you don’t have one and you don’t work for one? Sit back and let Linode do their job, they are by far the best provider out there. The cost of this type of infrastructure is gigantic and you wanna-be QBs have no idea what it takes to run a business.
Great job Linode. I know I’ve made the right choice by using you.
Excellent. I knew you guys were “on it”. I really appreciate the detail you provided.
Thank you for releasing this honest and detailed report
Regarding CloudFlare, did you shop around for any other DNS DDOS protection services? The reason I ask is because CloudFlare happily caches too many dodgy websites. Some sources that may be of interest:
http://news.netcraft.com/archives/2015/10/12/certificate-authorities-issue-hundreds-of-deceptive-ssl-certificates-to-fraudsters.html (large number of phishing certificates issued by CloudFlare)
http://www.crimeflare.com (non-profit that investigates CloudFlare and its customers)
I appreciate the update, but i find a bit late too.
Also i don’t really get why Mr. Forster signing this post?
And don’t get me wrong, i have nothing against him, i don’t doubt his intentions or knowledge.
But i expected a statement from someone from the top of the food chain . This was also one of my main problems when the events happened, its like nobody cares from the top management, until one of the engineers realized that they can’t be silent anymore.
I still have that feeling, and is pretty alarming .
It’s time to move to IPv6-only internet. Attacking a single address will become impractical if a host can have millions of them changed automatically in an unpredictable way.
Appreciate the info.
It is a minor point, I know, but status.linode.com should either be un-available over https, or have its own cert.
try this in chrome…
https://status.linode.com
Thanks Linode Team for acknowledging your challenges, and courageously taking adaptive actions 🙂
Great job! Didn’t know such a story ongoing since my site was on all the time. Really appreciate all the hard work of LINODE support team!
Thank you for the very interesting update. Best of luck for the future.
I’m also quite curious on who could benefit from such attacks in the first place.
I am using Cloud Flare to protect the blog from DDOS attack, is there any other best application available to replace cloudflare? Is there a way to stop the DDOS or brute force attack for wordpress sites?
Great write up & good to see such honesty and transparency. I think it is important for readers of this to understand that DDoS attacks can affect anyone at any time on any host. Obviously when you are on the receiving end of a nullroute it is not nice, but It’s important to note though that providers do not want for you to have downtime, but if a DDoS directed at you is affecting other customers and you don’t have some form of mitigation, there is seldom any other option than to take this action. As they said, ‘cut off a finger to save the hand’. I’m quite sure that if someone else is being DDoS’d that you would prefer to see them nullrouted than have your own service impacted, so that has to work both ways in my eyes.
It’s important to look at the issue objectively – DDoS attacks are not going to go away and really if you have concerns around protection then this does mean paying for a mitigation service, especially if outages will be more costly than the monthly sub.
@Srinivas – You’ll need a CloudFlare business plan for DDoS attack mitigation. Simply being behind CloudFlare on a free plan won’t give you this protection, and there isn’t another service that I am aware of that provides free DDoS protection without at least having some other paid service. Keep in mind that CloudFlare isn’t an application, but rather a service which is totally separate from your Wordpress sites. If you want to run something locally to stop a brute force attack then have a look at a plugin such as Wordfence, which is very effective. Another good plugin is iQ Block Country which uses GeoLocation – you can lock down your back end to whitelisted countries only. Plugins are not infallible, but they definitely add extra security. Another good way to stop brute force attacks is by not using obvious account names for the administration area of your site…lots of tools will try to brute force on usernames like ‘admin’ – as with any security approach, it’s all about the layers!
As a final note, I do always find it interesting when posts like this attract the critics who dish out ‘advice’ about how X and Y should have already been done, or that they are amateur, etc. I would like to know which fairytale jobs they have at companies that have everything 100% perfect with 100% uptime and 0% chance of outages or attacks…
Fair play Linode, tip of the cap.
Thank you for your honesty and transparency. Very very good post. Thank you for your hard work during the attacks even on holidays. Keep pushing Linode Team!
yeah thank you also for your transparency. I remember what happened, evthg gave tears and I think, as many people, we planned to move to another company. Even some days ago, I compared with AWS, reading their doc for RDS, EC2, ELB, S3 etc, but Linode, even with much less available options and possibilities if we compare to amazon, Linode stay for us a better company, with a great support and reactive, providing faster and cheaper solutions.
I started with Linode 4 years ago, I loved the service and I am not going to go away from you guys. I know how painful firefighting could be, thanks to your team for working so hard. And please do everything that could prevent this from repeating.
Hello,
on the article you said following
“our nameservers are now protected by Cloudflare, and our websites are now protected by powerful commercial traffic scrubbing appliances.”
but seems it is not anymore. did you moved away from cloudflare protection? if yes then why? many hosting giants now rely on cloudflare protection.
Thank you for this update and the recent additional high memory and $5 options.