Cloudflare Explains How Facebook Disappeared from the Internet

The folks at Cloudflare have published a fascinating look into the recent ~6 hour long downtime that the Facebook network went through, taking down not just the Facebook product itself, but also WhatsApp, Instagram, FB’s internal looks, and a lot more. It’s a somewhat technical explanation, but Cloudflare’s Tom Strickx and Celso Martinho have made it very easy to understand.

Today at 1651 UTC, we opened an internal incident entitled “Facebook DNS lookup returning SERVFAIL” because we were worried that something was wrong with our DNS resolver 1.1.1.1. But as we were about to post on our public status page we realized something else more serious was going on.

Social media quickly burst into flames, reporting what our engineers rapidly confirmed too. Facebook and its affiliated services WhatsApp and Instagram were, in fact, all down. Their DNS names stopped resolving, and their infrastructure IPs were unreachable. It was as if someone had “pulled the cables” from their data centers all at once and disconnected them from the Internet.

How’s that even possible?

It’s really interesting to see how a (possibly) minor piece of code can take down large parts of the internet like this. Honestly, it would be a good thing for the internet overall of Facebook disappears from the internet, but I feel for everyone at Facebook behind this issue. Major hugs to the people involved in bringing the network back up.

Then again, imagine messing up so bad that your boss ends up losing $6 billion.

October 5, 2021 ✚ Permalink

Vodafone India's BGP Hijack Takes Down Several Global Networks

Ax Sharma, writing for BleepingComputer:

A large BGP routing leak that occurred last night disrupted the connectivity for thousands of major networks and websites around the world.

Although the BGP routing leak occurred in Vodafone’s autonomous network (AS55410) based in India, it has impacted U.S. companies, including Google, according to sources.

You should also read Anurag Bhatia’s fantastic analysis here.

Time and again, these companies prove that initiatives like this one and this one need to be taken a lot more seriously for the Internet to become a better place.

April 19, 2021 ✚ Permalink

Is BGP Safe Yet? Cloudflare Launches Website to Check Whether your ISP Prevents Route Leaks & Hijacks

Louis Poinsignon writes on the Cloudflare blog:

BGP leaks and hijacks have been accepted as an unavoidable part of the Internet for far too long. We relied on protection at the upper layers like TLS and DNSSEC to ensure an untampered delivery of packets, but a hijacked route often results in an unreachable IP address. Which results in an Internet outage. 

The Internet is too vital to allow this known problem to continue any longer. It’s time networks prevented leaks and hijacks from having any impact. It’s time to make BGP safe. No more excuses.

In June 2019, large parts of the Internet were put offline for no fault of theirs, thanks to Verizon — just one of the many hundreds of Internet Service Providers across the world who do not have security practices and filtering in place to prevent such a thing from happening.

Tom Strickx wrote on the Cloudflare blog back then:

Today at 10:30UTC, the Internet had a small heart attack. A small company in Northern Pennsylvania became a preferred path of many Internet routes through Verizon (AS701), a major Internet transit provider. This was the equivalent of Waze routing an entire freeway down a neighborhood street — resulting in many websites on Cloudflare, and many other providers, to be unavailable from large parts of the Internet. This should never have happened because Verizon should never have forwarded those routes to the rest of the Internet. To understand why, read on.

And this sort of thing happens a lot. And it isn’t just an inconvenience, it can also cause tremendous damage. Lily Hay Newman, writing for Wired, says:

BGP disruptions happen frequently, generally by accident. But BGP can also be hijacked for large-scale spying, data interception, or as a sort of denial of service attack. Just last week, United States Executive Branch agencies moved to block China Telecom from offering services in the US, because of allegedly malicious activity that includes BGP attacks. 

To make this internet a better place for everyone, Cloudflare has today launched an effort to push ISPs to implement checks and filtering to prevent BGP leaks & hijacks. The company has launched IsBGPSafeYet.com, a website that lets you check whether your Internet Service Provider (ISP) or broadband provider has BGP filtering in place or not. You can run the test in your browser and get instant results.

Check Your ISP on IsBGPSafeYet.com →

Indian ISPs Fail the Test

I ran the test for my ISPs Jio & ION and both of them failed the test. I also asked a few of my friends to run the test on their respective ISPs, and so far all Indian ISPs are failing the test. Here’s a non-exhaustive list:

Indian ISPs Failing the Test:

If you’re using any of the above ISPs, let them know.

If you’re in India, please run the test on IsBGPSafeYet.com in your browser and let me know on Twitter, so I can update this list.

April 18, 2020 ✚ Permalink