Vodafone Idea BGP leak: global routing system must implement MANRS

At the end of last week, a significant BGP leak caused widespread network outages that impacted major network operators, cloud, and CDN providers. The incident on Friday, April 16th, 2021 was (yet another) classic origin hijack case from Vodafone Idea (AS55410), an Indian operator based in Mumbai and Gandhinagar.

The Vodafone Idea ASN was inundated with traffic, 13 times higher than average, leaving its users unable to access the internet. This was probably caused by a wrong advertisement made by one of their customers, as reported by Medianama.

We analyzed network and BGP data during the incident to understand what caused the issue. Want to know how the incident unfolded and the impact it had on end-user experience? Let’s take a look.

What Went Wrong And Where?

According to CAIDA ASRank, Vodafone Idea (AS55410) is an Autonomous System with five different providers:

• CenturyLink (AS3549-AS3356)
• PCCW Global (AS3491)
• CW Vodafone (AS1273)
• Bharti Airtel (AS9498)
• Tata Communications (AS4755)

Right before the incident, this AS was announcing 823 IPv4 subnets and 8 IPv6 subnets.

Here’s what we found analyzing RIS rrc00 route collector, the most populated deployed by RIPE NCC. On Friday, April 16th, 2021, 1:48:58 PM GMT, AS55410 went into a frenzy announcing on the Internet 34000+ networks not belonging to AS55410. These networks were already announced on the Internet by their legitimate owners. As a result, a large portion of user traffic got redirected to AS55410 instead of the proper destination, causing disruption of services for 3500 companies around the world.

Among the organizations impacted, we found most of the national incumbent telcos (Deutsche Telekom, TIM, Claro, Orange, Telefonica), CDNs (Google, Akamai, Edgecast), and even banks (Punjab National Bank). A number of Vodaphone services were also impacted by the hijack, such as Vodafone Portugal, Vodafone Italia, Vodafone Egypt, Vodafone Fiji, and Vodafone New Zealand.

Fig 1

RIS rrc00 collector recorded ~225k BGP packets with AS55410 as the originating AS in the AS path between 1:45PM GMT and 3:00PM GMT. Most were announcements of hijacked networks. Out of 73 peers sharing data with rrc00, 60 of which were sharing a full route, 64 received at least one hijacked network. This most likely indicates that the hijack spread towards most of the ASes of the Internet. As can be understood from the graph above (Fig1), it took about one hour to see most (but not all) of the hijacked routes removed from the collector, and hence from the Internet.

It is interesting to note that two of the providers of AS55410 did not have a good filtering mechanism in place to prevent such an event. This contributed to the spread of the attack. In the graph below (Fig 2), we focused our analysis on the evolution of hijacked networks by breaking it down by each provider of AS55410. In simpler words, we looked at the number of hijacked networks with an AS path that ended with ASprovider 55410.

Fig 2

The remaining three providers were able to avoid spreading the hijack event. To understand how this was possible, we contacted Marco Marzetti, a member of the peering team at PCCW (AS3491). He explained that they have several defense mechanisms in place on their routers to mitigate such an attack.

These mechanisms include accepting only those routes belonging to a selected list of prefixes and dropping RPKI invalids. Most likely, the first mechanism was the most effective, since (sadly) only ~20% of the prefixes involved signed a ROA and showed as RPKI invalid.

Another interesting aspect we found during our analysis is that 1062 networks belonging to 168 ASes are still seen by some peers of RIS as hijacked by AS55410.

Out of these 1062 networks, 991 of them are seen as announced via Bharti (AS9498), while the missing 71 networks are seen as announced only by CW Vodafone (AS1273). This could be a case of ghost routes, but most likely the network operators of AS55410 still have something to fix on their routing with these two providers.

Incident Impact

Let’s take a quick look at how the incident impacted our customers. Catchpoint’s BGP tools analyses our customer networks using our dedicated BGP infrastructure along with RIPE NCC RIS and University of Oregon Route Views Project.

Fig 3 (Click to enlarge)

By focusing on one of the affected networks during the incident time span, we can see that peers from all our data sources were impacted (Fig 3). Analyzing the public route collectors, we found that 234 of the peers of RIS and 162 of the peers of Route Views were seeing the network as hijacked. This involved peers from all over the world, as is demonstrated in the following snapshot of the Catchpoint portal (Fig 4).

Fig 4 (Click to enlarge)

Conclusion

Even though the madness was caused by AS55410, it is important to note that these kinds of events could have been heavily mitigated if all the providers of AS55410 were applying proper defense mechanisms on their routers as outlined by Mutually Agreed Norms for Routing Security (MANRS). Most likely, if they were dropping RPKI invalid routes and if each of the affected organizations were signing their network resources in RPKI, this event would have never occurred. This incident was a reminder to network operators that they must implement MANRS to deal with such routing security threats.

Learn more about BGP in our Network Administrator’s guide.