Cybersecurity attacks have become a weekly occurrence in many news columns. One recent example was that of one of our customers, QIWI payment system, successfully mitigating a 480 Gbps memcached amplified UDP DDoS attack.
While we at Qrator Labs would rather stay out of the news, such instances justify all the preparation that we put into mitigating for such attacks. To help others learn from our experience, I thought I’d recap several facts about amplification attacks, so that you too will be prepared ‘when’ the day comes.
Fact #1: Amplifiers always were, are, and would be an issue
Network Time Protocol (NTP) was the first protocol being abused (back in 2013) as an amplifier in DDoS execution. With hundreds of thousands of NTP servers deployed all over the web, this particular vector was quite lucrative for the attackers looking for a way to amplify their capabilities. By early 2014, NTP replaced DNS as the primary reflection/amplification vector. The Smurf and Fraggle attacks also used amplification for DDoS and go back to 1999.
In 2015, we saw the rise of botnets on a mass scale. They used UDP-based amplification creating the largest volumetric DDoS attacks ever observed. The combination of thousands of compromised devices with the NTP and DNS amplification made these attacks highly dangerous, as the Dyn and OVH denial of service in 2016 showed. Joint disruption of the command and control (C&C) servers for those botnets made it possible to avoid the danger of botnets taking over the web.
There was change in 2016, as the popularity of a number of alternative amplifiers increased, including Portmap, SNMP, SSDP, Chargen, MSSQL and CLDAP. This trend continued into 2017, with many skilful hackers targeting various protocol vulnerabilities, trying to find more unknown vectors.
Fact #2: Amplification does not change or evolve. It is just the new vulnerabilities that are found
In 2017, a group of Chinese cybersecurity researchers from 0Kee team revealed the memcached amplification vector (see presentation below) — it took only a couple of months for the record-breaking amplified DDoS-attacks to take advantage of unsecured memcached servers.
Being open-source, free, and available in almost every Linux distributive, Memcached, which has an extraordinary amplification factor exceeding 10,000x (Akamai reported amplification factor above 50k), is a powerful tool for causing disruption.
Although the researchers only took advantage of the amplification effects of the vector in 2017, the amplifier itself was introduced 10 years ago, when UDP protocol listening was added to the memcached — it’s that nobody cared for the amplifier itself at the time. Only now, with the continuous growth of bandwidth over the networks, has it become such an issue.
The general idea behind reflection/amplification has not changed a bit. First, the forged (IP-spoofing) requests are sent to the vulnerable (open to anyone on port 11211) memcached server, after which the UDP server prepares the response and, with the help of the attacker, delivers thousands of them to the target host, overwhelming the network channels with a flood. Massive amounts of Packets Per Second and Bits Per Second delivered could not be mitigated without a proper solution, since such traffic would make edge routing devices unavailable quite quickly.
Fact #3: When something works, it works; sometimes against you
The current world record for DDoS attack bandwidth, was a 1.7 Tbps assault on one of Arbor Network’s customers in 2017, followed by a 1.3 Tbps attack on GitHub (under Akamai’s DDoS-mitigation service).
The Shodan statistics still indicated a large number of active memcached amplifiers open to all the world’s crooks at once:
Memcached DDoS attacks: 95,000 servers vulnerable to abuse https://t.co/79BHSKsXR0
Analysis + guidance: @Akamai @arbornetworks @BarryRGreene @cloudflare @corero @dormando @JobSnijders @johullrich @Qrator_Labs pic.twitter.com/ivv3xX3XVR— Mathew J Schwartz (@euroinfosec) March 8, 2018
As you can see from the image in the tweet, mainland China and the United States are the top sources of memcached amplifiers.
Qrator.Radar also gathered the total statistics on freely available memcached servers.
That is a short reminder to all the ISPs, IXes and other internet businesses from the Qrator.Radar team on memcached amplification DDoS attacks.https://t.co/rC7RGQ3WUE pic.twitter.com/RfO92NnX8p
— Qrator Labs (@Qrator_Labs) March 6, 2018
Interestingly, the number of vulnerable memcached servers they’ve observed has dropped dramatically. One reason for this might be that operators (ISPs and transit operators) are becoming more aware of the problem and are working to mitigate the risk of their networks being used as reflectors in attacks.
DDoSmon also offers extensive statistics on the current targets of memcached amplification DDoS attacks, where the trend is evident.
Figure 1: Memcache attack trend since December 2017. Credit: DDoSmon
Fact #4: Everything that could be fixed, should be fixed
Mitigating against the risks of memcached isn’t a difficult task.
The current best practice is to apply rate-limits on all external facing ports and police UDP/11211 traffic down to something manageable. This protects both your own infrastructure and your customers.
Memcached users should disable the UDP support if they’re not using the UDP in data transfer. Also, memcached should listen only to the localhost, not all interfaces available, which is the default memcached installation scenario.
Furthermore, everyone more significant than a single entity or one person in control of some network resource should make sure that memcached servers are NOT exposed to all the Internet and firewalled.
NTT recommends adding memcached UDP/11211 to the same “exploitable ports” list as NTP, CHARGEN and SSDP. Below is a configuration example for IOS XR to rate-limit these amplification-sensitive UDP ports to 1% of the port’s capacity. Qrator Labs finds this way of handling memcached traffic extremely useful and effective and recommends it for global deployment.
ipv4 access-list exploitable-ports
permit udp any eq ntp any
permit udp any eq 1900 any
permit udp any eq 19 any
permit udp any eq 11211 any
!
ipv6 access-list exploitable-ports-v6
permit udp any eq ntp any
permit udp any eq 1900 any
permit udp any eq 19 any
permit udp any eq 11211 any
!
class-map match-any exploitable-ports
match access-group ipv4 exploitable-ports
match access-group ipv6 exploitable-ports-v6
end-class-map
!
policy-map ntt-external-in
class exploitable-ports
police rate percent 1
conform-action transmit
exceed-action drop
!
set precedence 0
set mpls experimental topmost 0
!
class class-default
set mpls experimental imposition 0
set precedence 0
!
end-policy-map
!
interface Bundle-Ether19
description Customer: the best customer
service-policy input ntt-external-in
ipv4 address xxx/x
ipv6 address yyy/y
...
!
interface Bundle-Ether20
service-policy input ntt-external-in
...
... etc ...
For further recommendations, check out Barry Greene’s tips for implementing Exploitable Port Filters.
Though the number of vulnerable servers on the Internet decreases, attackers are in search of new and more powerful ways to exploit vulnerabilities. Sysops in charge of memcached servers mostly seem to be either careless or ignorant, so there’s going to be a long tail while trying to shut down the remaining caches. This is why it’s still important to handle the memcached amplification problem somehow on an operator’s network.
Artyom Gavrichenkov is Chief Technology Officer at Qrator Labs.
The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note a Code of Conduct applies to this blog.