Amplification attacks: how a small request becomes a huge reply

On 28 February 2018 GitHub's network took in 1.35 terabits a second, carried by 126.9 million packets a second, from tens of thousands of machines in more than a thousand networks.¹⁷ Nobody had to break into them. They were memcached servers, caches that normally sit behind websites, left reachable from the internet with UDP switched on.¹⁷ Each one was doing what a cache does: answering a request. The requests came from the attacker, and every one of them carried GitHub's address in the field that says who sent it.

That is the whole trick of an amplification attack. The attacker never sends the flood. Honest servers do, because they were asked a small question with a forged return address and answered, generously, to the address on the envelope. Two failures make it work, and both can be fixed: networks that let a packet leave with someone else's address on it, and servers that answer strangers with far more than they were sent. The second kind may be running on your own server.

Reflection and amplification: two tricks in one packet

Every IP packet carries two addresses: where it is going and where it came from. The sender writes both. Routers forward a packet by looking at its destination; checking that the source is true is an extra step a network has to choose to take. Where nobody takes it, a machine can send a packet whose source address belongs to somebody else. That is called spoofing.

Reflection is what happens next. A server sends its answer back to wherever the request says it came from, so a forged request produces a reply that lands on someone who never asked. CISA's advisory on UDP amplification calls a server used this way an amplifier, and the result a reflected denial of service.¹ The server is a mirror, and the victim is whoever's address was written on the request.

Amplification is the multiplier. Some requests produce answers far bigger than themselves. A short question to a DNS resolver can return a long list of records; an old NTP monitoring command returns the last 600 addresses that talked to the server.³ Put the two together and every byte the attacker sends reaches the victim as tens or hundreds of bytes, sent by someone else. CISA's name for the combination, spread across many servers at once, is DRDoS: distributed reflective denial of service.¹

Look at who sends what. The attacker's machines send only small requests, and only to the reflectors. The reflectors send the flood. The victim receives packets from thousands of legitimate servers and never from the attacker's own address. Nothing in that flood looks hostile on its own: the packets come from big, reputable servers carrying genuine answers, which is why CISA calls these attacks hard to detect¹ and hard to prevent.² Blocking the senders means blocking resolvers and time servers, and does nothing to the attacker. Among the kinds of attack in the guide to what a DDoS attack is, this is a volumetric one: it wins by filling the victim's link, whatever runs behind it.

It also means a reflector has not been broken into. It is doing its job, and someone else chose where its answers go. A clean malware scan on it proves nothing, and the fix is configuration, not cleanup.

Why it is almost always UDP

Reflection needs a server that sends a useful answer to an address it has never heard from before. That describes UDP. RFC 768 keeps it deliberately thin: a small header in front of the data, no handshake and no promise that anything arrives.⁵ A datagram comes in, the application reads it and replies, and the RFC treats the sender's port as the natural place to send that reply.⁵ Nothing in UDP checks where a datagram really came from, so, as CISA notes, a forger can write in any source address it likes unless the application runs its own session setup on top, as voice over IP does.¹

TCP is different because nothing useful happens before the handshake. A TCP server answers a new connection with a SYN-ACK and sends no data until the client acknowledges it. With a forged source, the SYN-ACK goes to the victim, who never asked and never completes the handshake. From an ordinary server the victim receives a SYN-ACK about the size of the forged SYN, perhaps resent a few times, and no data ever follows. Almost always, then, but not always: researchers have shown that some middleboxes, devices in the network path that inspect traffic, break TCP's rules in ways that let them be made to answer forged TCP packets, in some cases amplifying far more than the UDP services in the next section.⁹ The common use of forged TCP is different again, filling a server's queue of half-open connections, which the guide to SYN floods covers.

Protocol designers now plan for this. RFC 8085, the IETF's guidance for anyone building on UDP, tells them to treat any exchange where a small question earns a big answer as a possible amplifier and to limit how it can be abused, and never to take a request's source address as proof of who sent it.⁷ QUIC, a newer transport built on UDP, writes a hard limit into its standard: until a client has proved the address it claims, a QUIC server may send it at most three bytes for every byte received from it.⁸ Three times is a ceiling chosen on purpose. The older services in the next section were designed before anyone set one.

None of this makes UDP itself the problem. Games, DNS, voice and QUIC all run on it for good reasons. The trouble is a UDP service that answers strangers with far more than they sent, or one that listens on the internet when it only needed to serve its own machine.

How much bigger: the published amplification factors

The size of the multiplier has a name, the bandwidth amplification factor (BAF). CISA measures it on UDP payloads, the data in the answer divided by the data in the request, and publishes a table of factors for common services, first supplied by the researcher Christian Rossow and extended as new attacks were reported.¹ His 2014 paper examined 14 widely used UDP protocols, from network services and games to file sharing and botnets, and found that every one of them could be abused for amplification.⁴ As a general case, CISA reckons a single request can turn into ten to a hundred times its own bandwidth.¹ The chart shows how far past that some services go.

Most of the large factors come from a feature meant for a trusted operator, left answering the whole internet. Game servers are the exception: answering strangers is their job.

NTP, 556.9. A time server has no reason to send a large reply to a clock. Older NTP servers shipped with a monitoring command switched on that lists the last 600 addresses to have contacted the server; with a forged source, that list goes to the victim. CISA's advice was to switch the command off, or move to NTP 4.2.7, where it is off already.³

DNS, 28 to 54. DNS runs over UDP, and a short question can have a long answer. RFC 5358 gives two reasons attackers favoured it: resolvers that would look names up for anyone were common in default configurations, and a tiny query can pull back a large response.¹² The same RFC puts the factor at around 80 for a scenario where the attacker first plants a large record in a zone it controls, then has it fetched through a resolver that supports the EDNS0 extension.¹² That figure is measured on whole packets, which if anything lowers a ratio (the worked example below shows why), so the gap from CISA's figure comes from the question asked, not from the yardstick.

CLDAP, 56 to 70. The connectionless form of LDAP, a directory protocol that belongs inside an organisation's network. CISA cites a report from November 2017 that ranked it the third most common reflection attack, behind DNS and NTP.¹

Game servers, 5.5 and 63.9. The query a server browser sends to list a game server is in the table too: 5.5 for the Steam protocol and 63.9 for the Quake network protocol, both answering a server information request.¹ These cannot simply be closed, so they need limits instead, as the defences below explain.

Memcached: the outlier

Memcached is a cache: an application stores values in it and reads them back quickly. Its own documentation is blunt that it must not be exposed directly to the internet.¹⁵ In February 2018 many instances were exposed anyway, with UDP enabled, and the memcached project's release notes describe what happened: attackers first wrote large values into exposed caches, then fetched them with UDP requests bearing a forged source.¹⁴ The attacker chose the size of the answer before asking the question.

Cloudflare published first-hand measurements at the time. In its own test a 15-byte request produced 134 KB of response, about 10,000 times; in the attacks it saw, a 15-byte request produced a 750 kB response, which it gives as 51,200 times.¹⁶ That figure counts a kilobyte as 1,024 bytes: 750 × 1,024 = 768,000, and 768,000 ÷ 15 = 51,200. CISA added memcached to its table at 10,000 to 51,000 after reports of an unprecedented amplification factor,¹ and GitHub's report gives the same ceiling: each byte from the attacker could become up to 51 KB at the target.¹⁷

GitHub's numbers also describe the packets. 1.35 Tbit/s is 168.75 billion bytes a second; divided by 126.9 million packets a second, that is about 1,330 bytes per packet. The flood was made of nearly full-size packets, which is what big answers look like.

The fix arrived as a changed default. Memcached 1.5.6, released on 27 February 2018, the day before the attack on GitHub, switched UDP off unless the operator explicitly turns it on.¹⁴ New installs no longer answer over UDP, but each exposed older install stays usable as a reflector until its owner upgrades or reconfigures it, the same pattern as NTP's fix a few years earlier.

A worked example: what 100 Mbit/s of requests becomes

Suppose an attacker has machines with 100 Mbit/s of upload between them, on networks that let forged packets out. Multiply by the published factors. The table takes the upper figure where CISA gives a range, and measures the result in server ports of 1 Gbit/s.

DNS alone fills a gigabit port more than five times over: 100 × 54 = 5,400 Mbit/s, or 5.4 Gbit/s. Memcached's 5.1 Tbit/s is theoretical: the attacker would run out of reachable reflectors, or the reflectors of uplink, long before that. Run the arithmetic backwards instead and it is just as striking. GitHub's 1.35 Tbit/s is 1,350,000 Mbit/s; divided by the published factor of 51,000, it needs only about 26.5 Mbit/s of requests. That divides a rate measured on the wire by a ratio of payloads, though, and the difference matters.

Payload bytes are not wire bytes

The factor counts payload. Every packet also carries headers: the UDP header is 8 bytes,⁵ and the smallest IPv4 header is 5 words of 32 bits, which is 20 bytes.⁶ Those 28 bytes weigh most on the tiny request. Take the 60-byte DNS question from the packet drawing at a factor of 54. On the wire the request is 60 + 28 = 88 bytes and the answer at least 3,240 + 28 = 3,268 bytes, a ratio of about 37 rather than 54. Still enormous, and the attacker pays only for the small side. Use the published factors to compare services, not to predict the exact size of an attack.

The same correction applies to GitHub. Counting headers, a 15-byte memcached request is 15 + 28 = 43 bytes on the wire. Cloudflare's 750 kB answer, 768,000 bytes, arrives in datagrams carrying 1,400 bytes of data each,¹⁶ so it takes 549 of them, each 1,428 bytes on the wire, about 783 kB in all. That is a wire ratio of about 18,200rather than 51,000, so the requests behind 1.35 Tbit/s were nearer 74 Mbit/s: still less than a tenth of a 1 Gbit/s server port.

100 Mbit/s of DNS requests at 54 times

5.4Gbit/s

Requests behind GitHub's 1.35 Tbit/s, counting packet headers

74Mbit/s

Each reflector's share of 5.4 Gbit/s, spread over 10,000

540kbit/s

What each reflector sees

Now look from a reflector's side. Spread the 5.4 Gbit/s DNS flood over 10,000 open resolvers and each one sends 540 kbit/s, about half a megabit a second. RFC 5358 observes that nobody in the chain except the victim handles much traffic, so none of them is likely to see anything odd in their own.¹² That is why the defences cannot wait for reflector owners to spot an attack. A server has to be set up so that it cannot take part at all.

Where to stop it: two places that break the attack, one that absorbs it

An amplification attack needs two things at once: a network that lets a packet leave with a forged source, and a server that answers strangers with a big reply. RFC 5358's step-by-step account of the DNS version has both in it: the forged query comes from a client on a network without BCP 38 and goes to an open recursive resolver.¹² Fix either failure and that path stops working. Only one of the fixes is in the hands of a server owner, and it is the one this guide ends on.

Source address validation: BCP 38

BCP 38, published in May 2000 as RFC 2827, sets out a simple rule for the router where a customer's network joins its provider: forward a packet only if its source address is inside that customer's own range, deny anything else, and log what was dropped.¹⁰ For a customer given 203.0.113.0/24, a packet claiming to come from 198.51.100.7 has no business leaving, so it never does. CISA's DNS advisory treats this filtering as the first step, and gives one way a router can do it: if the packet's source could not be reached back through the interface the packet arrived on, the source is taken to be forged.² RFC 3704, BCP 84, since updated by RFC 8704, extends the idea to networks with more than one provider and lists the ways to do it, from access lists to strict and loose reverse path forwarding.¹¹

Notice where the filter has to be. In the RFC's own example it sits on the link to the attacker's network.¹⁰ Filtering on your network does not stop anyone else forging your address; it stops people on your network forging everyone else's. BCP 38 is a favour each network does for the rest of the internet, and its cost falls on the network that deploys it rather than the one it protects. It is not perfect either: an attacker can still forge another address inside its own permitted range. Even so, the RFC counts traceability as a side benefit: behind the filter, an attacker has to use a real address that can be reached, and that points back to where it is.¹⁰

Closing open resolvers: BCP 140

A recursive resolver looks names up on behalf of its clients; an authoritative server answers for its own domains. The DNS guide covers the difference. A recursive resolver that answers anyone on the internet is an open resolver. RFC 5358, BCP 140, makes closed the default: out of the box, a name server should do recursive lookups for nobody outside its own networks.¹² In practice that means one of two shapes. A server that only answers for your own domains has recursion switched off, and a resolver answers only its own organisation's clients. CISA's advisory shows both for BIND, with recursion no; and an allow-recursion list.²

// Authoritative only: answer for your own zones, look up nothing for anyoneoptions {    recursion no;    allow-query-cache { none; };};// Or: a resolver for your own networks onlyacl ournets { 127.0.0.1; 203.0.113.0/24; };options {    recursion yes;    allow-recursion { ournets; };};

A resolver that listens only on a loopback address, like the local stub in the example further down, cannot be an open resolver at all, because nothing outside the machine can reach it.

Response rate limiting, for servers that must answer everyone

Authoritative servers cannot be closed: answering anyone who asks about your domain is their job. RFC 5358 accepts that they can still be abused, though they multiply traffic far less than open resolvers do.¹² For them the tool is response rate limiting, RRL. ISC, which maintains BIND, describes it as a budget kept for each client and each distinct answer: the budget refills by a set amount every second, each answer sent spends from it, and while it is overdrawn, further copies are dropped or cut short.¹³ A flood of identical answers to one forged address overdraws its budget in a moment. A real resolver asking about many different names barely touches it.

The design protects real clients too. A real client that misses an answer tries again, over UDP or TCP,¹³ and a retry over TCP needs a handshake that a forger cannot complete. RRL is off by default in BIND and switched on with a rate-limit clause; ISC's simple example sets 10 responses per second, as an illustration rather than a recommendation.¹³ ISC recommends it for authoritative servers only: on a recursive resolver it produces more false positives, and limiting who may use the resolver is the better fix.¹³ CISA extends the idea to every UDP service: configure it to refuse or cap abuse, as DNS rate limiting does, and put rate limits on public services that cannot be closed.¹

Capacity in front of the victim

The last box in the drawing belongs to the victim, and it cannot stop the attack, only survive it. A reflected flood fills the link before any server sees it, so it has to be absorbed upstream, where there is more bandwidth than one port. CISA's list for victims includes working with upstream providers and keeping their emergency contacts to hand.¹ How scrubbing and filtering work is the subject of how DDoS protection works.

Is your server a reflector? Checking your own machine

The practical question for anyone running a server is short: which UDP services on this machine answer the internet, and should they? Start by listing them. ss with -u shows UDP sockets, -l only listening ones, -n keeps ports as numbers and -p shows which program owns each socket.¹⁸

$ sudo ss -ulnpState   Recv-Q  Send-Q   Local Address:Port    Peer Address:Port  ProcessUNCONN  0       0        127.0.0.53%lo:53           0.0.0.0:*      users:(("systemd-resolve",pid=612,fd=13))UNCONN  0       0            127.0.0.1:323          0.0.0.0:*      users:(("chronyd",pid=701,fd=5))UNCONN  0       0                [::1]:323             [::]:*      users:(("chronyd",pid=701,fd=6))UNCONN  0       0              0.0.0.0:11211        0.0.0.0:*      users:(("memcached",pid=944,fd=27))UNCONN  0       0                 [::]:11211           [::]:*      users:(("memcached",pid=944,fd=28))UNCONN  0       0              0.0.0.0:27015        0.0.0.0:*      users:(("srcds_linux",pid=1203,fd=9))

Read the Local Address column. An address beginning 127 is loopback: only programs on the same machine can reach it, so the DNS stub on 127.0.0.53 and the time daemon's control port on 127.0.0.1 are fine. 0.0.0.0 means every IPv4 interface, the public one included. [::] is the IPv6 version of the same thing and * is a wildcard too, so both are as public as 0.0.0.0, while [::1] is IPv6 loopback, as private as 127.0.0.1. A specific public address is public as well. In this example the wildcard is right for the game server, which has to answer players, and wrong for memcached, which will answer anyone who can reach port 11211 over UDP, on IPv4 or IPv6. Then go down the list:

Two habits cover what the list misses. First, a firewall that drops inbound UDP to every port you do not deliberately serve, so a service installed next month does not become a reflector by accident; the guides to securing a Linux server and game server ports and firewalls cover the rules. Second, watch the direction of your traffic. A reflector sends more than it receives, so a UDP service whose outbound traffic is suddenly many times its inbound, most of it to one address, is being used. CISA gives operators the same sign to watch for: unusually large answers going to a single address.¹ GitHub spotted its attack from the other end, as an anomaly in the ratio of inbound to outbound traffic.¹⁷