SYN floods and SYN cookies, explained

A SYN flood has an odd signature. The bandwidth graph barely moves, memory is fine, the SSH session you already had open keeps working, and yet nobody new can connect to the site. RFC 4987, the IETF's write-up of the attack, describes exactly that: it stops new incoming connections to the targeted port and leaves outgoing connections, and connections that were already established, alone.²

The reason is built into TCP. A server has to remember a connection attempt before it knows whether the client is real, and the place where it remembers them has a fixed number of slots that, without cookies, a few packets a second can fill. SYN cookies are the trick that removes the memory: the server keeps nothing until the client proves it exists.

The handshake, number by number

Every TCP connection starts with three segments, and the TCP standard itself gives an example with real numbers. Peer A sends a SYN with sequence number 100. Peer B answers with a segment carrying both SYN and ACK, its own sequence number 300 and an acknowledgement number of 101. A finishes with an ACK carrying sequence number 101 and acknowledgement 301.¹ The ordinary handshake in the drawing at the top of this page is that exchange.

The off-by-one numbers are the point. When the SYN flag is set, the sequence number is the sender's initial sequence number (ISN), and its first byte of data will be numbered ISN+1, so the SYN itself uses up one number. The acknowledgement field carries the next sequence number the sender expects to receive.¹ B's "ACK=101" therefore means "I have your 100, send 101 next", and A's "ACK=301" means the same about B's 300.

Look at what A had to know to write 301: the number 300, which it could only have learned by receiving B's SYN-ACK. That turns the server's ISN into a small challenge, and it works only if the number cannot be guessed. RFC 9293 asks for ISNs built as a clock plus a secret: ISN = M + F(localip, localport, remoteip, remoteport, secretkey), where M is a timer that ticks roughly every 4 microseconds and F is a pseudorandom function that must not be computable from the outside.¹ Keep that recipe in mind. A clock plus a keyed function of the addresses and ports is, almost exactly, what a SYN cookie is.

The standard also names the states. A server waiting for connections is in LISTEN. Once it has received a SYN and sent its own, it is in SYN-RECEIVED, waiting for the acknowledgement that confirms the connection. When that arrives, both ends are ESTABLISHED.¹ A connection stuck in SYN-RECEIVED is what people mean by half-open. Where this handshake sits in a whole page load is covered in what happens when you open a website.

Where the server keeps a connection it has not verified

TCP keeps everything it knows about a connection in a record called the transmission control block, or TCB. RFC 9293 calls the CLOSED state "fictional" for exactly this reason: it is the state in which there is no TCB, and therefore no connection.¹ Every other state costs memory, SYN-RECEIVED included.

RFC 4987 spells out when that memory is spent. The moment a SYN arrives on a listening port, the connection moves to SYN-RECEIVED and part of a TCB is filled in from the SYN's headers; many stacks copy the listener's TCB, so fresh memory is committed for every SYN. Kernels limit how many of these can exist at once, and that limit is the backlog.²

Linux keeps two queues

On Linux the waiting happens in two places. The listen(2) manual explains that since Linux 2.2 the backlog argument an application passes to listen() sizes the queue of completely established connections waiting to be accepted, and that the queue of incomplete requests is capped separately by tcp_max_syn_backlog.⁶ The kernel documentation describes that setting the same way, as a per-listener limit on requests in SYN_RECV, at least 128 and rising with the machine's memory.⁴

That is how the manuals describe it. The code of current kernels works differently. When a SYN arrives, the kernel treats the listening socket's queue of half-open requests as full once it holds more entries than the socket's listen() backlog, and that is what triggers a cookie, or a drop when cookies are off.⁸⁹¹⁰ The backlog is the number the application passed to listen(), silently capped by somaxconn,⁶ which defaults to 4096 and was 128 before Linux 5.4.⁴ tcp_max_syn_backlog matters only when cookies are switched off, and then only to keep the last quarter of it for peers the kernel has talked to before.⁸ So on a modern Linux server one number, the listen() backlog, caps both queues.

An entry in the first queue is small. The kernel documentation puts a SYN_RECV request socket at about 304 bytes,⁴ and the ss manual lists syn-recv among the states kept as "minisockets" rather than full sockets.⁷

It is also patient. If the final ACK does not come, Linux retransmits the SYN-ACK: by default 5 times, the last about 31 seconds after the first, and it gives up on the attempt after about 63 seconds.⁴ So a SYN that nobody follows up occupies a slot for roughly a minute. That minute, multiplied by a fixed number of slots, is the whole attack surface.

How a flood fills the queue

RFC 4987 is precise about what the attack goes after. It does not try to overload the network or the host's memory. It tries to exhaust the backlog of half-open connections for one port, by sending a quick barrage of SYNs from addresses, often spoofed, that will never answer the SYN-ACKs. With the backlog full of bogus entries, legitimate requests are rejected.²

Spoofing does more than hide the attacker. The forged addresses have to be ones that stay silent, because a real host that receives a SYN-ACK it never asked for replies with a reset, and that reset frees the entry at once and makes room for a legitimate client.²

The backlog is not part of the TCP standards at all, so a full queue may silently drop new SYNs on one system and answer them with resets on another.² Either way the site is down for everyone not already connected.

The arithmetic: why a trickle is enough

RFC 4987 notes that the barrage only has to be as large as the backlog, and that it has to be repeated as soon as the entries from the previous one begin to time out.² Put Linux's numbers into that, for a server with cookies switched off. A queue capped at 128, the somaxconn default before Linux 5.4, with each spoofed entry held for about 63 seconds, is kept full by 128 SYNs every 63 seconds: 128 / 63 = 2.03, so about two SYNs a second.

A SYN carries no data, so it is only headers: 20 bytes of IPv4 without options and a TCP header of at most 60 bytes, because TCP's header length field is 4 bits counting 32-bit words and 15 x 4 = 60.¹ With no IP options that is at most 20 + 60 = 80 bytes, so take 80 as a generous figure: two a second comes to about 163 bytes per second, or 1.3 kbit/s. On a 1 Gbit/s port that is around a millionth of the capacity, far too little to show on a traffic graph.

A SYN queue at the old somaxconn default

128slots

How long an unanswered SYN holds its slot

63s

Enough to keep 128 slots full (128 / 63)

2SYN/s

Memory per half-open entry on Linux

304bytes

The obvious response is a bigger queue. Here are the sums for bigger ones.

Every row is small. Even 65,536 slots cost about 20 MB, so memory is never what runs out; the fixed cap and the long timeout are. A queue 512 times larger raises the attacker's bill from 2 packets a second to about a thousand. Bigger queues have their own limits: RFC 4987 reports that FreeBSD 4.4 was not designed to scale past backlogs of a few hundred.² Bernstein puts it plainly: large queues "make SYN flooding more expensive but don't actually solve the problem".³

Shortening the timer fails the same way. RFC 4987 calls a shorter SYN-RECEIVED timer flawed, because it breaks down when the attack rate is high or the backlog small.² Halve the timeout and the attacker doubles a rate that is still a trickle, while real clients on lossy links lose the retransmissions meant to rescue them.

SYN cookies: remember nothing, check the answer

The fix came out of a real outage. A SYN flood shut down mail at Panix, an ISP in New York, from 6 September 1996, when security experts generally considered the problem insoluble. Ten days later D. J. Bernstein pointed out how cryptography could solve it and worked out the details with Eric Schenk, who released a Linux implementation in February 1997.³

The idea is to stop storing the half-open entry at all. Bernstein defines SYN cookies as particular choices of the server's initial sequence number.³ RFC 4987 describes the effect: nothing is allocated for a connection in SYN-RECEIVED. Everything the server strictly needs, and most of the rest, is folded into the SYN-ACK's sequence number, and the completing ACK carries enough to rebuild it.² The memory moves out of the server and into the one number that a real client is guaranteed to send back.

Bernstein's page gives the original layout. The 32-bit difference between the server's ISN and the client's ISN is made of three parts: the top 5 bits are a time counter t modulo 32, where t goes up by one every 64 seconds; the next 3 bits encode an MSS (maximum segment size) chosen in response to the client's; and the bottom 24 bits are a secret function of the client's address and port, the server's address and port, and t.³

Follow one connection through a server whose SYN queue is already full.

1. 1

   A SYN arrives and there is no free slot

   The client sends SYN with sequence number 100. Instead of allocating an entry, the server computes the cookie from its clock, the MSS it will use, and its keyed function.

2. 2

   The server answers as if it had room, then forgets

   It sends a SYN-ACK with sequence number 100 plus the cookie and acknowledgement 101, and keeps no record of it.

3. 3

   A spoofed source never answers

   The SYN-ACK goes to an address that did not send the SYN. Nobody replies, and the flood has cost the server one outgoing packet.

4. 4

   A real client echoes the number back

   Its ACK carries sequence number 101 and an acknowledgement number equal to the server's ISN plus one. Acknowledgement minus sequence number gives back the 32 cookie bits.

5. 5

   The server checks the answer and rebuilds the entry

   It recomputes the 24-bit check for a recent value of the clock. If it matches, it rebuilds the entry from the MSS bits and the connection is established; if not, no connection is created.

Steps 2 and 5 are Bernstein's own description of the server's side.³ With made-up but consistent numbers: say t mod 32 is 13 (binary 01101), the MSS index is 5 (101), and the keyed function produces 3A9F12 in hexadecimal. Put together, the 32 bits are 6D3A9F12 in hexadecimal, or 1,832,558,354. The server's ISN is the client's 100 plus that, 1,832,558,454, and the client's final ACK carries sequence number 101 and acknowledgement 1,832,558,455. Subtract one from the other and 1,832,558,354 comes back, with 13, 5 and 3A9F12 inside it. The server stored none of those numbers in the meantime.

This is the RFC 9293 recipe from earlier, a clock plus a keyed function of the addresses and ports, with a few bits spent on data the server wants back, and Bernstein notes that it still obeys TCP's rule that sequence numbers increase slowly.³ The exact encoding is up to each implementation: RFC 4987 mentions a scheme with 7 time-dependent bits and 25 for the rest, and notes that any similar one-way hash works without affecting interoperability, since only the host that made the value ever checks it.²

How safe is a 24-bit check? A blind guess is right roughly once in 2²⁴, or 16,777,216, attempts. Bernstein says as much: whatever function is used, an attacker will eventually forge a connection after millions of random ACK packets, and he suggests making that harder by only rebuilding missing entries if the queue has overflowed recently.³ Keep in mind what a valid cookie proves. It shows that whoever sent the ACK received the SYN-ACK at the source address, which is to say the address is not spoofed. It says nothing about whether the client is friendly.

What cookies cost, and how timestamps win some of it back

Thirty-two bits are not much room, and a normal SYN carries more than an MSS. Its options can also ask for window scaling, selective acknowledgement (SACK) and timestamps, and a server that stores nothing has nowhere to keep those answers. Bernstein's original design paid for this openly: the server had to refuse options such as large windows and pick one of the eight MSS values it could encode, so the biggest effect of a flood on a cookie-protected server was that large windows stopped working.³ RFC 4987 adds that common schemes are incompatible with some TCP options unless they were designed for them.²

Timestamps are the way out. If the ACK that completes the handshake carries a timestamp, timestamps must have been agreed on the SYN and SYN-ACK, and later SACK blocks show that SACK was agreed too, so some options can be inferred without being stored.² FreeBSD 7.0 went further. Because the timestamp value the server sends is echoed back in the Timestamp Echo field of the client's ACK, it becomes a second place to hide state: FreeBSD used those bits for the window scale factors, SACK permission and TCP-MD5, for which a plain cookie has no room. It only works when the client's SYN carried a timestamp option in the first place.²

Linux does the same. When it sends a cookie to a client that offered timestamps, it hides the window scale, SACK and ECN choices in the low 6 bits of its timestamp value, and it picks the MSS from a short table (four values for IPv4), rounding the client's value down.¹¹ Timestamps are on by default.⁴ If the client's SYN carried no timestamp, there is nowhere to keep those choices, and the connection runs without window scaling or SACK.¹¹

Two costs have nothing to do with options. A cookie server cannot keep data that arrives on the SYN itself. And protocols in which the server speaks first have a subtle failure: if the client's final ACK is lost, the server never learns the connection exists and never sends its greeting, while the client believes it is connected and waits. RFC 4987 names SMTP, whose server opens with a "service ready" line, as the example.²

The Linux documentation takes a hard line on all of this. It says syncookies "seriously violate TCP protocol", rule out TCP extensions and can badly degrade services such as SMTP relaying, in ways your clients notice rather than you.⁴ Bernstein disputes this directly: he points out that every packet a cookie server sends is one a normal server could also have sent, and that a connection which loses large windows to a cookie would otherwise have been dropped.³ On a current kernel the warning about extensions holds only for clients that send no timestamps; with them, as above, Linux keeps window scaling and SACK.¹¹

Cookies are still a fallback and not the normal path. RFC 4987, written in 2007, judged SYN caches and SYN cookies the only viable end-host defences found so far, and advised against enabling cookies by default because of the option problems and because they could hamper future TCP extensions that need state.² Linux resolves it with a default of tcp_syncookies = 1, which sends cookies only when a socket's SYN backlog overflows.⁴ On an ordinary day every handshake keeps full state and every option; the trade-offs apply only while a queue is overflowing, when the alternative is turning clients away. Switching cookies off to avoid them just brings the refusals back.

SYN caches, SYN proxies and stopping spoofing at the source

Cookies are one end of a range of answers to the same question: what should a server store when a SYN arrives? A SYN cache stores a small, incomplete entry in a global hash table, placed by a hash of the addresses and ports mixed with secret bits so an attacker cannot aim at one bucket; each bucket has a limit, the oldest entry is dropped when it is reached, and the full TCB is allocated only once the connection is established.² The two combine well: when the cache is full, the server can send cookies instead of evicting entries.²

A SYN proxy moves the work off the server entirely. RFC 4987 describes firewalls that screen connection attempts until they complete and only then proxy them to the protected host, which moves the problem to the firewall and can change TCP's end-to-end behaviour.² Linux can do this itself: a netfilter synproxy answers the handshake with syncookies instead of connection tracking, so a SYN flood on that port cannot eat the limited conntrack table. It depends on syncookies and TCP timestamps being enabled, and because it does not discover the backend's MSS and window scale, you must set them to match.¹³

Every row in that table acts after the spoofed SYNs have arrived. The cleaner fix is for them never to leave the network they came from. BCP 38, published as RFC 2827, describes ingress filtering at a provider's aggregation point: the upstream router accepts packets from a customer only if their source address is inside that customer's prefix, say 203.0.113.0/24, so forged addresses cannot get out.¹⁴ The limits are just as clear: a host can still forge another address inside its own permitted prefix, although the attack is then traceable to that network.¹⁴

It only helps where networks deploy it, which RFC 4987 says is neither guaranteed nor likely, and it does nothing about botnets, which send from their real addresses.² On Linux routers and hosts you run, rp_filter applies the same idea as strict or loose reverse-path checking from RFC 3704.⁴ Spoofing drives other attacks too, notably amplification; how the layers of protection in front of a server fit together is the subject of how DDoS protection works.

Mistakes people make with SYN floods

• Reading it as a bandwidth attack. Against a server without cookies, a flood aimed at the SYN queue can be a couple of packets a second, so a quiet traffic graph proves nothing. A flood that also fills the link is a volumetric attack, a different problem with a different fix, covered in what is a DDoS attack.
• Fixing it with RAM or a bigger queue. Memory is not what runs out, and a bigger queue raises the attacker's packet rate only in proportion. Size the listen() backlog and somaxconn for real traffic, not for floods.
• Using cookies to carry real load. The kernel documentation says cookies must not be used to help a busy server cope with a legitimate connection rate. If SYN flood warnings in the log turn out to be real clients, it points to other settings instead, naming tcp_max_syn_backlog, tcp_synack_retries and tcp_abort_on_overflow.⁴ On a current kernel the queue size itself comes from the listen() backlog.
• Blaming a flood for a slow application. When the accept queue is full, Linux drops new SYNs and counts each one in TcpExtListenOverflows.⁵ To a client that looks just like a SYN flood. If that counter climbs while the cookie counters stay still, your application is not calling accept() fast enough, and the fix is in the application.
• Expecting cookies to stop every TCP flood. A valid cookie proves only that the source address is real. RFC 4987 notes that a SYN flood can also come from a botnet using real addresses, each machine suppressing its own reply to the SYN-ACKs.² Cookies handle that as well, since nothing is stored until an ACK arrives. What they cannot stop is machines that finish the handshake: that is a flood of real connections, which needs per-source limits and filtering in front of the server.

Checking your own Linux server

A few settings decide how a Linux server behaves under a SYN flood, and reading them changes nothing. The values below are an example; tcp_max_syn_backlog in particular depends on how much memory the machine has.

$ sysctl net.ipv4.tcp_syncookies net.ipv4.tcp_max_syn_backlog net.ipv4.tcp_synack_retries net.core.somaxconnnet.ipv4.tcp_syncookies = 1net.ipv4.tcp_max_syn_backlog = 1024net.ipv4.tcp_synack_retries = 5net.core.somaxconn = 4096

A tcp_syncookies of 1 is the default and the one you want: cookies only when a socket's SYN backlog overflows. The documentation offers 2, which sends cookies for every connection, as a way to test what cookies do to your traffic.⁴ If yours reads anything else, find out who changed it and why. With cookies enabled, the listen(2) manual adds, the queue of incomplete connections has no logical maximum and tcp_max_syn_backlog is ignored,⁶ which is the practical meaning of a cookie: the queue overflows into cookies instead of into refusals. A tcp_synack_retries of 5 gives the 63-second lifetime used in the arithmetic above.

somaxconn is only a ceiling. The number that actually caps each socket's SYN queue is that socket's own listen() backlog, and ss shows it: -l lists only listening sockets and -n keeps ports numeric.⁷

$ ss -ltnState  Recv-Q Send-Q Local Address:Port Peer Address:Port ProcessLISTEN 0      4096         0.0.0.0:22        0.0.0.0:*LISTEN 0      511          0.0.0.0:443       0.0.0.0:*

On a listening socket the two queue columns mean something different from usual. The kernel fills the receive queue with the number of completed connections waiting for accept() and the send queue with the socket's backlog.¹² In this example, port 443 starts sending cookies once more than 511 connection attempts are half-open, and a Recv-Q that sits near its Send-Q means the application is falling behind on accept().

During an incident, two read-only commands tell you which problem you have. The first counts connections in the syn-recv state, a state filter ss accepts directly;⁷ the second reads the kernel's own counters for cookies and listen-queue overflows.

$ ss -Htn state syn-recv | wc -l0$ nstat -az TcpExtSyncookiesSent TcpExtSyncookiesRecv TcpExtSyncookiesFailed TcpExtListenOverflows TcpExtListenDrops#kernelTcpExtSyncookiesSent            0                  0.0TcpExtSyncookiesRecv            0                  0.0TcpExtSyncookiesFailed          0                  0.0TcpExtListenOverflows           0                  0.0TcpExtListenDrops               0                  0.0

The kernel documentation defines them: TcpExtSyncookiesSent counts cookies sent, TcpExtSyncookiesRecv counts replies to cookies received, TcpExtSyncookiesFailed counts cookies whose decoded MSS was invalid, and TcpExtListenOverflows counts SYNs dropped because the accept queue was full.⁵ Read them together. Cookies sent climbing fast while replies stay near zero is a spoofed flood being answered and ignored, which is the defence working. Sent and received climbing together suggests real clients arriving faster than the queue allows, a capacity problem. Listen overflows climbing on their own point at your application.

The syn-recv count is the number of half-open entries the kernel is holding. If it stops growing at about the backlog of the port under attack, that socket's SYN queue is full and further SYNs are getting cookies, or being dropped if cookies are off.⁸ So when new clients cannot connect, read the syn-recv count and these counters before you read the bandwidth graph.