XDP and eBPF: filtering packets at the network driver

A Linux server does a surprising amount of work for a packet before any program sees it. The kernel sets aside memory to describe it, runs it past the firewall, looks up the connection it belongs to and finds the socket waiting for it. For a real request that work is worth doing. For the ten-millionth packet of a flood it is the attack itself: the machine is kept busy accepting garbage, and real traffic queues behind it.

XDP moves the decision to the front of that queue. The network driver hands each packet to a small program, one you wrote and the kernel checked, before the kernel has spent anything on it, and the program says what happens next. The program is eBPF, the same mechanism that tracing tools and container networking are now built on.¹ Two questions follow: how a kernel can safely run code it did not ship with, and why the driver is the cheapest place there is to throw a packet away.

eBPF: small programs the kernel agrees to run

The kernel is the one program on a Linux machine that sees everything: every packet, every system call, every file opened. Changing what it does used to mean getting a change accepted into the kernel source and waiting for it to reach your machines, or loading a kernel module: compiled code with full privileges, where one bug can take the whole machine down. eBPF is a third way: the kernel accepts a small program from outside, checks it, and runs it inside its own address space, with no patch to the kernel source and no module to load.¹

The programs are event driven. Each one is attached to a hook and runs when the kernel or an application passes it. The predefined hooks include system calls, function entry and exit, kernel tracepoints and network events, and kernel or user probes (kprobes and uprobes) can attach a program almost anywhere else.¹ The same machinery can time a slow disk, enforce a security policy or, as here, judge network packets. On the network path alone there are several hooks: XDP in the driver, tc on the way in and out once the kernel has built its own record of the packet,¹⁰ and socket filters on individual sockets further up.⁹

A program reaches the kernel as bytecode, instructions for a small virtual processor. You could write bytecode by hand, but in practice people write a restricted form of C and let LLVM compile it.¹ The virtual processor was designed with real ones in mind. eBPF has ten registers and a read-only frame pointer, up from the two registers of classic BPF, and it was designed to be compiled to native code with a one-to-one mapping: on 64-bit architectures every eBPF register maps onto a hardware register.⁵

What a program may call is limited too. It cannot call arbitrary kernel functions, only helper functions, a stable interface the kernel offers for jobs such as reading the time, generating a random number, reaching a map or rewriting a packet.¹ Loading one is a privileged act: the program goes in through the bpf() system call, usually by way of a library, and the process loading it must be root or hold the CAP_BPF capability, unless an administrator has enabled unprivileged eBPF, which allows only a reduced set of programs.¹ On its way in, every program passes two steps before it is attached to its hook: verification and JIT compilation.¹

The verifier: proof before the first run

The verifier is what makes it reasonable to let outside code into the kernel at all. It does not run the program to see what happens. It evaluates every possible path through it, and accepts the program only if that analysis finishes and shows every path to be safe.¹

The kernel documentation describes the work in two steps. The first checks the shape of the program's control flow and rejects, among other things, instructions that can never be reached. The second starts from the first instruction and descends every possible path, simulating each instruction and tracking what every register and stack slot holds as it goes.² Out of that simulation come the rules a program has to satisfy:

• It always finishes. A program may never block or loop forever. Loops are allowed only when the verifier can see an exit condition that is guaranteed to become true.¹
• It never reads what it has not written. The stack can be read only where the program has already stored something, and a program that reads a register it never set is rejected with an error such as R2 !read_ok.²
• It never reaches outside its memory. Out-of-bounds access is refused.¹For an XDP program that means proving, before each read, that the offset is still short of data_end, the end of the packet.⁷
• It checks what a map gives it. A map lookup can come back empty, and the verifier rejects a program that uses the result without testing it for NULL, even where the key is certain to exist.¹²
• It fits the verifier's limits. An unprivileged program may have at most 4,096 instructions, and the verifier stops after exploring 1 million instructions across all paths.³

Once a program is accepted, the kernel hardens it: the memory holding it is made read-only.¹

The verifier accepts bounded loops,¹ though older material says eBPF cannot loop at all, including the opening of the kernel's own verifier page, which still describes loops as disallowed,² and an answer in its design FAQ that calls safe loops "not clear yet".³

The verifier guards the kernel, not your intentions: it proves the program cannot crash or hang the machine and has no opinion on what the program is for.¹ A program that drops every packet on the machine passes verification without complaint. That is why loading one takes root or CAP_BPF, and why an eBPF program from someone else deserves the same scrutiny as any other root-level code.

JIT compilation and maps: native speed, and memory between packets

After verification, a just-in-time (JIT) compiler translates the bytecode into the processor's own instruction set, so the program runs about as fast as code compiled into the kernel.¹ Verification and compilation both happen once, at load time,⁶ so the cost of safety is paid up front. On the per-packet path the program is plain machine code.

A program runs once per event and then forgets everything, so anything it has to remember lives in a map. Maps are the kernel's generic storage for sharing data between the kernel and user space.⁴ The types include hash tables, arrays, LRU (least recently used) tables, ring buffers, stack traces and longest-prefix-match tables, and many come in a per-CPU variant as well as a shared one. Programs reach maps through helpers, and ordinary applications reach the same maps through a system call.¹

That split is the usual shape of an eBPF system. The program in the kernel is the fast path, which looks things up and counts; a normal program in user space is the control plane, which decides policy and writes it into maps. For a packet filter, the control program writes addresses into a blocklist map and reads drop counters back, while the XDP program checks the blocklist on every packet. Change the list and the next packet sees it, with nothing to reload. A longest-prefix-match map lets one entry cover a whole range of addresses, such as a /24.

Where XDP sits: before the kernel spends anything

Follow a packet into a Linux server. The network card writes the frame into memory, into a receive ring the driver looks after, and the driver picks it up. Without XDP, the next thing that happens is the expensive part: the processor allocates an sk_buff, the socket buffer structure that stands for a packet everywhere in the networking stack, fills it in and passes it on to GRO, the kernel's generic receive offload.¹⁰ From there the packet meets the tc hooks, the netfilter hooks that iptables and connection tracking use, the IP and TCP or UDP code, and at last a socket. By the time a firewall rule or an application decides the packet was unwanted, every one of those stages has been paid for.

XDP runs its program from a hook in the network device driver, each time a packet arrives, at the earliest possible moment after the hardware hands it over: before the kernel allocates its per-packet sk_buff and before it parses anything.⁶ The drawing at the top of this page is that path, with the program in the second box.

The two main exits show why that position matters. XDP_DROP discards the packet at the driver without spending any further resources on it. XDP_PASS means the processor now allocates the sk_buff (skb for short), fills it and hands it to GRO, exactly as if no program had run.¹⁰ A packet the filter passes costs what it would have cost anyway, plus the program's own run: in the paper's tests, an XDP program that only counted packets added 13.3 nanoseconds to each one.⁶

XDP is not kernel bypass. Frameworks such as DPDK take the network card away from the kernel and run the data path in user space; XDP does the opposite and moves the performance-sensitive part of packet processing into the kernel, ahead of the normal stack.⁶ The kernel keeps its drivers, its tools and its security model, and packets the program passes carry on as usual.¹⁰

The per-packet budget, worked through

The smallest Ethernet frame is 64 bytes, and on the wire each one also needs a 64-bit preamble in front and a 96-bit gap behind,¹⁶ so it occupies 512 + 64 + 96 = 672 bits. A 10 Gbit/s link full of them delivers 10,000,000,000 ÷ 672 ≈ 14.88 million packets a second. For one core to keep up, it has 1 ÷ 14,880,000 ≈ 67 nanoseconds per packet, and on a 3.6 GHz processor that is about 67.2 × 3.6 ≈ 242 clock cycles for everything the kernel does with each one.

The XDP paper measured how fast one core of exactly such a processor, a six-core Xeon E5-1650 v4 at 3.6 GHz running a pre-release of Linux 4.18, could drop minimum-size packets in different places.⁶Turned into time per packet, the results sit either side of that 67-nanosecond line.

Dropping in the normal stack with connection tracking, which the paper notes is on by default in many Linux distributions, ran at 1.8 million packets a second, about 556 ns each. The iptables raw table, which the authors chose because it gives the earliest possible drop in the normal stack, reached 4.8 million, about 208 ns each. XDP dropped 24 million a second, 41.6 ns per packet, a five-fold improvement on the fastest normal-stack drop.⁶ The paper found each of them scaling linearly with cores, so the cost of a 14.88-million-packet flood is easy to work out: 14.88 ÷ 1.8 ≈ 8.3 cores with conntrack, more than the machine had; 14.88 ÷ 4.8 ≈ 3.1 cores in the raw table; 14.88 ÷ 24 ≈ 0.62 of one core in XDP.

These are 2018 figures on 2018 hardware; newer processors and kernels post different numbers, and the ratio, not the absolute speed, is the lesson. And XDP is not the fastest bar on the chart. DPDK managed 43.5 million packets a second, 22.9 ns each.⁶ XDP's case is that it comes within a factor of two of that while leaving the kernel in charge of everything it does not drop.

Five verdicts: what the program can do with a packet

An XDP program is called with a small context, xdp_md, whose main fields point at the start and the end of the packet's data.⁷ There is no socket and no sk_buff yet. The program reads what it needs, perhaps consults a map, perhaps rewrites a few bytes, and returns one of five values, which the kernel's linux/bpf.h header defines as an enum.¹⁰

• XDP_DROP discards the packet. Because that happens so early, the packet is invisible to tools such as tcpdump, and the eBPF documentation suggests keeping your own record of drops to stay able to see them.⁷
• XDP_PASS hands the packet to the normal network stack, changed first if the program likes.⁷
• XDP_TX sends the packet back out of the port it arrived on, again optionally modified.⁷ A load balancer that rewrites headers and bounces packets on to a backend lives on this one.
• XDP_REDIRECT sends it somewhere else: out of another interface, including the virtual interface of a virtual machine, to another CPU for further processing, or to a program in user space through an AF_XDP socket.⁶ A program should never return it on its own: a helper call picks the target first, usually from a map of devices, CPUs or sockets.⁷
• XDP_ABORTED means the program hit an error. The packet is dropped just as with XDP_DROP, and the xdp_exception tracepoint fires so the misbehaviour can be monitored.¹⁰ It is also expensive, and the documentation advises against using it casually in production.⁷

A complete filter: a blocklist in C

Here is a complete XDP program, written the way the XDP project's own tutorials write theirs.¹² It drops IPv4 packets whose source address is in a blocklist map, counts what it drops per address, and passes everything else.

// SPDX-License-Identifier: GPL-2.0#include <linux/bpf.h>#include <linux/if_ether.h>#include <linux/ip.h>#include <bpf/bpf_helpers.h>#include <bpf/bpf_endian.h>struct {    __uint(type, BPF_MAP_TYPE_HASH);    __uint(max_entries, 65536);    __type(key, __u32);     // IPv4 source address    __type(value, __u64);   // packets dropped from it} blocklist SEC(".maps");SEC("xdp")int xdp_blocklist(struct xdp_md *ctx){    void *data     = (void *)(long)ctx->data;    void *data_end = (void *)(long)ctx->data_end;    struct ethhdr *eth = data;    if ((void *)(eth + 1) > data_end)      // prove the read stays inside        return XDP_PASS;    if (eth->h_proto != bpf_htons(ETH_P_IP))        return XDP_PASS;                   // not IPv4: not our business    struct iphdr *ip = (void *)(eth + 1);    if ((void *)(ip + 1) > data_end)        return XDP_PASS;    __u32 src = ip->saddr;    __u64 *count = bpf_map_lookup_elem(&blocklist, &src);    if (!count)                            // the verifier insists on this        return XDP_PASS;    __sync_fetch_and_add(count, 1);        // several CPUs may count at once    return XDP_DROP;}char LICENSE[] SEC("license") = "GPL";

Every read from the packet comes after a comparison with data_end, because the verifier will not accept a read it cannot prove stays inside the packet.⁷ The map lookup is tested for NULL before it is used, because without that test the program is rejected, and the counter goes up with an atomic add, because the same entry can be updated from several CPUs at the same moment.¹²

Native, generic and offloaded: where the hook really runs

Everything so far assumes the network driver itself calls the program. That is native or driver mode, and it needs support in the driver, which most major network card vendors have in recent kernels.⁷ Where a driver lacks it, the kernel can still run the program in generic mode, also called SKB mode. Generic mode always works, because it calls the XDP program after a socket buffer has already been allocated further up the stack, and the eBPF documentation is blunt about the cost: it negates the speed advantage of XDP.⁷ In the drawing at the top of the page, generic mode moves the program from the second box to the far side of the dashed line.

A third mode, hardware offload, runs the program on the network card itself. It needs support in both the driver and the card, and only select Netronome devices offer it.⁷

You can see which mode a program is in without special tools: ip link shows xdp for driver mode, xdpgeneric for generic mode and xdpoffload for hardware offload.¹⁴ The XDP project's tutorial shows the generic case on the loopback interface:¹³

$ sudo ip link show dev lo1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 xdpgeneric qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00    prog/xdp id 408 name xdp_prog_simple tag 3b185187f1855c4c jited

The trap is in how programs get attached. Asked for plain xdp, ip link tries driver mode and, if the driver has no native support, the kernel quietly falls back to generic. Asking for xdpdrv turns that fallback off, so a driver that cannot do it returns an error instead.¹⁴ If xdpdrv fails, your driver has no native XDP, and the eBPF documentation then recommends a tc program over generic XDP: once the socket buffer has to be built anyway, tc can do more with the packet for roughly the same cost.⁷

For the blocklist program above, the round trip looks like this: compile it with clang into an ELF object and attach it with ip link, as the XDP project's tutorial does,¹³ then block one address with bpftool, read the map back and check the mode.¹⁵

$ clang -O2 -g --target=bpf -c xdp_blocklist.c -o xdp_blocklist.o$ sudo ip link set dev eth0 xdpdrv obj xdp_blocklist.o section xdp$ sudo bpftool map update name blocklist key 203 0 113 7 value 0 0 0 0 0 0 0 0$ sudo bpftool map dump name blocklist$ ip link show dev eth0

The third command is the control plane at its simplest. The key bytes go in the order they appear on the wire, so 203 0 113 7 is 203.0.113.7, because the program looks up saddr without converting its byte order; the value is the drop counter's eight bytes, starting at zero. The next packet from that address is dropped with nothing reloaded, and the dump shows its count. If a filter benchmarks badly, check the mode on the last command's first line before anything else: a program in generic mode pays for the socket buffer it was written to avoid.

What XDP is used for: dropping floods and balancing load

The eBPF documentation names the notable uses plainly: DDoS protection, load balancing and high-throughput packet filtering.⁷

DDoS filtering on the server itself

In the XDP paper's DDoS test, one core was given a baseline load of 35,000 TCP transactions a second, standing in for real users, while a growing flood of small UDP packets that matched a filter rule hit the same machine, so the two had to compete for the same core. Without the XDP filter, the legitimate transactions halved at 3 million attack packets a second and were effectively zero just below 3.5 million. With the filter in place they held steady at about 28,500 a second, against the 35,000 offered (the paper does not break down the difference), until the attack reached 19.5 million packets a second.⁶ The authors' summary is that one core comfortably handles 10 Gbit/s of minimum-size attack traffic.⁶

The filter is transparent to the applications on the host, so nothing about the service has to change. And on a machine that runs virtual machines, the filter can be installed in the hypervisor and protect every guest on the host at once.⁶ For the rest of the defence around it, from upstream scrubbing to rate limits, see how DDoS protection works; for the flood aimed at the TCP handshake, see SYN floods and SYN cookies.

Load balancing: Katran

Katran, the load balancer Facebook released as open source, is a C++ library and a BPF program that together make a layer 4 forwarding plane on top of XDP.¹¹ For each packet, the XDP program hashes the header to choose a backend server, wraps the packet in an outer header addressed to that server, and sends it straight back out of the interface it came in on. The backend unwraps it and replies to the client directly, and the load balancer's configuration lives in BPF maps.⁶

It is fastest in driver mode and scales with the card's receive queues, because the program runs independently on each queue and keeps its state in per-CPU maps, with no locks.¹¹ On the paper's test machine, Katran forwarded 5.2 million packets a second on one core and 29.3 million on six, against 1.2 and 7.3 million for IPVS, the load balancer built into Linux: a 4.3-fold gain.⁶

Limits, and the mistakes they cause

It only sees incoming traffic. XDP programs are called for every packet a device receives, and for nothing it sends.⁷ To filter or rewrite outgoing traffic you need the tc hook, which can attach to both ingress and egress¹⁰ but is called with a socket buffer that already exists.⁸

It cannot help once the link is full. The program runs on the server, after the packet has crossed the wire. A 1 Gbit/s port carries at most 1,000,000,000 ÷ 672 ≈ 1.49 million minimum-size frames a second, and whatever is sent beyond the port's capacity is lost before it reaches the driver, real traffic included. Floods bigger than the link have to be absorbed upstream; see how DDoS protection works.

Its drops are invisible to tcpdump. A quiet capture on a filtered server does not mean nothing is arriving.⁷ Read the filter's own counters, which is why the example above counts every drop into a map.

The fallback to generic mode is silent. A program that fell back to generic keeps running but has lost its speed advantage, and some actions are not available that far up the stack.¹⁴⁷ Attach with xdpdrv, as shown above.

Large MTUs can refuse it. Without fragment support there is a maximum MTU between 1,500 and 4,096 bytes depending on the driver, and a program cannot be attached to a device configured above it.⁷ Jumbo frames and XDP need planning together.

The verifier shapes the code. Every packet read needs its bounds check, every loop a provable end, and the whole program has to verify inside the complexity limit.³ The deeper a program parses, the more paths the verifier has to walk, which is one more reason to keep XDP for quick decisions made from the first bytes of a packet.

Drop in XDP what you can recognise cheaply and early: addresses on a blocklist, malformed headers, traffic that does not look like the protocol a port serves. Pass everything else to a stack that is better at the rest. Keep state in maps, count every drop, and confirm the attach mode before you trust a benchmark.