Our eBPF port scanner reported ports it never probed
A postmortem: how an AF_XDP engine turned a full send queue into "filtered" ports, the kernel limit behind it, and what AF_XDP copy mode really costs.
tcpcat is a port scanner written in Go with two SYN-scan engines. One sends probes through ordinary raw sockets. The other, enabled with --ebpf, attaches a small XDP program to the network interface and moves frames through AF_XDP sockets. That second engine is the one the project was built around. This post is about the evening we found that, under load, it was reporting ports as filtered that it had never sent a single packet to.
Short version. With 4,096 concurrent probes, the AF_XDP engine found 3 of 80 open ports in our lab. 91% of its probes never left the machine; they were reported as FILTERED anyway. Three bugs combined: a send queue that gave up when full, timeouts that started when a frame was queued instead of when it was sent, and a kernel limit of 32 frames per sendto in AF_XDP copy mode that our transmit path ignored. All three are fixed in v1.7.0. The fix for the fourth finding, that copy-mode AF_XDP is simply slower than raw sockets, was to stop using it there.
The setup: a benchmark that checks answers, not just times
We measure tcpcat in a Docker lab: two target containers, each listening on 40 TCP ports picked from a fixed seed, every other port closed. A full scan of both is 131,070 probes, and the lab knows exactly which 80 ports are open. Each tool runs five times, timed by hyperfine, and every run is compared with that list. A run that misses an open port is invalid, however fast it was. (The protocol has its own post.)
That rule is the only reason this story exists. A timing-only benchmark would have shown the eBPF engine doing just fine.
The first symptom: a speed limit nobody set
On a Linux 6.12 server, the AF_XDP engine took about 5.4 seconds for the full scan with a rate ceiling of 1,000,000 packets per second, and about 5.8 seconds with a ceiling of 25,000. The ceiling barely mattered: the engine topped out near 24,000 frames per second on its own. The raw-socket engine, on the same lab, was soon doing the same scan in 1.3 seconds.
Both engines share the same pool of workers. Each worker sends a probe and waits for its reply before taking the next port, so throughput is roughly workers ÷ round-trip time. If the engine were latency-bound, adding workers would speed it up. So we tried:
Workers (-w) | Time | Open ports found |
|---|---|---|
| 256 | 6.0 s | 80 / 80 |
| 1,024 | 5.4 s | 80 / 80 |
| 4,096 | 1.3 s | 3 / 80 |
Four times the workers bought almost nothing, and at 4,096 the scan finished fast because it had stopped scanning. That is not an exotic setting: tcpcat's default is 32 workers per CPU core, which is 4,096 on a 128-core machine.
What the JSON report said
We ran the 4,096-worker scan again and read the per-port reasons in its report:
119,255 FILTERED XDP TX Ring Congestion (Fragment dropped)
10,548 CLOSED RST Received (AF_XDP)
1,258 FILTERED No Response (Timeout)
9 OPEN SYN-ACK
119,255 of 131,070 probes had never been transmitted, and each one was reported as a filtered port. A firewall had nothing to do with it. The scanner had failed to send, and wrote that down as a fact about the target.
Bug 1: a full queue turned into an answer
All probes go through one TX ring, owned by a single flusher goroutine that batches frames into one sendto. Workers hand frames to it through a Go channel of 8,192 slots. When the channel was full, the send was non-blocking:
select {
case xdpTxQueue <- frame:
default:
return false // caller reports "XDP TX Ring Congestion" as FILTERED
}
The fix is to wait for room (with a bound) and let the workers slow down to what the flusher can send. A slower scan is fine. A wrong one isn't. If a probe still cannot be sent, the port is now reported as Not probed, the end of the scan says how many there were, and the exit status is non-zero.
Bug 2: the timeout clock started too early
With waiting in place, the scan stopped dropping probes, but at 4,096 workers it still missed a few open ports, this time through timeouts. The replies that did arrive took 53 ms at the median and up to 115 ms, on a virtual link whose real round-trip time is well under a millisecond.
The probe's timeout started when its frame entered the queue. With thousands of frames ahead of it, a probe could spend its whole timeout waiting in line, time out before it had even left, and get retried, which put one more frame in the queue. The RTT estimator was learning the length of our own queue.
The flusher now tells each probe when its frame has actually been placed on the TX ring, or that it could not place it. The attempt's clock starts then. Median reply latency went back to about one millisecond.
Bug 3: 32 frames per kick
Starting the clock at the right moment exposed the last problem: thousands of probes could not get onto the ring at all, because the ring (2,048 descriptors) was full. Something was putting frames on it faster than they left.
The interface was a veth, which has no zero-copy support, so AF_XDP ran in copy mode. In that mode the kernel's transmit path (net/xdp/xsk.c) sends at most TX_BATCH_SIZE, 32 descriptors, per sendto, and returns EAGAIN if more are waiting. The AF_XDP library we use stops at that EAGAIN. Our flusher put 64 frames on the ring per batch and kicked the kernel once, so up to half of every batch stayed behind until the next batch's kick. Under load, the leftovers piled up until the ring was full.
The fix kicks again while the kernel keeps consuming descriptors, and stops as soon as a kick makes no progress, so it never spins while holding the socket lock. It only runs in copy mode, where the limit applies:
pending := txRingSize - xsk.NumFreeTxSlots()
for kicks := 0; pending > 0 && kicks < 8; kicks++ {
sendto(xsk.FD(), MSG_DONTWAIT) // EAGAIN/EBUSY/EINTR are fine
left := txRingSize - xsk.NumFreeTxSlots()
if left >= pending {
return // no progress: the device is busy, the next flush retries
}
pending = left
}
One honest detour: we suspected this limit first and tried the extra kicks on their own, at the default 256 workers. Nothing changed measurably (5.39 s before, 5.14 s ± 0.73 after), so we set the change aside. It only mattered once the first two bugs were fixed and the ring could actually fill. A fix that does not move the number you are watching may still be right. It was just aimed at the wrong number.
With all three fixes, the same lab scan finds every open port at every concurrency we tried:
| Workers | Before | After | Timeouts after |
|---|---|---|---|
| 256 | 80 / 80 | 80 / 80 | 0 |
| 1,024 | 80 / 80 | 80 / 80 | 0 |
| 4,096 | 3 / 80 | 80 / 80 | 0 |
| 8,192 | not run | 80 / 80 | 0 |
And then we stopped using it in copy mode
Correct is not the same as fast. In copy mode the kernel still copies every frame, as it does for a raw socket, and all transmission goes through the one thread that owns the TX ring. The raw-socket engine sends from every core. On the lab, the same full scan took 6.74 s with AF_XDP in copy mode and 2.26 s with raw sockets, both finding 80 of 80.
So --ebpf now checks how AF_XDP came up. If the driver offers no zero-copy, tcpcat says so and runs the same scan on raw sockets. --ebpf-copy keeps AF_XDP anyway. Where the engine should shine is a NIC with zero-copy and several queues, which we have not measured yet. Until we have, we do not claim it.
One more thing: a panic on a cable pull
While testing failure paths, we brought the interface down in the middle of a scan. The AF_XDP library panics on any sendto error other than EINTR, EAGAIN or EBUSY. Here it was ENETDOWN, from the flusher or from the receive loop sending a RST, and the whole process died. Transmits now go through a wrapper that recovers, logs the error once, and marks the remaining ports as not probed. The report is still written, and the scan exits 1 because it is incomplete.
What we took away
- Never report a state you did not observe. "I could not send the probe" and "the target did not answer" are different facts. Merging them is how a full queue became a firewall.
- Start the timeout when the packet leaves. Anything that queues before the wire will otherwise show up as network latency, and then as loss.
- Check correctness before speed. Every one of these bugs made the scan look faster.
- AF_XDP copy mode is not a fast path. Without zero-copy, it adds a single-threaded bottleneck to a kernel copy that raw sockets also pay.
- Read the kernel source for your transmit path. The 32-frame limit is a few lines in
xsk.c, and the library above it did not mention it.
Reproduce it
The fixes are in #60 and #61, released in v1.7.0. The lab is in the repository:
git clone https://github.com/NycolazSec/tcpcat && cd tcpcat
./benchmarks/run.sh # full protocol, needs Docker
RATE=1000000 ./benchmarks/run.sh # no practical rate ceiling
Our figures come from a virtual interface on a shared server. If you have a NIC with AF_XDP zero-copy support, we would very much like to see your numbers.