NUMA Optimizations in the FreeBSD Network Stack

I will discuss optimizations to keep network connections and their resources local to NUMA domains. These changes include:
Allocating NUMA local memory to back files sent via sendfile(9).
Allocating NUMA local memory for Kernel TLS crypto buffers.
Directing connections to TCP Pacers and kTLS workers bound to the local domain.
Directing incoming connections to Nginx workers bound to the local domain via modifications to SO_REUSEPORT_LB listen sockets.
I will present data from real Netflix servers showing an improvement of almost 2x on AMD EPYC (85Gbs - 165Gbs), and 1.3x on Intel Xeon (140Gb/s - 180Gbs). I will present data from the Xeon system showing a 50% reduction in cross-domain traffic.
Drew Gallatin
Drew started working on FreeBSD at Duke in the 90s, and was one of the people behind the FreeBSD/alpha port. He worked on zero-copy TCP optimizations for FreeBSD and was sending data at over 1Gb/s before gigabit Ethernet was generally available. He spent a decade at Myricom, optimizing their drivers. After a brief hiatus at Google, he landed at Netflix, where he works on optimizing the FreeBSD kernel and network stack for content delivery. He worked on the optimizations to serve unencrypted Netflix traffic at 100Gb/s, and then on more optimizations to send encrypted traffic at 100Gb/s.

[@phaub]

Thank you for the upload. This is very interesting.

[@kimp_ossible]

27:00 Maybe Xeon performance without Siloing dropped due to CPU security fixes present on Intel that aren’t on AMD?

[@Transmissions]

I wonder if this was with SMT enabled or disabled?

[@jkmicha]

i doubt smt matters very much since the cpu load will be on the memory controllers and on the AES stack

[@mc4ndr3]

bruh i can add 10 more GB/sec and drop saturation to zero with this simple trick... jes pull out cpus and nics until you don't have 2 numa boundaries. no more numa boundary crossings. win monies. dream big. alternatively, present copies of the data on redundant disks, one copy per cpu. again, numa crossings drop to zero. no matter which nic the packet come in, the data will be there for that nic's cpu.