iPerf3 tuning - single instance reaching 87Gbps on a 100Gbps Ethernet NIC

[@benja2a]

Nice video, good explanation.

[@munirawan3609]

Thanks for such an informative tutorial. What could be the possible ways to tune iperf3 for UDP based traffic flows for 25 NIC?? And is using steams and server client pairs the same thing?

[@ewenchan1239]

Sorry for resurrecting an old video of yours, but just testing this now: I'm also Using a Mellanox ConnectX-4 100 Gbps dual VPI ports, one is set to IB, the other is set to ETH. The two ETH systems are linked together directly (point-to-point) connection with a copper DAC. Ryzen 9 5950X, with 128 GB DDR4-3200 Unbuffered, non-ECC RAM, running Promox 7.4-17 (Debian 11), using the Debian Infiniband drivers. Single instance of iperf3: server: iperf3 -s clinet: iperf3 - c -l 1M -Z 10 second average: 58 Gbps *edit* Tested with normal iperf rather than iperf3: Host-to-host, 8 parallel streams: 96.9 Gbps (out of a possible 100 Gbps) CentOS 7.7.1908 VM to CentOS 7.7.1908 VM via a Linux (ethernet) bridge -- single stream: 23.4 Gbps. 8 parallel streams: 23.0 Gbps.

[@kaisercheng6178]

I have almost the same testing value like you. New Linux kernel seems slower.

[@ewenchan1239]

@@kaisercheng6178 I don't think that it is necessarily a kernel thing. There are so many variables and factors at play here, that you can probably pareto the most significant parameter which contributes to the result(s) that we are seeing. Conversely, there is also a theory where, for my 5950X, I would need at least Linux kernel 5.4 to even BEGIN to remotely use said 5950X. The old 3.10.1062 kernel that ships with CentOS just won't cut it for my 5950X. I don't necessarily need the newest kernel, but I also can't use a kernel that's THAT old neither.

[@kaisercheng6178]

@@ewenchan1239 We just try Centos 7.2 and Ubuntu 22.04, they have around 10% difference. but this is not 100G test, it just a 1Gbps test. Centos 7.2 is faster (aroung 940Mbps with iperf3)

[@boemlauw]

The MTU size is freaking me out, also be sure you (while testing) disable iptables (And SELinux if mint has that) as they will choke your cpu at this rate. Please set your your mtu on both NIC'c to 9000, (ifconfig enp1s0 mtu 9000) [ 11] 0.0-25.0 sec 36.9 GBytes 12.7 Gbits/sec [SUM] 0.0-25.0 sec 288 GBytes 98.9 Gbits/sec

[@ranjanshrestha8456]

nice one. what's the default tcp window size used in iperf3 ? thanks.

[@munirawan3609]

For TCP it's 128K and for UDP it's 1470.

[@nicholasjohnson3542]

Is this using IPoIB or a pure 100GbE card?

[@cpspot]

Strictly 100Gb Ethernet. No InfiniBand.

[@Mr_ST_720]

Thanks ..min threads is better