IP over 40Gbit Thunderbolt vs 10Gbit Ethernet. Thunderbolt is slower!

[@fanshaw]

Wikipedia: When connected to a Thunderbolt device, the per-lane data rate becomes 10 Gbit/s and the four Thunderbolt lanes are configured as two duplex lanes, each 10 Gbit/s comprising one lane of input and one lane of output. So each lane is 10gb/s - that's your max in one direction unless you can bond and load-balance the channels. There's probably a bit more overhead than ethernet because of the different signal types (data, video) which have to be accounted for. To speed up small files, you could try tar'ing them up. Then the transfer is one file so less latency waiting for the receiving node to work out the file-system for each file mid-transfer. This is the problem with Apple. You only get what they give you. There's no 40/100g network options for when you actually want to go fast. I recently put in a PCIE quad ssd card in my server for $30 + storage drive costs.

[@jeffcullen6573]

Math tells us 100 gigabytes over 10 gigabit should take under 90 seconds. Under 30 seconds over 40 gigabit. Your results are well within any reasonable margin of error, and your bottleneck isn't the network. Run an iperf test to prove network link performance, not a file copy. --a network engineer

[@tronicdude6]

Please do this! Happy to give instruction if you're not sure how.

[@postnick]

The Mac SSD may not be able to write that fast. you'd need a ramdisk test.

[@jeffcullen6573]

@@postnick could also be single channel SMB playing a role.. regardless, iperf is the source of truth when it comes to measuring network throughput.

[@amateurwizard]

Yep, it has a lot of overhead, unless you're on a Linux machine where you can do unholy things to thunderbolt to eek out more networking performance. The 10Gb limit comes from the fact that that is all Thunderbold 4 reserves for PCIe bandwidth, which it uses here for networking. (not the usual display / data applications which should reach near 40G)

[@franckleveneur676]

In my experience, copying 1000s of smaller files takes longer than 1 large files. I believe it has to do with checking the file has been copied on the other end + your are actually doing random writes instead of 1 potential sequential write. Random writes are slower than sequential writes no matter what, even with SSD (ask ChatGPT for clarification). That’s why you larger files size but fewer are much faster to write (copy) I would probably compress all the 1000 files into 1 file copy over and decompress.

[@abyssPonderer]

Thunderbold does add some over head and also thunderbold 3/4 reserves around 8GiB of the bandwidth for video. When I tried this in windows and linux, it gave me around 20GiB of throughput. This overhead specially impact sequential small transactions, which I thinking is what is going on, copying with multi threads should give you better results, maybe even increasing mtu could help but I don't think the thunderbolt network driver have that feature.

[@lahmyaj]

I believe Thunderbolt Networking has a built-in restriction of 10GbE speeds for some reason. Not sure why but possibly something to do with how it multiplexes data, it leaves room for DisplayPort and PCIe bandwidth.

[@abyssPonderer]

The standard only reserves around 8G of bandwidth for video, and the rest (around 32G) is for everything else, I have test this in windows and linus, which gave me around 20G of throughput

[@maxhollmann1790]

Not sure if you're still looking for ways to copy a bunch of small files from external media, but if you have access to the physical device and it's mostly empty, one thing you can do is ignore the files and copy the entire filesystem directly. You would then need to mount the resulting image to access the files, but you wouldn't need the drive at that point. This approach should give you the theoretical maximum, aka minimum of portable drive read, connection interface and host drive write speed, without any file creation overhead.

[@TazzSmk]

at this point it would be interesting to see how 25Gbe (or even the dual 25Gbe with SMB multichannel) would perform, Sonnet does make such Thunderbolt adapters

[@matthewdaley7535]

When I get access to them I will test it. One thing that worry about the dual 25Gbit adapters is that Thunderbolt is only 40Gbit and obviously 2 * 25Gbit is 50 Gbit. Maybe the 2nd 25 Gbit port is performance limited.

[@IainBrewSydney]

I’ve used this trick a few times with original thunderbolt, however when I got my new M3 Mac and tried to copy from my i9 Mac, exactly as you found, thunderbolt was going to blow out to 9 hours, 1GB Ethernet completed the copy in 3

[@mattrixx_audio]

That's really interesting... thanks for sharing!

[@The_Roy_Gee]

Did you turn on the jumbo packets?

[@terosma]

Yep, default MTU 1500 will slow ethernet for sure, change MTU to 9000b on both system

[@GrishTech]

In windows, there is robocopy, which allows you to specify a /mt:16 command line switch (number can be specific to the number of threads you want). I’ve compiled hundreds of thousands of file at near line speed. Adjust the thread count as you see fit. I wonder if somebody like that exists for Mac.

[@ErikSutton1]

I opened mine up, removed that noisy fan, stuck an NVME heatsink on it and it's still going strong. No issues and get 9.8gb speeds on my local network👍🏼 the outer aluminum case is a bit hotter, perhaps a metal cooling surface can help with that, but otherwise hapy🤷🏼‍♂️😁

[@darylcheshire1618]

to copy many files faster, try and make it one big file such as a zip or pkzip or similar. Each file copy has an overhead, such as file name write the FAT (or equivalent). One big file is always faster than several hundred small ones, even in a directory tree.

[@matthewdaley7535]

Unfortunately this doesn’t help in my use case. Once the zipped file is on the server I still have to unzip it. This involves reading the single file and writing all the small files, which is the original problem. I know you can unzip the file from the server, but there are many cases where granting users access to the server is a bad idea. Including mine.

[@HawkWeisman]

How many times did you run these tests? The numbers look similar enough that I’d just wonder about noise. You might want to rerun each test a larger number of times and compare the averages.

[@stevehascall4441]

Synchronous writes may be the bottleneck in your situation. One fast way to copy lots of small files very very fast would be to use ZFS SSD RAID with a RAM disk intent log. If that’s too insane you could try a file on a fast SSD for the intent log.

[@LukasReinhart]

You could look into Thunderbolt enclosures made for external GPUs and put a 25G network card in there but it’s unlikely that you can get Mac drivers. Maybe you could find a card that is made to go into the Mac Pro. For getting the full speed you also should check for things like jumbo packets being enabled.

[@matthewdaley7535]

25G thunderbolt enclosures are relatively expensive because there are only a couple of 25Gig cards that work on Mac. I will get to testing 25gig eventually. I've not seen 25 Gig tested on how it goes with lots of small files either. The jumbo frames idea is a good one though.

[@blackjack4494]

Don't use file copy to test connection speed but rather iperf/3

[@matthewdaley7535]

Iperf wont tell me how much extra time it takes to copy small files compared to large ones.

[@blackjack4494]

@@matthewdaley7535 that is true. However you wanted to test throughput in general and obviously small files aren't limited by network speed directly but rather by the OS and CPU. Most modern programs and OS won't send a single small file per network packet but rather send a chunk with multiple or parts of it.

[@nick-dogg]

There’s definitely overhead and a slight increase in latency.

[@LanceBryantGrigg]

This isn't a method of proving speed of networking. This was a method of proving speed of copy. To test speed of networking you need to do something equivalent to iperf and leave it at that.

[@area51xi]

As others mentioned you can’t use disk read writes to test network speeds. Please read up on iperf.

[@s.patrickmarino7289]

You might try changing to a smaller packet size. If you don't want to do that, you could try zipping the files before you send them. Then, it would only be one big file, not lots of smaller ones. In addition, you would get some benefit from compression. The question is, would compressing and decompressing the files take longer than the transfer time. I don't know. One silly question on the Thunderbolt side, are you sure you are running TB4 cables? It would be kind of sad if you were running cables for an older version of USB C. (How far are the cables running?)

[@matthewdaley7535]

These are all good proposals. I used TB4 cables. Compressing before transfer is slower than just copying as, for my main problem, the files are video with 1 file per frame and about 12MB per frame.

[@s.patrickmarino7289]

@@matthewdaley7535, Just a thought but if you compress it, you would only be transferring one file. If you use tar, you don't compress, only bundle. That might reduce the per file overhead without taking the time to do the heavy compression.

[@iham1313]

so even in worst case scenario the thunderbolt 5 connections will be faster then 10gbps ethernet

[@JAFOpty]

Alex Ziskind is getting around 60gbps with TB5, so I dont expect 4 to be full bandwidth either. Did you test with iperf?

[@matthewdaley7535]

I didn't test with iperf because I'm trying to solve a very specific performance problem around transfer of files.

[@darylcheshire1618]

I thought you’d need a crossover cable to join two compters? Perhaps doesn’t apply to macs?

[@LegendaryJim]

nah, that only applies to older ethernet NICs, because the Tx and Rx pairs were fixed. Many modern NICs can flip the Tx and Rx channels in firmware. I don't know much about thunderbolt, but I think every channel is bi-directional.

[@darylcheshire1618]

@ Thanks, I didn’t know that. You’re a legend.

[@Xfacehack]

You probably already made sure, but you didn't encounter any harddrive/ssd/nvme bottleneck while doing this? You made sure that the interfaces are the actual bottleneck?

[@matthewdaley7535]

This is a fair question that I didn't cover in the video. I've used Mac Studios a lot and the SSD in the Studio is pretty fast. Blackmagic disk speed test shows just under 5000M/s write and faster for read in the 512GB model. I also know that copies from Thunderbolt external SSD's can transfer to the internal disk at least 2000MB/s and that's probably the external drive slowing it down to that. Doing SMB multichannel copies across the network with 2 * 10Gbit NICs on the Studio gets 1800MB/s. This tells me the bottleneck probably isn't anywhere else in the Studio.

[@MatthewWalster]

I wonder if Thunderbolt operates with parallel lanes/channels... Can you retry both transfers at the same time, so that it's essentially copying two things at a time rather than sequentially?

[@matthewdaley7535]

Great idea. I'll give this a try.

[@MatthewWalster]

@@matthewdaley7535 remember as well that your disk may be the bottleneck at some point too.

[@edc1569]

Played with this a while back, think you could get 20G between a windows machine and a linux server.

[@bujin5455]

My guess is Apple's drivers and IP stack are not optimized for Thunderbolt, and are running a virtual 10GiE interface on top of the Thunderbolt port. Of course, that's just pure speculation on my part. There are a lot of weaknesses with the testing performed. For instance, how fast can the mac copy those files on an I/O subsystem level? What if you created two ram disks within the same machine, and copied the test sets between them? This would be your absolute upper limit. I've performed these test, and they aren't as fast as you'd think they'd be. No where close to Apple's advertised memory bandwidth statements.

[@matthewdaley7535]

A virtual capped 10Gbit interface fits with the data. I don't think the issue is with the rest of the system as USB (Thunderbolt) drive copies are much faster. The RAM copy is a great idea. Can you tell me the software you used to create the RAM disks please?

[@bujin5455]

@@matthewdaley7535 You just do it from the command line. Google 'mac cli ram disk'.

[@mrkdosmil2879]

I think Jeff Geerling have talked about this on of his videos. Thunderbolt networking on Macs have an issue.

[@hpsfresh]

Just use external thunderbolt enclosure with nvme disk installed or external thunderbolt network card

[@matthewdaley7535]

Multiple computers cant share a single thunderbolt enclosure at the same time but using SMB over IP means the storage can be shared. I'm interested in this testing because I want better performance from my network storage.

[@MRMsysdotnet]

JUMBO FRAMES

[@frederickstaats]

Don't confuse the physical layer data rate of 10 Gbit/sec per lane for Thunderbolt 4 and the protocol data rate over those lanes. Thunderbolt transports several different packetized protocols over the same physical layer which have different priorities. Thunderbolt 4 mandates four bidirectional lanes with approximately 10 Gbit/sec of independent bandwidth. Video gets transported with embedded DisplayPort protocol with highest priority (as its traffic must arrive in time for your display to work correctly). Other protocols include embedded include USB and PCIe signaling which are bursty and use what bandwidth is remains. Thunderbolt 4 implements the embedded PCIe 3.0 protocol which is limited to 32 Gbits/sec and can't use the remaining 8 Gbits/sec which is reserved for video and other protocol overhead. Note: in most computers a physical PCIe 3.0 data lane on the mother board is attached to the thunderbolt controllers PCIe. A single data streams maxes out at 32 Gbit/sec before encoding overhead, 8/10 removes 20%, 64/66 3%, or 128/130 1.5% depending not the exact protocol and context. Network adapters attached to Thunderbolt can come in two types USB controllers that connect to the embedded USB protocol, and the faster lower overhead PCIe controllers that connect to the PCIe protocol. "Thunderbolt Networking" emulates a PCIe controller network card without physically having real network controllers. Thus the theoretical maximum with Thunderbolt Networking is 32 Gbits/sec and the practical limit is less than that with both the physical protocol overhead, the logical protocol overhead, and any other traffic. Also effective bandwidth is highly dependent on the network packet size... I always configure Thunderbolt networking with Jumbo (9000 byte) packets rather than the default (1500 byte) packets which for me is a 30-40% performance gain over the default. Connecting my MacBook Pro M1 Max -> CalDigit TS4 -> MacBook Pro M2 Max I constantly get 14.7 Gbits/sec iPerf bandwidth with my high resolution wide screen monitor also attached to the docking station. When I turn off the monitor it jumps to 20.4 Gbits/sec iPerf bandwidth. I see these as respectable numbers given the overheads of the protocols and other shared hardware on the docking station. (Note: I get the about the same effective speed with or without the docking station in between the computers). With Thunderbolt 5 in the default symmetric configuration the embedded PCIe 4.0 and is twice the speed at 64.0 Gbits/sec. In addition some of the overheads are fixed in size or optimized so I would expect (but have not yet been able to test) that Thunderbolt Networking properly configured would be a little more that 2x faster.

[@seanpower4670]

what about fiber ?

[@matthewdaley7535]

I use 10Gbit fibre and copper ethernet a lot and there is not detectable difference between them in performance. If there was, we would use the better one for sure.

[@jfkastner]

Powermanagement can also cause this - I've seen Servers where the CPU was almost Idle during Data move/copy hence the Clock speeds were lowered down - which then affected Network performance

[@area51xi]

You will never go faster than your hard drive speed. It has nothing to do with the thunderbolt. You cannot reach 40 gigabits because your drives can’t go that fast.

[@acewright-n4q]

Of course they are the same they are both using ethernet which means the same NIC just because they are going over the a different connector doesn't mean you will get the benefit of it processing packets faster you still are using the NIC to make the packets and read the packets.

[@matthewdaley7535]

This doesn't make sense. Ethernet can run at 1, 10 ,25 ,40 or 100 Gigabit (or more). Why would Apple restrict the Thunderbolt network adapter to 10Gbit?

[@sir_anders]

its not but thanks for the clickbait

[@matthewdaley7535]

My tests show evidence that it is. What makes you think IP over Thunderbolt is quicker than 10 Gbit Ethernet?