Why Are SMB File Transfers Slow Over A VPN?

[@databeestje]

The other thing that people forget is that the upload speed of your own connection limits to how fast you can copy files to the main office . The default reaction to support is "But I have a 600mbit connection", yes, download to you, and a 60mbit upload that realistically limits it to 6MB/s. It's a matter of perspective.

[@RaidOwl]

Really informative stuff and a question I've always had. Thanks, Tom!

[@philippemiller4740]

I love your videos mister owl

[@RazrBurn]

I’ve been trying to explain this to people who insist on using SMB to send data to the other side of the world over a VPN. Thank you for the video. When they ask why we shouldn’t use SMB, I’m just going to send them a link to this video.

[@gamebrigada2]

Microsoft is getting to fixing it. Crossing my fingers that SMB over QUIC will be available in non-Azure edition server sku's soon.

[@RazrBurn]

@@gamebrigada2 MS has been trying to fix it for several years now. They have improved it but it’s still awful over high latency connections. I feel they are going to have a hard time fixing it because of the need for backwards compatibility. As well as both ends needing to run the newer version. I think they would be better off developing a new open protocol and letting SMB (very) slowly fade away.

[@gamebrigada2]

@@RazrBurn SMB over QUIC is SMB in name only. Yes you can open a share on a server in Windows seamlessly, but that's pretty much where the similarities end.

[@RazrBurn]

@@gamebrigada2 Sounds promising. I'll be intereste to see how as it's possibly adopted.

[@BradleyHerbst]

I really appreciated this video but it would happen even better if you included some alternative ways to transfer files besides SMB that wouldn't be as affected by latency.

[@Ninjay2]

Agreed or any SMB tweaking to help reduce the ill effect.

[@FreddyF1977]

sFTP will be quick.

[@James_Knott]

You might try SSHFS or FTPS. These also have the advantage of being encrypted, so you don't even need a VPN.

[@philippemiller4740]

To my knowledge sftp encrypts only the traver transfers but the credentials are still unencrypted.

[@James_Knott]

@@philippemiller4740 I typed SFTP by mistake. I meant FTPS and corrected it. Regardless there are a few protocols, based on SSH or TLS, that could do the job and they have the advantage of being designed to work over TCP. Back when SMB was first created, it was designed to work on bare 802.2, without any concept of being routed beyond the local LAN.

[@jdijulio1]

I think the script of a great follow-up video would start something like this, "Tom from Lawrence Systems here, and in today's video we're going to do a follow-up to our last video where we explained why SMB is a poor choice to use over VPN and why X is much better suited for these use cases"

[@LAWRENCESYSTEMS]

I was going to put it in this video but that get's complicated so it will be a second video.

[@SureshotCyclonus]

I agree. Sometimes you are just stuck using SMB at the application level. So you either deal with the limitations or you try to make a bandaid solution such as having two truenas hosts replicate access the VPN. This would allow your local application to use SMB. Replicate over the VPN quicker and have the data at the other end. Not the greatest solution but it is better than nothing.

[@jdijulio1]

@@LAWRENCESYSTEMS awesome! Very much looking forward to this follow-up!

[@MrMcp76]

NFS, iSCSI, SMB, SFTP. Am I missing any? I use iSCSI to connect my SAN to my ESXi host I use NFS to create shares between Linux VMs. I use SMB to create shares on my Linux server to be accessed by Wondows clients. SFTP to backup data to my backup server. VPN is used to securely connect to my servers, and rarely used for file transfers. I think this is the best setup, but looking forward for Tom's next video to see his thoughts on it.

[@Book_Bird]

@@LAWRENCESYSTEMS Can't wait

[@AaronSchmidt52]

I learned a thing today, thanks for sharing Tom! I've had this issue myself and always woundered why the slow file transferes.

[@jedilee]

Omg, I could used this video for soooooooooo many meetings!

[@bwatk15]

I think DFSR was designed with this in mine. It is the back end replication for Active Directory Domain Services SYSVOL. The problem is people misusing it for what it wasn't designed for (it is a lockless fileshare) . For example it can be set up for one way syncing Read-Write share to a Read share or two way which is where things go wrong when badly designed setups cause issues.

[@iggienator]

Thank you for explaining it so clearly

[@jfkastner]

Fragmentation and packet size issues are the main cause. MTU, payload size your VPN protocol, SMB blocksize, Window sizes/scaling etc all need to be "fine tuned" for your specific needs. Sadly though most SMB servers can not change those parameters "on the fly" based on destination, so you either have good VPN performance OR LAN performance (which most users/admins prefer) - but not both. FYI I had spent 5 days on an OpenVPN connection with Wireshark mostly and got about 75% of the WAN speed through the VPN with SMB. Your router and ISP do not care what traffic they carry!

[@game.streamer]

Any thoughts on fine tuning a site to site vpn connectivity through fortigate firewall ? The internet down is 400 mbps and up 40 mbps. The file transfer speed I is 1 mbps. although the firewall appliances have a vpn throughput upto 1 Gbps.

[@custard131]

nice video but now you got me interested in seeing how other file sharing protocols perform in these conditions

[@udirt]

especially for OpenVPN there's some buffer parameters you really, really want test adjusting. send / receive buffers are disputed but worked very well where I tried them. but - if the link speeds are just too low or for badly optimized applications, it's often better to give access to a terminal server where people then work.

[@James_Knott]

IIRC, SMB was also noisy, at least on the local LAN. Several years ago, during the first time I worked at IBM, I was reading some internal documentation about adapting it to work over IP. It had a lot of issues that made it a generally bad protocol. Originally, it was strictly a local LAN protocol, before IP made routing possible.

[@hotstovejer]

One of the places I've worked at used global protect for VPN. When I would go to pull up Active Directory Users and Computers over the VPN, it would take 20 minutes. No, I'm not making it up. They transfer that data over SMB1. Painful AF. I ended up just writing some powershell to pull what I needed from AD.

[@Darkk6969]

Yep. One of the reasons why I have a jump box on-site that I can RDP into.

[@Saturn2888]

What should I use instead of SMB over VPN for Windows machines?

[@ajama1335]

So what's the alternative to SMB when using VPN?

[@BeamDeam]

Webdav and NFS

[@Ultrajamz]

How about NFS? Or what is the next alternative that is friendly with many os types?

[@Magickmaster3]

Isn't NFS security based on IP addresses? I'd like to have user-based auth. It could work well with Wireguard though

[@eugenesmirnov252]

@@Magickmaster3 no. NFS v4 has complicated authorization mechanism.

[@peterdee1900]

Great video!! I definitely had the same question.

[@oleksandrlytvyn532]

Hello, which tool was used to add additional latency? Would be interested to see how it's done or know via which tool it was done

[@LAWRENCESYSTEMS]

pfsense limiters docs.netgate.com/pfsense/en/latest/trafficshaper/limiters.html

[@TGUK9]

Has anyone got around to testing but hopefully SMB over QUIC (Win11/Server 2022) on a VPN?

[@willblanton3120]

To clarify this, it’s not related directly to SMB, but rather the TCP transport layer. Because of the error/congestion correction in TCP, there is something called a window. This means that the client and server agree that X number of bytes will be sent before an the sender waits for the acknowledgment that all packets were received. There is also a feature that’s called window scaling which changes the size of that window based on the connection and can drastically affect your transfer speeds. What’s happening here is that the higher latency is forcing the TCP window scaling to decrease, making the window smaller and therefore throttling the transfer speeds. So even if you have a tunnel with a full gig connection on both ends, this additional latency could mean your throughput is actually limited to something similar to a 100Mbps connection. A couple work arounds are to play with your operating system to find if there are methods that could affect the window scaling to allow for high speed but high latency connections, but I’ve found this to be pretty tricky, especially on Windows machines. The other option is to run multiple streams whenever possible, because each stream will have its own window. But this obviously relies on the application implemented supporting that, which is another issue entirely.

[@James_Knott]

Or move to something like SSHFS or FTPS, both of which are designed to work over TCP.

[@Lue30499]

@Will Blanton Yes and no. For example if you test with FTP, which is also TCP, you will not see this symptom as bad as it is with SMB. The SMB protocol is very noisy with every block sent needing to be acknowledged at the smb layer as well. (along with metadata requests like file locks and for file permission checks) TCP may exacerbate this with TCP also doing its sequencing and what you mentioned (number of in-flight datagrams before requiring ack), as each block will be made of multiple TCP datagrams. But as far as I know, changing TCP window size and scaling doesn't fix this issue (unless hitting 100s of ms) to where its close to line speed of the VPN/tunnel. It isn't the core issue.

[@berndeckenfels]

Thats Not correct, SMB makes it even worse than tcp (tcp can have fairly large windows)

[@berndeckenfels]

@@James_Knott ssh/sftp also uses application level windows. I think of traditional protocols only WebDAV and ftp/s are purely relying on tcp. However in any case I would use a replication tool (async transfer)

[@karakara980]

Thank you Will for your additional explanation, that’s make sense.

[@aaronjden]

I had no idea. Thanks for the useful video!

[@brycez7352]

Thank you so much for this video..

[@inadaizz]

Seriously thank you.

[@OzSigns]

Hi Tom, love your stuff. Would you have any detailed tutorial on how to access shares over the internet? I already have vpn , tunnel for services etc.

[@LucidEnemy]

my biggest question here is what are good alternatives that the simple windows end user can use to use another protocol or solution, my clients typically end up with unraid+tailscale and mounted shares on the machines 98% of the time there local so latency is low however the 2% they use the SMB share over the VPN however where talking about 1-5mb word files so it doesnt matter as much as the backup that runs only during the day so as to have them local I would like to switch that to at night

[@LAWRENCESYSTEMS]

I am not aware of anything "Simple" that works other than using tools that synchronize files such as Syncthing.

[@dheijnemans]

My biggest question is, where is the punctuation?

[@LAWRENCESYSTEMS]

​@@dheijnemans ¯\_(ツ)_/¯ English is hard

[@justinatwell8187]

I connect to a VPN hosted inside my Asus router. I noticed most of my SMB transfers over this VPN max out at a certain speed. Is this speed influenced by the CPU of my router?

[@jas9450]

That thumbnail is hilarious 🤣

[@f-s-r]

Very interesting. What do you use to mount SMB file shares? I tried to do that some time ago, and the result was awful, like if the file transfer was never going to finish. On the office i use rsync with the SSH-FTP protocol in windows with cygwin to update quite a lot of files over VPN. It seems to work very well.

[@Sanosukejp]

Great video. Many ty

[@StarATL]

Great explanation, but you left us wanting to know more…..like what we should be doing to address this. Is WebDAV or something else the way to go?

[@LAWRENCESYSTEMS]

Webdav is an option.

[@8xpdhpckkg]

Follow up Question then: do other protocols have similar penalties? What protocol would you recommend for large point-to-point file transfers over WAN?

[@LAWRENCESYSTEMS]

Streaming protocols work better.

[@8xpdhpckkg]

@@LAWRENCESYSTEMS thank you :D I don't know if you've done a video about this topic already, but it would probably make for a good one

[@420isMySweetHoney]

@@LAWRENCESYSTEMS can you recommend some?

[@ramrod2k]

would like to know why. Sth. to do with UDP vs TCP?

[@mitch7918]

I’m going to be sending this video a lot

[@buldozzer3456]

I had SMB Multiplexing running over l2tp in a LAB. It scaled not that bad.

[@Daniel1987H]

Always wondered, why smb over OpenVPN sucks balls, but over wireguard it's basically fine.

[@tacioandrade]

And what protocol do you recommend for this type of transfer? I'm asking this because I use sshfs, but I'm looking for more performant ways to solve the problem.

[@LAWRENCESYSTEMS]

WEDDAV as a protocol would work better, another option would be Syncthing.

[@tacioandrade]

@@LAWRENCESYSTEMS Very good to know! In this case I will see a way to integrate webdav in Proxmox for example, when I need to migrate VMs between different datacenters for example and do the tests. Thanks

[@im.thatoneguy]

WebDAV has the obnoxious limitation though of no large files on windows without lots of adaptation.

[@Ultrajamz]

So seems then SMB also isn’t great for wifi then due to latency from that?

[@coveringgrape5251]

I get 50-80 mbps smb over wifi so I find it fine. For comparison I get about 30-50 mbps with ssh based transfers (SCP or SFTP)

[@Ultrajamz]

@@coveringgrape5251 MBps or Mbps

[@tundrastreaming]

Is there an alternative protocol for Windows to use? I know NFS exists for windows, but it doesn't have a UI and isn't really suited for normal people who don't want to mount via command line

[@CoreyThompson73]

If you want to optimize SMB over a VPN, make sure all your node types are "p-nodes", so they don't waste efforts on broadcast traffic, like b-, h-, and m- type node use.

[@Lue30499]

Thats not the SMB protocol, thats NetBIOS. Altho they do work in tandum it is separate.

[@2008mjb]

Where was this when I was having to repeatedly explain this at work starting 3 years ago.

[@Mitchell7790]

Hi Tom, have you tested SMB over QUIC at all? This is what Microsoft introduced in Server 2022 for accessing SMB shares via services such Azure Files. As far as I know there hasn’t been any other improvements in SMB over WAN other than this.

[@im.thatoneguy]

Problem is you have to have Windows Server Enterprise Azure Edition

[@am3777]

How were you able to add latency in your test?

[@LAWRENCESYSTEMS]

Using limiters in pfsense

[@studiociodo]

The problem is clear, and the solution? Witch protocol YOU raccomand? WebDAV, FTP, NFS. Every of these protocol can easily implement in Windows. If security is demanded throw VPN how can you obtain usability on typical file server office situation?

[@jamesmyers777]

Awesome video thanks

[@drooplug]

Given the drastic increase in work from home vpn usage and desktop virtualization, they would get this sorted.

[@Lann91]

Ok so what do you suggest to use instead? I want to share a network drive for work related stuff, really need the speed.

[@LAWRENCESYSTEMS]

Look for a web based file sharing tool

[@bcredeur97]

What’s an alternative that is just as or almost as easy for your average joe to use to move his files around?

[@LAWRENCESYSTEMS]

I am not aware of anything "Simple" that works other than using tools that synchronize files such as Syncthing, Dropbox, or OneDrive.

[@chmoduk]

ftp. Explorer can work with it just fine, or an ftp client.

[@someguy0523]

Thanks, as a Windows guy tried getting around the SMB limitation by creating an iscsi lun over the vpn (gigabit internet on both ends). No problem creating the iscsi lun, but when transferring a file to the new lun I'm getting the same speed as smb. Topping out at 4MB/s, latency between sites is in the 16ms range. Thoughts?

[@LAWRENCESYSTEMS]

iSCSI was not designed to work over VPN

[@someguy0523]

@@LAWRENCESYSTEMS I know, I just wanted to see if I could bypass the SMB limitation ...basically I am looking to do offsite backups to a Synology NAS at a family member's house

[@FinderX]

SMB is even worst in WIFI, only 2 mbps (megabit per sec) on 2GHZ, and a little faster on 5GHZ, but no much...

[@dezejongeman]

awsome had I known this earlier. would NFS perform any better?

[@LAWRENCESYSTEMS]

Nope, it suffers from the same issues.

[@pnowikow]

That explains why

[@johnrambo6549]

Can you do a video on how to prevent ARP poisoning on Pfsense

[@HisLoveArmy]

So is there a protocol that works better than SMB but can also work on Windows?

[@LAWRENCESYSTEMS]

Nope, nothing built into Windows.

[@icedutah]

How did you add the latency to the connection?

[@LAWRENCESYSTEMS]

Using advanced firewall rules in pfsense.

[@icedutah]

@@LAWRENCESYSTEMS Cool, I just tested and it's working. For testing latency what is the best to use? Prio, fifo, Codel, Taildrop, pie, red or gred?

[@zax123]

Hi guys, I was wondering what you'd suggest to transfer files between two Windows machines on opposite ends of a VPN. Instead of SMB that is... Obviously SMB is the most convenient using File Explorer, but perhaps there's something almost as convenient that doesn't have the SMB latency-effect?

[@UntouchedWagons]

Maybe SFTP or webdav?

[@cyriltemerev71]

I wonder that too. Since some anniversary update to w10 its got support for NFS, thou it has its own challenges. Im afraid that one would have to look into a specific details of the use case and go from there. Im afraid that for a lot of cases the convenience of SMB on windows will trump the speed issues. Thou again - it depends ;)

[@pepeshopping]

Just increase your TCP Window size.

[@FreddyF1977]

... the solution I came up with was setting up an sFTP server.

[@dheijnemans]

Robocopy with the /MT flag set. Perfectly capable of saturating your bandwidth if your disks can handle the thrashing.

[@Krakkel]

Now do a tutorial on NFS on Windows please

[@James_Knott]

Ouch! Several years ago, I had NFS set up between OS/2 and Linux.

[@mjj2u2]

Well Done!

[@PatrykPabo]

what about NFS ?

[@LAWRENCESYSTEMS]

Still really slow over vpn. I am not aware of anything "Simple" that works other than using tools that synchronize files such as Syncthing.

[@biomerl]

I hate to be that guy, but the title is a bit off- Why are* smb file transfers slow over a vpn I had to google it - is is for singular, are is for plural. "transfers" is plural so you are supposed to use "are" instead of "is"

[@LAWRENCESYSTEMS]

Oops, you are correct!

[@user-xv1vm5xc1f]

Perfect timing

[@june5646]

Thats why ya use WebDAV !

[@MrMcp76]

Any traffic that has to traverse the firewall is going to have a bad day with latency, especially SMB and VPN connections. Keep it on the switch as much as you can. VPNs are for security, really, not speed. If you want speed setup a virtual desktop you can remote into. Great video!

[@cpuuk]

Come back kermit, all is forgiven ;-)

[@markarca6360]

One answer: LATENCY.

[@mabmachine]

Nice demo. Its physics, if you double the latency you essentially halve the realized bandwidth.

[@drtvkilla]

Because it needs to zap up to space and back

[@Ultrajamz]

Space = Microsoft servers heh

[@pepeshopping]

Just increase your TCP Window size and throughput will be better.

[@Lue30499]

Yes and no. For example if you test with FTP, which is also TCP, you will not see this symptom as bad as it is with SMB. The SMB protocol is very noisy with every block sent needing to be acknowledged at the smb layer as well. (along with metadata requests like file locks and for file permission checks) TCP may exacerbate this with TCP also doing its sequencing and what you mentioned (number of in-flight datagrams before requiring ack), as each block will be made of multiple TCP datagrams. But as far as I know, changing TCP window size and scaling doesn't fix this issue (unless hitting 100s of ms) to where its close to line speed of the VPN/tunnel. It isn't the core issue.

[@seeingblind2]

because you failed to connect the flim flam to the torbotrizer!!! This is the most basic of things!!!

[@eugenesmirnov252]

mtu. mss. router. fw-rules. And how we were switched to rsync from smb?