I built a HIGH PERFORMANCE hardware Firewall! - Sophos SG 330 Rev2

I've been chasing down a performance problem with my virtual #pfSense firewall for about 6 months now with inter-VLAN routing in our #homelab. Join me as I go through the process, start to finish, troubleshooting, and ultimately somewhat fixing the issue using a modified #Sophos SG330 Rev2 firewall as my new platform for pfSense! Some mistakes were made.
In this video, we talk you along as we trouble network performance issues, choose a hardware firewall, test its performance, and upgrade it to make it (kinda) faster!
*PRODUCT LINKS*
Sophos SG330 Rev2: ebay.us/UmQEEW
Intel i7-6700K: ebay.us/Qd4crB
*GET SOCIAL AND MORE WITH US HERE!*
Join our Discord server! It's a great way to chat with us!
🎮 / discord
Please consider subscribing! Follow us:
📸 / 2guystek
🐦 / 2guystek
💻 / 2guystek
Visit our store!
🏬 www.amazon.com/shop/influence...
If you would like to support us in other ways, please become a Patreon 😁 / 2guystek
*TIMESTAMPS*
0:00 The problem
1:12 The Network Diagram!
2:17 IPERF, the testing tool of choice
2:43 Initial test results
4:01 Enter the Sophos SG330 Rev2
7:09 Initial attempt to install pfSense on Sophos SG330 Rev2
8:17 Testing inter-VLAN routing on the new hardware
9:17 Proof pfSense is single threaded
10:15 Why we don't use a layer 3 switch for inter-VLAN routing
11:56 Upgrading the CPU on the Sophos SG330 Rev2
13:35 Testing throughput with upgraded Sophos SG330 Rev2
14:25 Using LCDProc to get the LCD working on Sophos SG330 Rev2
15:05 Closing! Thanks for watching!

[@2GuysTek]

**PRODUCT LINKS** Sophos SG330 Rev2: ebay.us/UmQEEW Intel i7-6700K: ebay.us/Qd4crB

[@johngalea2285]

I enjoyed the article ... well done. I too moved back to a physical box for my firewall from virtual. My issue was different, I was getting inconsistent DSL initialization which was inexplicably solved with physical.

[@ekon12]

Really good vid! I was just doing some research to implement a series of VLANs on my home/homelab network and after surfing reddit, I basically came to the same conclusion you did. You need hardware for a proper firewall/router!

[@2GuysTek]

That's sadly the truth! We're big proponents of virtualization, and having the FW be virtual gives great flexibility. But in our case hurt our performance. Thanks for the comment!

[@IanGSully]

Well… I just switched to pfSense the other day on my network… originally I had just a regular Netgear router, but I figured I would switch to something different, and something virtual on one of my server… and pfSense I thought was the most bad a$$ firewall I have ever used! Now… I haven’t had any issues with the virtual firewall yet… because I am still learning about it, and I am loving it so far!

[@antonybellerive-cossette9236]

iPerf by default is also single threaded. This might explain why you don't see much of a difference since in this case your "bottleneck" may be the CPU of your iPerf client/server. iPerf can use multiple threads to simultaneously send and receive data. Use the -P switch followed by the number of threads to use. iperf -c -P 4

[@KALTBLUTWOLF1]

That tast would only be meaingful if the real life workload that he tried to address was also multi threaded, which it wasn't in his case of copying files. But maybe tools like robocopy with its /mt switch might help in this case

[@antonybellerive-cossette9236]

It all depends on the sharing protocol used. NFS is multithreaded while SMB isn't (unless SMB multichannel is used). Other thing to keep in mind is pfSense is limited by its kernel-based packet processing which out of the box isn't made for 10Gbps+. People who want more than 10Gbps should look at TNSR instead, which leverage Vector Packet Processing (VPP) and deliver substantially greater packet-processing performance and throughput.

[@2GuysTek]

If only TNSR was _also_ a firewall...

[@Traumatree]

I am running a custom built pfSense with a i7 8700k + 32GB of RAM with a Chelsio adapter with 2 x 10GE SFP+ port and I can route at 10Gbps both direction no problem, while the CPU isn't even flinching pass 10% much. I would say with the many labs experiment I did in the past with pfSense on many fanless box, VMs under ESXi, Proxmox, XCP-ng, Virtual Box and Hyper-v, that the drivers used for the NICs will greatly influence the throughput. For example, under Vmware, anything intel E1000 setup for drivers will slow down to a crawl whatever performance benchmark you will do. pfSense kernel also use 1 thread per network session, so CPU single core performance will also be an issue. Sub 3Ghz single core performance will NOT be able to sustain bi-directional 10Gbps. Another thing to consider if you ware using SPF+ Transceiver is their compatibility. If they are not, they will over heat, and if you are using a RJ45_to_SPF+, it will impact your performance too - or reset your switch and or port or both - happened on a Ubiqui EdgeMax 48-port a few years ago.

[@kittysreview9055]

Another solid video! Thank you!!

[@hughw.]

Excellence as always.

[@markbeck2236]

Another reason why a dedicated hardware solution is nice is that your VPN connection can get faster. I moved my pfsense installation from a VM to being the only OS installed on a particular box and I saw my OpenVPN throughput increase from ~50/~50 Mbps to 157 / 193 Mbps on the same exact hardware.

[@ig00g1e]

I use Nftables with Fastpath over Firewalld with dual port Mellanox 100G Connext-6 NICS on a PCI5 Threadripper Pro . We do east/west firewall and inter-vlan routing for 20 Vlans, 300 nodes..

[@TheKeirsunishi]

If you started another transfer over the network whilst doing the iperf test, perhaps another iperf test between two other vlans, would pfsense be able to use one of the other cores?

[@Froggie92]

question for the proof pfsense is CPU bound: i can see thats its single threaded, but would you get gains from faster memory? i just switched my virtualized pfsense from an i5-4570 to an i7-8700 and saw clonezilla nfs transfer speed increase from around 6gbs/min to 20gbs/min i dont know if i can pin that to faster cpu clock tho: it was two back to back same tests, just the second time was on an i7-8700 with ddr4 on nvme i dunno how much the core clocks influenced it tho, i would guess memory had a hand in that increase, but also im curious on dual channel vs quad channel but yeah optiplex 3430 for $200 on ebay, plus a 10g card, im curious to see what one of those Erying i9-11900H's could do

[@apigoterry]

try using -P 5 on iperf to get higher throughput. mine, using Intel(R) Xeon(R) E-2124 CPU @ 3.30GHz, i can get 6-7gbits/s speed

[@memack101]

Interesting video.

[@t4thfavor1212]

Add -P 4 or -P 8 and you'll get closer to 10Gbps. I suspect the vm could have gotten closer as well, but probably not as high as the bare metal install.

[@sonicalstudios]

check out the Dell r210, perfect pfsence box...just needs a 10GiB NIC

[@alk_dl]

if you have used -P parameter with parallel connections you would have maxed 10g network...even with vm

[@shetuamin]

Hi. Can you make a detail post about your grafana dashboard setup? I am very much interested about this. Thanks.

[@2GuysTek]

You're not the first to ask! We'll see what we can do!

[@JasonsLabVideos]

GOD DAMNNNN!!! you rocked this video !! Nice man !! How about a Supermicro Box with the Xeon-d in it and dual 10gig sfp+ ports ? about the same price as the Sophos box ?

[@2GuysTek]

That’s a good question! I really do love the ports up front that the Sophos provides.

[@JasonsLabVideos]

@@2GuysTek Exactly why i bought my SuperMicro unit ! ports up front mate with the ports on the switch nicer too !! You sure rocked this video man !

[@appleseed_316]

i know this is an older video but i found this as i am looking to do this too. Were you tagging all 4 vlans on the same physical interface, If so you might get better throughout moving each vlan to a seperate physical interface

[@2GuysTek]

The connection to the downstream switching is a single trunk with multiple connections, yes. I like your thought process, and it might be something to test later on! Thanks for the comment!

[@bluesteelbass]

Heatsink for NIC controller(s) in the back looks smallish for so many ports, as well as being 10G... I think there is a lot of CPU utilization for routing traffic, versus just using the CPU for rules and inspection on that motherboard. If that is a stabdard pci-e slot, just with a strange header orientation, I wonder if putting a server NIC in the slot would see any speed increase in your iPerf tests. All about the bus speed for these things in the end. If you can do a stable overclock, there should be a notable difference.

[@2GuysTek]

You bring up some really good questions about the thermals of the Intel NICs in the box. Next time we do some maintenance we'll throw some better heatsinks on it! Thanks for the suggestion and watching!

[@WidowMakerSilent]

holy thermal paste bud!

[@user-qo6kf6om3q]

Seems like part of the reason the 6700k doesn't get you much increase in performance might be the increased TDP since the box wasn't designed for the 91W part. I would imagine there would be at least cooling limitations, if not straight up power limitations. Would be interesting to see what an i7-6700 (non-K) would be capable of, though my guess would be that it would be similar based on the core clocks (10% > i5-6500).

[@2GuysTek]

This is possible. I’m actually going to make some minor cooling changes to the unit here in the near future, I’ll definitely check again!

[@mhnieuwenhuis3583]

Noctua NH-L9i-17xx Has a fitting heatsink with heatpipes. The fan won’t fit without cutting a hole in the 1u enclosure though.

[@Raidflex]

So I was looking at picking up one of these SG 330 Rev 2's and they all seem to be $800 and up. Also most of the ones being sold are Rev 1, which do not have SFP+. Is there another place besides eBay to find used Sophos hardware?

[@2GuysTek]

Something seems to be going on in the hardware aftermarket lately. There are a lot of things that are just getting out of control price-wise. If you can’t find an affordable SG with SFP+, it might be time to consider building out a small form PC and throw in an SFP+ card.

[@mhnieuwenhuis3583]

Sophos sg series is end of sales June 2023, updates ending 2025. Prices should be going down. If I had more gear to attach I might try a CPAC-4-10F in the flexiport.

[@andreiyurevich6336]

Have you tried to use this sandwich from Sophos-branded Portwell PC and PFsense for a real case, like SMB?

[@2GuysTek]

I run a small business off my hardware, and it performance perfectly.

[@Cyhawkx]

Yes, this is a year old, but you skipped an important bit of information. What Hypervisor and what were the settings? Incorrectly configured VM settings will cause this, as well as what else is running on the server.

[@2GuysTek]

I believe I touch on it in the video, but everything we run here is VMware ESXi. And completely agree that how you configure the VM and the host's workloads affect performance, unfortunately misconfiguration was the source of the bottlenecks. Thanks for the comment!

[@DigiDoc101]

I have been eying a Sophos SG450 Rev. 2 comes with SFP+ card, dual ssd, dual power. It is offered for $450, expensive for what offers? does this eliminate cpu bottlenecks for homelab use? I am on a similar boat. Great video!

[@2GuysTek]

It did largely for me. As you saw in the video, we're getting great throughput on the 10gig connections, and removing the issue of service interruption when doing updates to our virtual infrastructure was a big win. That SG450 sounds like a beast! It's definitely a step above the SG330 we used! Give it a shot if you've got the money and the interest! I'm sure it'll serve you well for a long time! Best of luck!

[@DigiDoc101]

Do these run loud? Compared to desktop fans or synology drive.

[@2GuysTek]

I can’t speak to your model, but the SG330 isn’t any louder than a Synology NAS.

[@simonong5839]

Could you recompile of sense and optimised it?

[@2GuysTek]

Unfortunately that's not an option. Thanks for the question!

[@hasanmujeeb8922]

Guys I’m wondering if I can install pfsense on Sophos XGS series if anyone has tried it pls lemme know

[@2GuysTek]

I have been _trying_ to find any information I can about the internal hardware on the XGS line, and have not found a definitive answer. If the system is running on x86 hardware like ours is, it's very likely you'll be able to easily install pfSense on one. The risk is the case where the XGS is running on ARM instead of x86, and in that situation you'd be out of luck. Best of luck and let us know if you have success!

[@hasanmujeeb8922]

@@2GuysTek its really hard to find information about sophos hardware, I thought it has a main x86 processor and secondary arm processor but if it’s only built arm I think I’m just gonna stick with XG series

[@Prime0pt]

Why is avoiding use of ACLs on a switch is best practice? Whos best practice is it?

[@2GuysTek]

Great question! Typically Layer-3 switches have specially designed chips in them called ASICs that are designed to switch packets between interfaces incredibly fast. ASICs do one thing really well, but really only that thing really well. Processing and filtering ACLs requires general computing because rules needs to be calculated per-packet, which is handled by the CPU inside the switch. The CPU typically has enough overhead for a few ACLs here and there, but the more ACLs you have that need to be processed, the greater chance your switch's CPU won't be able to keep up. At a minimum you'll experience performance issues, at a maximum (and I've seen it with certain Cisco L3 switches) they'll crash. In network design theory, it's widely held that if you need to filter packets between networks (inter-vlan, edge, or whatever) you should use a firewall. Unlike a network switch, a firewall's job is to apply rules and filter packets accordingly. This is what a firewall is designed to do, and it does it very well. There are caveats here worth mentioning. 1) If you only have a few ACLs you'll be fine running them on your L3 switch. 2) Technology is always getting faster, it's entirely possible there are L3 switches out there that can handle a massive amount of ACLs and not break a sweat. Hope this helps!

[@Prime0pt]

@@2GuysTek then you should know then unlike firewall that analyze full packet switch ACL read only IP headers on packets . So it need much less compute power. And problem may starts on cisco only when you use extended ACLs that analyze also ports information from packets. So you suppose to use proper tools for each case understanding what you doing and why you doing that no saying this is best practice and this is not best practice.

[@TVJAY]

Can you make a video on Grafana?

[@2GuysTek]

What would you like to see specifically? Let us know and thanks for watching!

[@TVJAY]

@@2GuysTek I would love to see how to set it up (from start to finish) AND get pfsense data into it.

[@Jedi3699]

@@2GuysTek I would like to see how to set it up as well, start to finish

[@psycl0ptic]

don't use ufs...you have zfs option right there.

[@northblue8216]

Hmm, don't you need to re-install PFsense if you change the processor? I think so.

[@2GuysTek]

Not at all. The pfSense dashboard correctly recognizes the CPU and shows its features. Potentially if you were to move to an entirely different generation of CPU that may be the case, however, I still suspect it wouldn't be.

[@northblue8216]

@@2GuysTek Ok, thank you.

[@Darkk6969]

No you don't need to reinstall as FreeBSD will handle the changes just fine. I even moved the hard drive from my old i7 to Ryzen 9 and it booted up just fine. Only time you need to reinstall if you're switching from X86 to ARM which require a completely different OS.

[@KombiGnome]

Wait, you wear glasses?

[@2GuysTek]

Don't I look more respectable?!