OPNsense Proxmox High Availability Homelab Network Cluster (Maximize Uptime!)

[@LithiumSolar]

19:30 Looks like you could use a PoE-powered switch there, would solve the power problem 21:01 How do you have so many bad punches? Are you using solid conductor cable and a proper punch tool??? The keystone route is a great idea, just costs $$$.

[@DigitalSpaceport]

I'm got a replacement panel. It's going to all get a repunch and if there is issues will be easy to fix from now on. I think I had too little slack on the back for some of them. I don't know why I didn't make a proper service loop. Okay I do know why but it was a poor reason.

[@LithiumSolar]

@@DigitalSpaceport Yee I learned the hard way to leave a service loop. It's not fun re-pulling an 80ft line under fiberglass insulation because it's 1ft short LOL (looking at you porch IP camera....). Along the same idea, running me some OM3 to a few locations and trying to decide if I should get LC-LC keystones so I can patch on the front of the rack or if that's a waste of time/money and just run the OM3 directly to the switch. Patches would look nice but +$10/ea really.

[@TheSasquatchjones]

Love it! Yep...got a hodgepodge of cables too that I have made great efforts to clean up, but the clutter bug in me always finds a way around that 😂. Good content.

[@DigitalSpaceport]

My problem is it's fast to just pop in a replacement, but hard and time consuming to fully remove. I'm taking steps to fix this but yeah clutterbug is me also

[@TVJAY]

You could use a VLAN for the WAN and eliminate the need for the small switch. One issue I had when doing this was the switch took the only available IP address on boot. I am sure this is a configuration issue but I never found out why it was doing that.

[@ianmock2]

Majority of cable modems out there lock the first seen MAC address on the port to whatever device made the DHCP request. If your switch made a DHCP request on the LAN side of the cable modem, the cable modem would lock to that MAC until rebooted or cable disconnected. Sometimes only a reboot would clear it. Best thing to do is ensure that your VLAN only has the firewall making the WAN DHCP request. If the switch is doing it, statically set the IP on the switch.

[@DigitalSpaceport]

Sounds like that switch was a managed switch/L3 switch since it could trunk. That works also but a different configuration. L2 non-smart switch doesn't need extra steps as the MAC is preserved in the VM config as it moves around hosts. L2 just gossips all packets to all ports based on MAC so it's simplistic but highly reliable and less configuration.

[@klaernie]

For using smart switches and using a VLAN to run the WAN, it's really important to disable spanning tree and LLDP on the port going to the cable modem, else these switch MACs might be the first being discovered. Learned this the hard way - even twice, since I forgot to save the switch config after fixing it the first time.

[@nguyener6615]

Thanks for the video - can you recommend a L2 switch I can use to connect the modem to? This is my first foray into Proxmox and Opnsense, and I am trying to figure out the requirements for each node. My main prox/opn machine has 4 ethernet ports, ETH0/net0 (Connected to unmanaged 2.5gb switch), ETH1/net1 (WAN), ETH2/net2 (Connected to 1gb omada managed switch), and ETH3 (unused) So would I just need 2 ethernet ports on each Prox node? one WAN is connected to the L2 Switch and the LAN is connected to omada switch? Or since I am using 3 of the ethernet ports on the main system (net0, net1, and net2), I have those mimicked on the other prox nodes, but do they need to match where they are connected? (net0 to 2.5gb switch, net1 to WAN L2 switch, and net2 to omada 1gb switch)

[@DigitalSpaceport]

Hi. I use this L2 switch. geni.us/EthernetSwitch-8port It is nothing fancy but has been working without issue moving packets this entire time. If you want to mimic the setup I have here, you need 2 dedicated physical ports and 2 dedicated physical switches. You nailed it on the setup. Keep in mind your VMs will see VRMB0 VRMB1 and the nic that is specified in the bridged ports. Those need to be consistent across your proxmox hosts in which switch they are connected to. I dont follow on your final question but consistent mappings do matter here. Once you get the LAN and WAN working the concepts will be very clear to you and then adding or adjusting additional nic ports will make sense.

[@nguyener6615]

I figured it all out! to make it simpler, I just moved the Omada 16 port poe switch onto the 2.5gb switch so I only need to mimic the WAN and 1 LAN on each device. The only issue I have now is that seemingly, the LXC's lose their IP address some bit after migration...

[@DigitalSpaceport]

LXC under Network...net0...MAC address do you have one typed in? If it says auto yes it likely does lose MAC as IIRC LXC doesnt do live migration like VM does.

[@jesusleguiza77]

Hi, I have a question, could you help me? For managing high availability, if I have an internet provider and they set up a bridge according to the MAC address of my network interface on Proxmox 1( for public static ip), won't it get confused with the same MAC address of the network interface on Proxmox 2? Cheers.

[@DigitalSpaceport]

In this video I demonstrate the opnsense virtual machine keeps the same mac address. Its what allow this to work seemlessly. The VM has the MAC and is not running on more machines then 1 at a time. Its copied to all machines but the HA monitor keeps only 1 active VM running.

[@jesusleguiza77]

​@@DigitalSpaceport That's the question I have, so the MAC is managed in the OPNsense VM, and not in Proxmox itself? There shouldn't be any issues with the Proxmox same if NICs being active and connected to the same internet provider.

[@DigitalSpaceport]

The VM owns the mac. Using an L2 switch as I do here in this video, in the specific configuration i show takes care of their being only 1 vm active on WAN.

[@jesusleguiza77]

@@DigitalSpaceport Great, thank you so much! That's just what I needed.

[@jesusleguiza77]

@@DigitalSpaceport I have another question. If I only want to migrate 2 VMs to proxmox2, but keep OPNsense on Proxmox1, how should I set up the connections so that the 2 VMs can still connect even with OPNsense on Proxmox 1? using vlan on the interface bridge connected in opnsense

[@klaernie]

Any suggestions on running opnsense as a HA pair with CARP despite having WAN being DHCP?

[@DigitalSpaceport]

I'm in that same boat, this preserves the MAC address on the VM so there is no restarting needed of the Cable modem. This setup that I am using can also be 1 part of a CARP failover network. The second OPNsense carp member should be a physical box and have 3 ports. One connected to the WAN switch, one connected to the LAN switch, and one in a VLAN used for pfsync interface. Each of the HA cluster members would need to have a member interface in that pfsync VLAN as well. It is a decent amount of setup, but that is very likely next up for me also. Then it will be a truly indestructible network.

[@klaernie]

@@DigitalSpaceport sadly I've so far found no way to make a DHCP address a CARP. Even having a true static IP doesn't help, since the provider infra disconnects me, if no DHCP requests arrive for 24h..

[@IEnjoyCreatingVideos]

Nice job! Thanks for sharing the video with us!💖👍😎JP

[@fredericomachado4116]

How did you connect the WAN port to the netgear switch?

[@DigitalSpaceport]

Just plug it in. Its a L2 switch so it just barks all packets to all ports regardless. The same MAC floating on the VM on the WAN port segments via is the real special sauce here.

[@chase11119]

20:48 you look like you're gonna tell me it's cool to not do drugs

[@DigitalSpaceport]

I get weird comments like this very very infrequently but I have no idea why. Are you a bot?

[@chase11119]

@@DigitalSpaceport I am not a bot🤣

[@DigitalSpaceport]

Okay then. What's with the comment? Im not sure what you mean.

[@chase11119]

​@DigitalSpaceport think of it like this though. You're 16-17. The dare program is putting on an assembly. The guy comes out, sits on the chair backwards and says it's not cool to do drugs. Just the way you were kneeling and 20:40 it looks like you were doing the surfing rad hand signal

[@chase11119]

​@@DigitalSpaceportbut I digress. I love your videos. You're extremely informative and I look forward to seeing your posts

[@cqwickedwake7651]

Exactly what i wanted to do, working on it hehe

[@billo6014]

How are the discs combined into one disc?

[@DigitalSpaceport]

They are not actually combined here. They are operating separately on each machined. The key is to have the VM live on the rpool zfs pool, which is the default. Then it allows pmxcfs to work seemlessly like this.

[@pryzm3745]

At this point Amazon ought to pay YOU for hosting space 👌👌👌

[@DigitalSpaceport]

I'm good with that

[@notaras1985]

Can you run it on raspberry pi

[@DigitalSpaceport]

Strongly recommend only going with the official package that does get support. Proxmox doesn't support arm chips yet. You can get it working on arm devices, but it's likely to have issues that are hard to troubleshoot.

[@Fiberton]

First things first dump the cable if you can get fiber to your home. Latency and bandwith is so much better.

[@DigitalSpaceport]

It's less then 1/4 mile build out to my house, but it's 18K to build that from both Verizon and ATT. I just can't justify the costs but I pester them every 6 months for a fresh quote. One day!

[@Fiberton]

@DigitalSpaceport Hopefully they eventually move fiber to the area. 18k is steep.

[@Bergeronwebdesign]

it would be more resilient with vCenter. Promox only has HA. vCenter has FT (fault tolerance). and those are two very different things.HA require the VM to boot up, so you have down time. FT runs a shadow copy and if the vm fails the shadow copy comes online and never experience any downtime. you can run a constant ping with a response time of 1ms and you will not lose a ping when the primary vm goes and when the secondary comes up. this is HA but it is not the most resilient when looked at from the endusers point of view.

[@DigitalSpaceport]

One could do Ceph for similar in Proxmox, but the VM instance does not boot up as you are saying. It's memory state is copied and it's active the entire time. You would have seen much longer then a 1ms break otherwise. Also VMware is no longer end user attainable cheap.

[@markdeckard7651]

vCenter is dead, dingaling