How To Bond NICs In Proxmox That Support VLANs

In this video, we show how to configure Proxmox to bond multiple network interfaces and support VLANs
We show how to bind multiple interfaces together for more network throughput or network redundancy
And cover how to re-configure the management interface so it's in your management VLAN
We show how to configure a network switch to support VLAN tagging for Proxmox
And how to add additional VLAN interfaces to Proxmox as well as how to configure virtual machines to support VLAN tagging
Useful links:
pve.proxmox.com/wiki/Network_...
=============================
SUPPORT THE CHANNEL
Donate through Paypal:
paypal.me/DavidMcKone
Donate through Buy Me A Coffee:
buymeacoffee.com/dmckone
Become a monthly contributor on Patreon:
/ dmckone
Become a monthly contributor on KZbin:
/ @techtutorialsdavidmckone
==============================
Steps taken
1) Configure Proxmox
The default installation creates a dedicated management interface so we need to reconfigure Proxmox to bind the physical interfaces into a bond
The quickest way is to edit the /etc/network/interfaces
cp /etc/network/interfaces /etc/network/interfaces.bak
nano /etc/network/interfaces
To load balance traffic, the recommendation is to use LACP, for example
auto lo
iface lo inet loopback
iface enp0s25 inet manual
iface enp7s0 inet manual
auto bond0
iface bond0 inet manual
bond-slaves enp0s25 enp7s0
bond-miimon 100
bond-xmit-hash-policy layer2+3
bond-mode 802.3ad
NOTE: If your switches support technology such as MLAG, you can connect the interfaces to different switches. Otherwise they need to be connected to the same switch
To support network redundancy, an example would be
auto lo
iface lo inet loopback
iface enp0s25 inet manual
iface enp7s0 inet manual
auto bond0
iface bond0 inet manual
bond-slaves enp0s25 enp7s0
bond-miimon 100
bond-xmit-hash-policy layer2+3
bond-mode 802.3ad
bond-mode active-backup
bond-primary enp0s25
Either way, we need to reconfigure Proxmox with a management VLAN interface and make the Linux bridge VLAN aware. For example, to use VLAN 100
auto vmbr0.100
iface vmbr0.100 inet static
address 192.168.100.10/24
gateway 192.168.100.254
auto vmbr0
iface vmbr0 inet manual
bridge-ports bond0
bridge-stp off
bridge-fd 0
bridge-vlan-aware yes
bridge-vids 2-4092
Then we reboot Proxmox to make sure the changes are applied
2) Configure Switch
Because Proxmox is now doing VLAN tagging but the network switch is not, we cannot access Proxmox remotely anymore so we need to reconfigure the switch ports
We need to turn the port into a trunk port i.e. one which supports multiple VLANs and set the default, PVID or native VLAN to something unused
Once this is done, the switch will send VLAN tagged traffic to Proxmox and our connection will be restored
3) Add VLAN interfaces to Proxmox
Proxmox now has a VLAN aware bridge, so we can add VLAN interfaces to it for direct access to a storage VLAN for instance
To do this select the computer in the left hand pane, navigate to System | Network and from the Create drop down menu select Linux VLAN
Change the name to match the bridge and VLAN ID, for example for VLAN 200, we'd use vmbr0.200
Give the interface an IP address and mask, for example, 192.168.200.10/24
Then click Create
On the switch, update the port connected to Proxmox to allow this VLAN
Proxmox should now have access to other computers on your network in this VLAN
4) Assign VMs to VLANs
Because Proxmox and the switch are using VLAN tagging, virtual machines will need to support VLANs
For existing virtual machines, select the VM in the left hand pane then select Hardware
Click on the Network Device and click Edit
In the VLAN Tag field, enter the VLAN ID then click OK
On the switch, update the port connected to Proxmox to allow this VLAN
Repeat this for any other interfaces the VM has
For new VMs, add the VLAN tag as part of the installation process
Credits:
LoveLife | Instrumental Prod. Blue Mango | EQMUSEQ.COM by Don Da Vinci
soundcloud.com/eqmuseq/loveli...
proxmox vlan aware,proxmox vlan bridge,proxmox vlan trunk,proxmox vlan aware linux bridge,proxmox vlan interface,proxmox networking,proxmox networking tutorial,proxmox networking vlan,proxmox,proxmox tutorial,proxmox network,proxmox network vlan,proxmox network bond,proxmox ve network configuration,proxmox network setup,proxmox bond,proxmox bond nics,proxmox vlan,proxmox vlan tagging,proxmox redundancy
00:00 Intro
00:45 Overview
04:10 Configure Proxmox Networking
16:06 Multiple Interfaces - Throughput
36:46 Dual Interfaces - Redundancy
52:58 Add VLAN interfaces to Proxmox
59:33 Assign VMs to VLANs

[@TechTutorialsDavidMcKone]

If you want to learn more about Proxmox VE, this series will help you out kzbin.info/www/bejne/qXm6ioiqZbtgmZo

[@laberpapa]

Thank you! very helpful for an Proxmox Newbie! 😀

[@TechTutorialsDavidMcKone]

Good to know the video helped

[@jayjarrett732]

I cannot thank you enough on this. I ended up not using the tagging but your video explained how to set up the vmbr0 correctly. Thank you!

[@TechTutorialsDavidMcKone]

Thanks for feedback; Always appreciated And good to know the video was helpful

[@mithubopensourcelab482]

Excellent Video by all standards. You made it so easy. I read Proxmox documents on this at least 10 times, but could not grasp it properly. After watching your video I have realized what I actually read. Bravo. 100 out 100 .

[@TechTutorialsDavidMcKone]

Thanks very much for the feedback, I really appreciate it And glad to hear the video helped

[@Bill_the_Red_Lichtie]

Fantastic! Clearly explained why and how.👍

[@TechTutorialsDavidMcKone]

Good to know the video was useful

[@hyperprotagonist]

I know this is a year old but lord did it help me out of a sticky situation.

[@TechTutorialsDavidMcKone]

That's good to know

[@dbishop9085]

These videos are much longer than I ever care to watch, but I have yet to find a proxmox video for this guy that has let me down. If I had started here instead of looking for shorter videos for two hours, I would have long since been done! Thank you for posting these videos!

[@TechTutorialsDavidMcKone]

Good to know the video was helpful It is proving difficult to keep videos like this short as I wanted it to cover both the network switch as well as Proxmox VE itself That's because usually folks know about one part of the puzzle but not both

[@dbishop9085]

@@TechTutorialsDavidMcKone Well, it taught me a valuable lesson. Firstly, to check to see if you have something on the subject FIRST. I have learned more than once that your videos are the place to get it right the first time. And secondly, some things arent going to work in a shorter time frame. A longer video also gives time to explain the important details that the other ones were missing. Its worth pointing out that once I set up the interfaces file as you have it, I was still able to access the PVE node before aggregating the ports on my unifi UDMPro. Im not sure if that is by design or if it was able to somehow use one port still?

[@TechTutorialsDavidMcKone]

@@dbishop9085 Well if the switch thinks the server is still on one port and pve only uses that port the traffic will still work, depending on what the untagged vlan is

[@TechTutorialsDavidMcKone]

@@dbishop9085 Well traffic in an untagged VLAN can still be exchanged. And until the switch is configured they'll both still have an active link. But that doesn't work when using LACP in bonding as both ends needs to use it

[@RayBitton]

I needed that. Thank you so much for sharing your knowledge. Greatly appreciated.

[@TechTutorialsDavidMcKone]

Glad it was helpful!

[@RayBitton]

@@TechTutorialsDavidMcKone Just finished watching the entire video again because yesterday I received my first homelab server (HP Z840, Dual E5-2680, 256GB RAM). I installed Proxmox with my dual NICs in LACP configuration to my pfSense Box with 2 VLANs. So awesome!!! Thanks again David.

[@H22Designs]

Hi David! I stumbled across your channel a few days ago. Your content is EXCELLENT! You are awesome at explaining these concepts and practices! Thanks!

[@TechTutorialsDavidMcKone]

Thanks for the feedback, much appreciated And good to know you're finding the videos useful

[@DarrylGibbs]

Likewise! I'm working through the VLANs and Proxmox videos. Really encouraging stuff, and clearly demonstrated! Thanks again!

[@franciscooteiza]

Amazing video! Thanks for sharing your knowledge.

[@TechTutorialsDavidMcKone]

Thanks for the feedback and good to know the video was helpful

[@iteasy1208]

Soo much value in the vid. Thank you

[@TechTutorialsDavidMcKone]

Thanks for the feedback and good to hear the video was helpful

[@vincentmartin2528]

I have a Dell r720 with 4 x1g NICS and a Cisco 3750g switch. I was struggling for over a day trying to figure out how to get the Proxmox IP to work with the bond. Almost borked my connection a couple of times while bumbling around with it. Thankfully I ran across your video and it helped me understand the problem and the solution. Thank you VERY much for the video and explanation. Note to Proxmox: Jeezus...could you make it any harder to do something so basic and simple? FIX THIS. Otherwise, great product. Thanks!

[@TechTutorialsDavidMcKone]

PVE sits on top of Debian which is the real hypervisor So all the network changes are to the Linux OS But I've since learned you can do all this through their GUI if you'd prefer Not good for automation, but still an option

[@vincentmartin2528]

@@TechTutorialsDavidMcKone Yeah, that makes sense now that you mention it. Yes, I think maybe a shorter video on doing it via the GUI, which is how I think ProxMox would prefer it done, would be great. It might also be a bit easier and less daunting to some people as well.

[@tiagomenegon87]

Very good, solved my problem, thanks for making the video.

[@TechTutorialsDavidMcKone]

Thanks for the feedback. Glad you found the video helpful

[@NB-568]

Great video, very helpful!

[@TechTutorialsDavidMcKone]

Good to know you found the video helpful

[@TAL74]

Thanks for the video . Was finally able to set up my Proxmox cluster with trunks ports. The Cisco 2960 Love it too :-) The only point where I don't quite understand is that Proxmox only uses one gateway. Wanted to isolate my MGNT traffic and let the updates run through a different interface. Wishing the channel the best and keep up the good work Greetings from switzerland

[@TechTutorialsDavidMcKone]

It's a routing/security thing A computer should have only one interface configured with a default gateway. Ideally that should be the one used to access the Internet Because, every other network you need to access, unless it's directly reachable through an interface, requires a static route configuring E.g. let's say interface 1 has an IP of 172.16.1.1/24 and will be used for Internet updates. Then it should have a default gateway as there are too many IPs out there Interface 2 has an IP of 192.168.1.1/24 and Proxmox needs to reach 10.1.1.0/24 through that interface Assuming there's a firewall with an IP of 192.168.1.254 that can provide that access, then Proxmox would need a static route of 10.1.1.0/24 which points to 192.168.1.254 The reason is that if both interfaces had a default gateway instead, the computer would send some traffic out interface 1 and some out interface 2. Each has a different IP address and nothing would work

[@TAL74]

@@TechTutorialsDavidMcKone thanks for the detailed answer David, it makes sense now. In the beginning there are a lot of things to teach and to remember. Fixed it with a router on the stick. works well I'll try to solve the double NAT problem in future:-) Have a good start into the weekend Greetings Andre

[@user-qv3dr4zx8t]

Thanks

[@TechTutorialsDavidMcKone]

Appreciate the feedback and good to know the video was helpful

[@raul230285]

Nice

[@TechTutorialsDavidMcKone]

Thanks for the feedback, much appreciated

[@OfficialRoot]

David, good work! David, please make a Proxmox virtualization video if possible where Mikrotik is a gateway in the Hetzner cloud. From experience, so that the network is secure and with vlans for virtual machines

[@TechTutorialsDavidMcKone]

Thanks for the feedback, I really appreciate it While I can see a benefit of cloud services for a business that wants to save money and make everything in that cloud publicly available I wouldn't recommend them for anything else unless it's a last resort I've heard of clouds that have been hacked and even exploited by internal staff So I won't be doing much involving clouds for the foreseeable future

[@maherkhalil007]

Great

[@TechTutorialsDavidMcKone]

Thanks for the feedback, it's really appreciated

[@RobertFoxL]

Excellent videos! Keep up the great work! Quick question - I saw another video which suggested a different way to manage VLANs and Bonds under Proxmox - especially when passing the VLANs to an OPNSense VM - so I configured mine by creating Linux VLANs directly off the bond0 (bond0.10 & bond0.20) - then I created a separate bridges for each VLAN (vmbr10 with port/slave bond0.10) - then I passed these onto the OPNSense VM directly - which means I don't have to manage VLANs under OPNSense - just in Proxmox . . . Are there disadvantages to doing it this way?? Thanks in advance and Happy Holidays . . . Robert

[@TechTutorialsDavidMcKone]

If everything is virtualised, then having multiple virtual switches/bridges can simplify things a bit and the extra resources required probably aren't a concern I actually do that on my ESXi server for instance as everything is virtualised on that because it's a Lab and I don't want any interference between that and my real network But VLANs only apply to the physical environment to save costs on buying multiple switches. You don't have to use them in a virtual environment, unless a VM requires connectivity to physical devices So I have storage switch, a management switch, etc because VLANs have no relevance. Any VM that needs access to those networks gets a vNIC in the relevant virtual switch Granted you don't have to tag a vNIC with a VLAN ID doing this, but you still have to assign it to the correct switch anyway, so... The downside is every new network needs a new virtual switch/bridge creating and every network that's no longer needed needs one taking away, so it requires more admin That may not be of much concern for say a home network, but it's very relevant to PVE which is mainly used as an Enterprise hypervisor Now, you don't have to do VLAN tagging for virtual firewalls mind The VM either gets multiple vNICs, each with a different VLAN ID assigned by the hypervisor, when they're connected to a single virtual switch/bridge Or, each vNIC is connected to a different virtual switch/bridge for which VLANs have no relevance Either way, the firewall just sees itself having multiple interfaces that need configuring The problem with having multiple virtual switches/bridges though is when you have a hybrid network In the "real world" I have computers on both the physical network and the virtual network and they need direct access to each other Everything is isolated into separate VLANs and the physical firewall requires access to most VLANs In that situation, I find it much easier to manage a single bridge that handles connectivity to the physical network Because then what I'm doing with the virtual switch/bridge is what I would do with a physical network switch If I need a new VLAN, I add it to the physical switch and the port(s) that the Promox interface(s) connect to and any VM needing access to that VLAN will get that VLAN ID tag in its vNIC The bridge is already configured to allow most VLANs, Proxmox itself doesn't need interfaces in a VLAN, unless it needs to be accessed itself on that VLAN, so there's very little work needed to add and remove networks And if I want to add more interfaces to Proxmox, to give it more bandwidth or additional redundancy, I can do that all within a single bond between Proxmox's bridge and the physical switch

[@RobertFoxL]

@@TechTutorialsDavidMcKone Thanks David for the comprehensive response! Very helpful! Once again, Happy Holidays and Stay Safe! 😷

[@jdratlif]

If you weren't using vlan tagging, could you use a non-LACP bond mode and do this with an unmanaged switch?

[@TechTutorialsDavidMcKone]

You don't need VLANs or LACP to bind links together But both sides of the link do need to be configured so that they're in agreement and you can't configure anything on an unmanaged switch so you still need a managed switch

[@sidneyking11]

Hi David, I am stumped on adding a second internal sata drive in a proxmox server to be used as a shared drive for the VMs to be-able to access it as a file storage. How to setup hard drive passthrough so that a VM that is running docker with a container that is running jellyfin to access that drive. Can this be done? is so what is the best method? I see that some people create a vm and run TrueNAS but I think that is too many layers and makes things more complicated. Thank You

[@TechTutorialsDavidMcKone]

I've used a TrueNAS VM myself for this sort of thing Containers aren't as secure as VMs as it exposes direct access to the hypervisor So I normally avoid them for that reason alone But you'd probably also have to battle the likes of AppArmor which controls application access I have heard of issues trying to allow one container to access another because of this If you create a VM and install TrueNAS in that, any VM can have direct access to it if they have vNICs in the same network Even physical computers can have access if VMs are accessible from the LAN

[@bhupindersingh3880]

Hi David, Great Video. Can you guide me on an issue I am facing issues getting the Proxmox to talk to the switch. Firewall to switch are configured for the VLANS and they are pinging both ways. But unable to ping the switch from the Proxmox Interface --- Getting unreachable on the command line. Have set up a bonded interface on both sides (switch & Proxmox). In Proxmox I have called this bond in a bridge restarted the network and tried pinging My management network is on a separate card and subnet. Can you give some pointers ? iface enp3s0f1 inet manual auto ens4f1np1 iface ens4f1np1 inet manual auto ens4f0np0 iface ens4f0np0 inet manual auto bond1 iface bond1 inet static bond-slaves ens4f0np0 ens4f1np1 bond-miimon 100 bond-mode 802.3ad bond-xmit-hash-policy layer2+3 auto vmbr0 iface vmbr0 inet static address 172.16.100.31/16 gateway 172.16.0.1 bridge-ports enp3s0f0 bridge-stp off bridge-fd 0 auto vmbr3 iface vmbr3 inet static address 172.20.124.5/24 bridge-ports bond1 bridge-stp off bridge-fd 0 bridge-vlan-aware yes bridge-vids 2-4092 Any pointer shall be helpful

[@TechTutorialsDavidMcKone]

First thing I notice is that the bond is set to static but it has no IP address, so it should be set to manual After that I see you have LACP configured, so the switch needs to be using that as well And unless this is a high end switch chassis for instance, both server interfaces must be connected to a single switch Because the bridge is VLAN aware it's better to set up VLAN interfaces in PVE as shown in the video, but the switch ports will need to be configured to tag traffic as well In the current configuration, PVE will send traffic without a tag out its interface The switch may then put this into a different VLAN to its own interface and so there's no connectivity It's best not to use untagged traffic and so tagging traffic on both sides is the best practice

[@michaelcooper5490]

Hello David, Can you do a video with Netgear GS108PEv3 with VLans w/Proxmox please, Thank you.

[@TechTutorialsDavidMcKone]

Something like this is on my to do list but it's taking a while to arrange I bought a GS108Ev3 for this very reason so I'd suggest checking out my review for how to configure VLANs on that For now though, I'm still struggling to find some low power computers to go with it Basically a lower power motherboard and iGPU that can handle a 10GB NIC but the parts are either not cost effective or not available

[@michaelcooper5490]

@@TechTutorialsDavidMcKone Hello David, I found the HP Elite Desk mini G2 800 was a perfect fit for me at 35w to 60w it's as low power as you can get. I have 5 of them right now and they all have 8 cores and 32 gb of ram and 250 gb nVME storage in them and they work perfectly.

[@TechTutorialsDavidMcKone]

​@@michaelcooper5490 Thanks for that I've been looking at the HP ProDesk SFF for 8 core options but the idling time was hard to find I had just started looking into the EliteDesk for newer CPUs so will check the 800 range that you've managed

[@bzdzgwa]

Can proxmox be configured to route packets between VMs in different VLANs so packets don't have to go through an external router?

[@TechTutorialsDavidMcKone]

It could be it wouldn't be advisable because if a VM were ever compromised it would provide a means to try and access Proxmox VE from it A virtual firewall would be a better choice But if you just wanted open routing then a VM configured as a router

[@0ChAnTi]

hello david, what about building a bond0 over bond1 and bond2, when you have a tier3 network setup?

[@TechTutorialsDavidMcKone]

An interesting idea but I don't think it would be practical I assume bond1 goes to switch 1 and bond2 goes to switch 2? Otherwise you may as well just have lots of interfaces in one bond going to a switch The problem I see with a tiered set up is I don't think it could be done or wouldn't work as expected A bond is a logical interface made up of physical interfaces Even if the software on the computer will let you bind logical interfaces together, I don't think a switch would let you do that, or at least I haven't come across one And even if you could, you need centralised switch management of the switches so they share this bond0, a bit like Cisco's Nexus switches Otherwise we'd have the same problem as trying to create a single bond to two different switches because switches don't like a MAC address moving around from one port to another, hence the preference for active/passive NIC teaming In other words, if bond0 only exists on the server, sometimes traffic would appear on switch 1 and then on switch 2 and the switches would have to deal with a flapping MAC address, just the same as if we'd created a single bond on the server but none on the switches

[@0ChAnTi]

@@TechTutorialsDavidMcKoneActually, it is a SDWAN setup with a vsphere virtualized server.

[@TechTutorialsDavidMcKone]

If it's all within the hypervisor there's no need for bonds as it's unusual to only lose one virtual switch as they all run from the same code But physical server to switch is either one bond to one switch, or one bond to multiple switches provided the switches have centralised management and can present themselves to servers as a single switch

[@hendranatasaputra2826]

can u give explanation why we dont choose OVS?

[@TechTutorialsDavidMcKone]

To keep things simple A Linux Bridge can do VLANs and we have one already configured for use "out of the box". Any change needed is then minimal OVS does offer extra features but unless you need those I don't see a gain in using it

[@hendranatasaputra2826]

@@TechTutorialsDavidMcKone is it stable to using OVS?

[@TechTutorialsDavidMcKone]

Open VSwitch has been available for a long time It's included with Debian for instance and that's heavily focused on stability And it's used by companies in production environments So I'd have no concerns with using it if I wanted to virtualize everything for instance

[@dhs3771]

Hello, can you give some lessons on openvswitch, please?

[@TechTutorialsDavidMcKone]

It's not a top priority for me I'm afraid but I will add it to the list