Proxmox SOFTWARE DEFINED NETWORKING: Zones, VNets, and VLANs

I made a Proxmox VLANs, Bridges, and Bonds tutorial awhile ago, but since then, the Software Defined Networking module has come out of tech preview! So it's time to take a look at it!
With SDN, you can manage your Proxmox VNets and VNet Zones cluster-wide, and enforce permissions on users who can configure VNets for VM and Container resources. While the SDN has additional functionality for managing DNS, DHCP, and VXLAN, those are still in tech preview so I'm only going to cover the basics today. Let me know if you want a follow-up on those topics!
Proxmox also has a section in their admin manual on SDN which you may find helpful, as it covers all options thoroughly:
pve.proxmox.com/pve-docs/chap...
Support me on Ko-Fi if you enjoy my content and find it useful:
ko-fi.com/apalrd
Feel free to chat about my upcoming projects on Discord!
/ discord
Timestamps:
00:00 - Introduction
00:30 - Beta Features
01:10 - Upgrade Install
02:07 - Pre-SDN
03:54 - Post-SDN
08:22 - VLAN Zones
09:38 - Permissions
13:23 - QinQ Zones
16:59 - Tech Preview Review
#proxmox #virtualization #homelab #networking

[@stephendetomasi1701]

I'm about 8 minutes in and my head is already spinning, but it looks like a great tutorial. Thanks again for covering this stuff - if only official documentation was this good!

[@drewlarson65]

It's worth a few watches, I use some of his videos for reference regularly.

[@patrickcasavant1044]

Can't wait for the evpn/vxlan part!! :) Your explanations are awsome!

[@jasonm2477]

Im happy to see that im not the only one who always chooses those vlan id's in test networks

[@falazarte]

You are my go-to channel for learning networking! You deserve more than a coffee

[@apalrdsadventures]

Thanks!

[@autohmae]

honestly, I think developers with networking knowledge are the best networking educators.

[@snowballeffects]

As always - thorough, informative and easy to digest. Thank you!!

[@DavidVincentSSM]

thank you for this video, i've been waiting for someone to post on the new SDN features!

[@marcogenovesi8570]

I've used the beta plugin for a year, very excited this is now released

[@drewlarson65]

I got way too excited when I say this video come across my feed! Well done apalrd!

[@MarkConstable]

Damn, I was hoping this would include VXLAN and EVPN, but I guess that would deserve a followup video all by itself anyway. My use case is distributing a public /24 across all nodes in a cluster without any help from upstream.

[@apalrdsadventures]

Are you the next-hop for upstream or is it expecting the whole /24 subnet to be on-link?

[@MarkConstable]

@@apalrdsadventures Woops, missed your reply earlier. I want the entire public /24 to be available across all 3 nodes. I think we need that BPG EVPN VxLAN tutorial. Pretty please 🙂

[@apalrdsadventures]

It's also an option to push /32 routes from the VM itself into an IGP, and then aggregate those in BGP. All Proxmox hosts advertise the /24 upstream, then route amongst themselves to the destination.

[@2APatriot]

Wow you made this easy. Already got it running on the test lab

[@FrancescoCarucci]

same... two months and I couldn't get it working, 5 minutes from this video and it's up and running great...

[@gasparem16]

you are the man! I've been looking for a good video on SDN in proxmox! Thanks a lot for your great videos and tutorials!

[@Felix-ve9hs]

I will definitely be using this with my Proxmox hosts, so much better than dozens of VMBR bridges or remembering VLAN IDs. :)

[@PatrickBulteel]

Wow. Great explanation. Look forward to the rest. I'm about to deploy a Netbox server so I can use the IPAM portion. That's going to be interesting.

[@DawidKellerman]

Yes ! I was to lazy to figure it our and I have not watched your video but you already have a like from me!

[@_andrey___]

Hey, nice tshirt.

[@zparihar]

Amazing Bud! You're amazing!

[@HarmoniousVibrations]

Perfect timing, thank you ❤

[@fcojperez]

Well done, nice video. Thanks for sharing your knowledge 👍🙏

[@FrancescoCarucci]

Sir, you are a legend.

[@nevermetme]

You can even give permissions to a single vnet. Though currently not in the DC->Permission panel. But if you select the Zone in the tree view, you can select the vnets and define permissions for it on the panel on the right side. Great video and nice explanations :)

[@apalrdsadventures]

Thanks for the info!

[@hegharm]

Thanks for the review of SDN Proxmox. The topic that remains unsolved is how to harm access to servers from the Internet, for example, to several web servers on different virtual machines.

[@apalrdsadventures]

I left that out because it's still an SDN beta feature currently, I want to wait for it to be more finalized.

[@hegharm]

@@apalrdsadventures Thanks for answer.

[@eDoc2020]

This is great, even if it's just as a way to refer to different VLANs without using numbers. "Port groups" is one of the things that was nicer in ESXi. Now the only thing missing (that I cared about) is the ability to have ISOs stored in a hierarchical layout. I like to keep my data sorted. I guess it would also be nice if VM disk resources also had customizable names. "vm-101-disk-1" in a ZFS status view doesn't mean much but "adserver-bootdisk" does.

[@Darkk6969]

This SDN feature makes me wonder about setting up something like vxlan to route traffic between ProxMox clusters via the WAN. I'll have to look into it.

[@apalrdsadventures]

SDN supports vxlan as well, it's still part of the tech preview. I'll do a video on that eventually. Unicast vxlan is pretty simple to setup but doesn't scale to super large clusters like EVPN does, but EVPN is way more complex.

[@autohmae]

BGP announcing MAC-addresses for routing, I ... hadn't expected that one, but it actually sounds pretty great. That might be a great way to scale large installations.

[@patrickcasavant1044]

Yes take a look at MP-BGP.

[@autohmae]

@@patrickcasavant1044 I knew it was used for MPLS, IPv6 and IPv6 and it could be used for other things in theory... but just never considered MAC-addresses

[@edwardvanhazendonk]

SDNs are very nice, may I ask for a little drawing next time you are creating nd explaining this? You talk us through with what you are achieving which is great but a picture upfront might give us just a bit more info and insights. This does not take away that you are great in explaining. Keep up the good work and thanks for sharing.

[@apalrdsadventures]

I'll make sure to add drawings to the evpn / vxlan video!

[@zparihar]

Looking forward to VxLAN

[@louissenderler6866]

It will be great if you can demo how each SDN Zone works and what networking scenarios they are ... especially for QinQ, VXLAN and EVPN.

[@hans-ulrichfluck8076]

Danke!

[@apalrdsadventures]

Thanks!

[@mediatv1867]

Thanks for the video! And like for Лайку)

[@gautamkrishnar]

thanks

[@mcsv]

superusefull, thank you! Have you heard Ice-Mc's "Laika"?

[@Cmdrlucky8]

If you can send routed packets via UDP to proxmox entities in different broadcast domains, could you use this for multicasting to different domains? I'm thinking like fog imaging to different VLANs

[@apalrdsadventures]

vxlan does exactly that, and yes it's designed for bridging across a layer 3 routed network. It supports multicast as well, but via unicast flooding (e.g. if there are 5 Proxmox nodes, a multicast packet sent from 1 will be unicast to the other 4 nodes as 4 separate packets).

[@karloa7194]

Question about the VLAN zone. Does this mean that the trunk link between the Proxmox node and the network switch can be done via the SDN VLAN zone? I'm using OpenvSwitch and created IntPort for each VLAN tag. For what I can tell from your video, there is no need to create the OvS tags anymore. The tags are now done in SDN VLAN zone. Is that correct?

[@apalrdsadventures]

The trunk is still configured in Network for each node. You name the trunk the same on each node, and Zone/VNets will be parented to the trunk interface by SDN. In my case, the trunk is vmbr0. VNets are equivalent to vmbr0.x in this case. When using OVS instead of Linux Bridge, SDN will create the IntPort automatically for the VNet. So the OVS Bridge is again the trunk, and individual IntPorts are not created manually.

[@tvojejbabkydedko]

is there a way or reason to implement SDN if i use pfsense as router and currently use separate vmbr bridges to separate interfaces?

[@apalrdsadventures]

SDN would help you organize and name the interfaces, if you are using separate vmbrs they would become separate Simple Zones in SDN with proper names.

[@falazarte]

What about OVS? I dont see mufh love for OVS, isn't a sort of SDN tool?

[@antionline8856]

hi @apalrdsadventures love your videos. can you make a video on how to setup pfsense hosted on proxmox and out to mikrotik with vlans? thanks

[@seapro4018]

Very interested. I wonder if you could answer or suggest a method that I'm trying to accomplish. I have a DR site with replicated/restorable servers and backups. I need to ability to create a virtual network whereby I can load/test/restore my servers from Site A on Site B and have them communicate with each other - but not the internet. After mounting all the servers, then I would initiate a RDP session to 1 of the servers and then be able to communicate with all the other servers on that virtual network. The networks are different between the 2 sites and the vm's also have different vlans on them. I can currently restore/mount a server at Site B from Site A I'm unsure how to tackle this but would want the solution to be simple. Is a Bridge the easiest method over another VLAN or SDN ? thanks - mark

[@alshayed]

Are you able to get it to work with a VLAN based VNet that uses the same VID as the management IP? Like if vmbr0 (vlan aware with pvid 1) has 10.0.0.2 and you create a vnet tagged vid 1 does everything work? In my testing once I do that my management ip address stops responding.

[@apalrdsadventures]

It will create a new bridge bound to the vlan ID for the VMs, which will remove it from vmbr0. So no, it won't work in this case. You could add some lines to /etc/network/interfaces manually to fix this, giving an IP on the new VNet.

[@hotrodhunk7389]

My question is can i do a vlan for proxmox hosts without an external managed switch? All the research i did showed that a non managed switch would just ignore the vlan tags and send it out anyways?

[@apalrdsadventures]

It depends a bit on the switch. Some switches will ignore vlan tags but still pass them as part of the packet, which is fine if all of your devices are vlan-aware but can royally confuse any devices on the network which are not vlan-aware. Other switches will strip vlan tags. If your switch can't handle VLANs and you need to carry multiple VNets between cluster nodes without routing, your best bet is vxlan. In a small cluster, unicast vxlan is way easier to setup than bgp evpn vxlan. It will tunnel each vnet inside of UDP on the outer ('underlay') network, so you will lose some payload space (lower MTU) as a result. I'm going to do a video on this as it matures fully. Some routers (I tested with Mikrotik and OPNsense) can also do unicast vxlan, so the whole setup can be done all the way to the router without supporting VLANs on the physical network at all. SDN won't help you configure your router though, just the Proxmox side.

[@hotrodhunk7389]

@@apalrdsadventures wow that sounds perfect! I should have just spent another $10 and got a managed switch 🤣😂🤣 but being super cheap is part of the fun for me. Thank you I will definitely look into unicast vxlan! Definitely will wait for your video. 😁 Openwrt guide would be perfect for me... Just putting it out there...

[@apalrdsadventures]

I don't use OpenWRT myself, although being Linux-based it should support unicast vxlan (and also bgp evpn vxlan with frr), if the system has enough memory of course. Unless OpenWRT compiled it out on their kernel build, which I don't think they did. In Proxmox SDN, the 'basic' way is to create a VXLAN (not EVPN) zone, and set all of the IP of all of the Proxmox nodes (separated by commas) in the peer list, and it *should* just work. Proxmox *should* compute MTU for you automatically (and it's going to be around 1440 or so).

[@mx338]

Interesting that Proxmox is embracing more enterprise data center features, makes me wonder if they want to enter vSphere/OpenStack territory.

[@apalrdsadventures]

VXLAN / EVPN are both working quite well already, but still being in tech preview I didn't want to talk about it just yet. (there are also some IPv6-related quirks with vxlan which are the fault of nvidia basically abandoning ifupdown2 after buying Cumulus Networks).

[@zyghom]

@2:07 - installation of dnsmasq is forgotten here and it will not work until installed ;-)

[@ChrisDePasqualeNJ]

Love your content: My environment New Proxmox 8.1 on hp elitedesk with additional USB 1GB adapters. Problem is, while following your tutorial creating VNet I get this error: netlink : error: netlink: enx00051bc91f64.6: cannot create vlan enx00051bc91f64.6 6: interface name exceeds max length of 15. So is there anyway to rename the two USP network adapters? I believe they were auto created using the mac.

[@apalrdsadventures]

yeah, that's the character limit. enx interfaces are already 15 letters long, so you can't add anything on the end. You can write a rule to give an adapter with a specific MAC a specific name, instead of the default. See here: www.apalrd.net/posts/2023/tip_link/ In your case you'd create one file for each, with a different MAC and name, and after reboot they will get renamed. You will need to update your network configs to refer to the new name, so be prepared for that (this might require manually editing /etc/network/interfaces to replace enx123456 with enge0 for example). If you ever replace that USB NIC, it won't find it any more (MAC will be different) and will create an enx123456 interface, so just edit the new file with the new MAC and reboot and it should come back up under the right name.

[@ChrisDePasqualeNJ]

@everyone IF i rename the interface from enx00051bc91f64.6 to say, enx1f64 in the /etc/network/interfaces file along with other references and save and reboot do you think that will work or will I just break my install. Please feel free to give your thoughts. Thank you,

[@ChrisDePasqualeNJ]

@@apalrdsadventures Thank you! You are so smart! Honestly I'm so impressed. Sorry I made the comment below before seeing your reply. I will let you know how things turn out. 🙂

[@X0M9JKEEE]

Лайка (:

[@hpsfresh]

Like for the t-shirt

[@zyghom]

I usually understand your videos, today was kind of... no. Probably lack of SDN basics. But still nice video. I don't see any usage of SDN but again: I simply don't get it (yet) ;-)

[@damiendye6623]

Just the same as VMware distributed switches

[@Solothedrunk]

I was running into this Warning: WARN: missing 'source /etc/network/interfaces.d/sdn' directive for SDN support! I was able to fix it by adding source /etc/network/interfaces.d/* to the BOTTOM of the /etc/network/interfaces file.

[@apalrdsadventures]

Ah yeah, that will show up if you updated from a previous version of PVE. It's included now. You can add it anywhere in the file, top or bottom.

[@ernestoditerribile]

Your Keyboard looks a lot like a modern iteration of a IBM Model F/M series keyboard

[@VitaliySunny]

Nice shirt

[@ertanerbek]

The simple version only works for guests on the same host, it does not work on a cluster basis. Or they have some needs other than SDN.

[@apalrdsadventures]

The Simple Version is designed to be routed in a cluster (each cluster node has a subnet, and the host acts as a router + DHCP/RA server)

[@ertanerbek]

@@apalrdsadventures It doesn't work quite as designed; guests on the same host can talk to each other, but cannot talk to guests on another host.

[@apalrdsadventures]

Each host would be a different subnet, so VMs will get an IP from the subnet of their host, and can route across to other subnets via the host. Not all of this is implemented yet, but that's the design goal of Simple Zones.

[@ertanerbek]

@@apalrdsadventures Dude, you don't select any uplink in simple zone. How will SDN know which interface to send traffic from? Simple zone is a system that works on a host basis, not on a cluster basis.

[@apalrdsadventures]

It doesn't send from a specific interface, it's routed using the system routing table. The PVE host's IP on the zone is the gateway for VMs in the zone, and PVE is routing at layer 3. Presumably if you are using it in this way you either configure your upstream router with static routes back to the Proxmox hosts, or use an IGP like OSPF/IS-IS (or even BGP) to exchange routes in the underlay.

[@OlgerdGolub]

майка клевая - привет от лабродвора

[@Superturisto]

Oh, never changing that T-shirt, are you? Channeling your inner russian, huh?

[@autohmae]

10:37 honestly, is this a quirk...? by some interpretation I would say this is intended behavior.

[@apalrdsadventures]

Oh I agree it's a good behavior for the permissions issue, but it's something you need to be aware of if it comes up.

[@autohmae]

@@apalrdsadventures that's probably true !

[@mikekane9734]

Are you russian? What is your tshirt about?

[@apalrdsadventures]

I am not Russian, it's the first dog in space (Laika). I have a collection of space-related shirts and this one always gets way more comments than the James Webb Space Telescope one.

[@mikekane9734]

@@apalrdsadventuresHah, yeah! in fact she was one of two. Thank you for the video!

[@bluearcherx]

and people wonder why vmware is better

[@apalrdsadventures]

Until vmware decides you're too small to sell to

[@Glatze603]

Hi, can you perhaps speak a little slower and more clearly? Your sound quality is relatively poor, making it difficult to understand you and the automatic translation only works sporadically. Thanks a lot 🙂

[@youtubear02xdax]

If find his paste of speed very good. Every sentence precise and without any impurities like other KZbinrs do. (With other KZbinrs you have to watch a 30min video for 5min worth of useful content. Here you watch a 20min video with 40min pure information which is all useful) It does require basic knowledge about the topic though, probably not the best for complete beginners. But every video of him is gold worth :D

[@Glatze603]

@@youtubear02xdaxit is not the content I am talking about! It is the audio quality. I don´t understand if you talk too fast.

[@grumpyoldman5368]

@Glatze603 In the player you can set slower or faster playback speeds, so you might try setting 0.75 and see if that helps you understand.

[@Glatze603]

@@grumpyoldman5368 It would be enough for me if the automatic translator could do it properly. But this also requires clearer pronunciation, so speaking a little slower and more clearly. Maybe it would also help if the audio recordings were a little better.

[@MarkConstable]

@@grumpyoldman5368Yes to speed up/downs, and we can be grateful there is no background music!