Software Defined Network Guide - SDN - How To Create

Proxmox 8.1 dropped some great features including SDN (Software Defined Network) and Notifications. Find out how to create a SDN quickly in this video.
Recommended Hardware: github.com/JamesTurland/JimsG...
Discord: / discord
Twitter: / jimsgarage_
Reddit: / jims-garage
GitHub: github.com/JamesTurland/JimsG...
00:00 - Introduction to SDN
00:57 - Installation
01:49 - How To Create SDN
06:27 - Using SDN On A VM & Testing
08:03 - Outro

[@FTLN]

HI Jim, thanks for the video. Been waiting for a tutorial for SDN :) :) Would love to see a deep dive on this !!

[@Jims-Garage]

No worries. I'll look to cover in more depth at a later time. It's pretty similar to a standard network though once you understand how to do the first one.

[@MarkConstable]

Excellent. Please explore more SDN configurations and use cases.

[@MadChristianX]

I just wanted to take a moment to thank you for the fantastic work you are doing on your KZbin channel. Your videos are incredibly informative and helpful. I'm particularly interested in Proxmox and its SDN capabilities. If possible, I would love to see a deeper dive into this topic in your future videos. Your insights and detailed explanations would be greatly appreciated. Thanks again for your hard work and dedication to educating others. Best regards, Christian

[@Jims-Garage]

Thanks so much for the feedback, Christian. I will come back to it in future. Interested to see what it's capable of.

[@kgottsman]

Jim. Quality video again. You have quickly become my favorite tech/homelab content creator. I definitely watch out for all your latest videos. Keep up the good work.

[@Jims-Garage]

Thanks, I have something a little different planned for tonight 😊

[@raulgil8207]

your videos are getting better and better, thanks !!!

[@Jims-Garage]

Appreciated, still learning 😂

[@mikeandersen8535]

Just discovered your channel, and you have videos for a lot of the stuff I am in the process of setting up in my homelab: Opensense, separating all VMs to a single VLAN, SSO, private CA...and experiment with Zero Trust ideas... without having use for any of it (but fun to play around with). So a +1 for the sub from me. ;)

[@Jims-Garage]

Thanks, and welcome. Sounds like you have tons of exciting things planned. Zero trust is something I want to cover soon.

[@mikeandersen8535]

@@Jims-Garage A friend of mine has a zen-like saying: "Zero Trust is a road... not a destination". Quite the contrast from what we hear from big tech, advertising their products. ;) Working with security, this "assume breach"-way of looking at everything, is quite useful. Never had to do the "dirty work" myself, which is the reason for wanting to do it in my homelab.

[@lightechoes]

Thank for you covering this!

[@Jims-Garage]

My pleasure!

[@LampJustin]

Really neat, been working on BGP EVPN at work for some time now and will definitely try out the integration!

[@Jims-Garage]

Bgp is supported in Proxmox now

[@swubutu]

Hi Jim, Thanks for your Tutorials. decent clean and easy to follow inc this one ;) Plus 1 on comment of FTLN... woudl be nice to get e better overview , how different subnets via SDN can talk to eachother !

[@Jims-Garage]

Thanks 👍

[@adrianniwa1438]

Nicely explained! Thanks

[@Jims-Garage]

Glad it was helpful!

[@BekoPharm]

Nice. It was always a pain in the neck to do this all manually. This is really a missing puzzle piece finally falling into place.

[@Jims-Garage]

I totally agree. It's a welcome addition.

[@muhammadabidsaleem7048]

Hi Jim, Thank you for the SDN video. There is very minimum content on proxmox SDN. Please make detailed video on it along with the whole scenario while the proxmox sever/s is/are connected to the switches in balance-rr or Active/Backup interface.

[@Jims-Garage]

Thanks, I do plan to revisit in the future

[@SamWhitlock]

One use case I think would be useful to cover is how to have these SDNs (especially VLANs which many of us use to segment our network) interact with existing mechanisms (e.g. existing pfSense / opnSense firewalls). What's a bit confusing to me is how one can continue to delegate things like DHCP to these firewalls while still using the automatic provisioning / VLAN tagging associated with the SDN features. Thanks as always for this incredibly helpful content!

[@Jims-Garage]

You're welcome. I'm going to revisit SDNs later with some more advanced setups.

[@rabinassar]

Thanks for the vid, worked on the auto mtu, but had to do a restart to enter container console.

[@Jims-Garage]

Great, thanks for confirming.

[@-rm-rf]

Nice!

[@Jims-Garage]

Thanks 👍

[@mikkun_]

Just to note, the "test" bridge in the VM's network option is the VNET name, not the Zone name.

[@DavidVincentSSM]

i'd like to see a deep dive into vxlan if possible.. thanks for the excellent video!

[@Jims-Garage]

Thanks, it's on my to do list

[@reneb5222]

Hi Jim. Your quick. Asked yesterday and now already :), Great one. I am going to use it for my RKE2 Servers.The Apply button is a pain. Was looking for that one as well when it first came out. Btw if you are wondering where the config files are? These are in /etc/pve/sdn

[@Jims-Garage]

Great, thanks for letting me know

[@joshharding6925]

Thanks Jim for the introduction to PVE 8.1 SDN. Looking at your /etc/network/interfaces file, the system hosts 2 x quad NIC adapters (or 4 of the same NIC card... twice). Where do you stop configuring the actual hypervisor and start configuring the data center's SDN? I have a cluster of physicals with an onboard NIC and a quad PCIe NIC card. The NIC's presented to the hypervisor are eno1 and enps0f0, enps0f1, enps0f2, and enps0f3. The latter 4 are configured in a lag bond0. Normally I would configure a vmbr1 to bond0, and VLAN's from vmbr1. I don't know where to stop configuring the hypervisor and start configuring SDN. If you're able to expand on that, would be excellent!

[@berrabe3917]

i think is more like the same as creating local vmbr without bridge-ports and integrate with iptables set to srcnat or masq for the internet

[@arpansenofficial]

Great video, Explained everything very well. Do you know how this can be used to port forward? I was thinking that maybe this could replace the times we had to create opnsense vms just to have a NAT network with web ports forwarded for hosting multiple different env apps.

[@YM-xz6xt]

Hi Jim, great video again and clearly explained! I was wondering if it would be possible to host a website with a cloudflare tunnel behind a SDN by limiting the access to only internet, pretty much like a vlan.

[@Jims-Garage]

As it's a reverse tunnel I believe that would be possible.

[@Sam-kx7jc]

Great video! I've been looking for ages on how to set up NAT on Proxmox, but I can't seem to get this working on my end. My VM's are not being issued an IP address when using the SDN network bridge for some reason that I can't figure out.

[@ClemMorton]

Can you please do an explanation on how to configure SDN when the hosting provider gives you say 10 separate public ip addresses and gateways. As well as a whole ipv6 /64 sub net. I found your explanation very useful. will sub and follow.

[@Popcorncandy09]

This looks great ! what would be the example here when combining with a virtualised firewall such as sophos-xg?

[@Jims-Garage]

It's effectively creating a separate software network behind Proxmox. This means inter-VM traffic on the same SDN isn't routed through the firewall (Sophos-XG).

[@hangqaq6240]

I am doingsome staff on this SDN feature. I want to learn about the EVPN feature but I really dont know how to setup it. I think this feature will help me to create cross cluster netowrk(Not just cross node).

[@leong2757]

Hi Jim, thanks for all the informative videos you make. I’m new to homelab and I tried to follow the video but I run into one problem. When I create the simple SDN, and let’s say I used 192.168.20.x, if I create multiple vms using the same machine they’re able to communicate, but if use the same 192.168.20.x on a different machine, the 2 machines using the 192.168.20.x aren’t able to communicate. Do you know why that is?

[@Jims-Garage]

The 'simple' SDN is Proxmox node specific. You'd need vLAN or vxLAN to do multi-node. This would likely traverse your switch/firewall, unless you have a direct connection between Proxmox nodes.

[@SatriaAdyPradana]

awesome, so we won't need a dedicated VM as router. I tried using EVPN in proxmox 8.0, but I think "simple zone" is easier.

[@Jims-Garage]

Technically, no. Albeit if you're referring to a firewall I strongly recommend one.

[@SatriaAdyPradana]

@@Jims-Garage _ so far I use vyos to give DHCP to the EVPN network. When I see the "simple zone" again, there is no DHCP option. Maybe I have to upgrade to v8.1.

[@Jims-Garage]

@@SatriaAdyPradana AFAIK this is an 8.1 feature

[@timoclemens5379]

I've setup the SDN like you and its working so far. On my internal network (unifi) I created a static route to reach the SDN, however I cannot ping a VM within the SDN. From a VM to my PC it works. Can you tell me what I need to do?

[@tcasex]

I'm no network engineer...just a security engineer. What is the benefit of doing this vs just vlan on your physical firewall and segmenting it out that way? I'm assuming most companies aren't using "Proxmox" in true production environment, atleast I've never ran into it at any company I've worked for. Additionally, if you're using docker networking, then this becomes all quite convoluted and adds additionally complexity for management. I'd venture to guess this would be for more of an enterprise setup where you would want to isolate clients on separate vm networks? (i.e: proxmox in an enterprise setup). Just trying to understand the benefit of something like this.

[@Jims-Garage]

Can be useful in cloud environments where all networking is software based. You can have a single wan IP, and lots of subnets behind proxmox. In a homelab it's useful for bandwidth, you can have fast networking between hosts rather than being limited by a physical switch. E.g., writing to a Nas can be very fast!

[@thorstenebers2862]

great for clustered vms

[@Jims-Garage]

Absolutely, also helps with cable management 😂

[@TheArpitkoberoi]

Will the simple zone subnet span across nodes? Can VMs connect with each other across nodes or do I need VLAN or VXLAN?

[@Jims-Garage]

I believe it requires vxlan and vlan.

[@TheArpitkoberoi]

Vlan didn’t work since my switch isn’t vlan aware, also I needed a new subnet since this is being created for pfsync of my virtual pfsense HA setup. Used vxlan and worked right away.

[@Jims-Garage]

@@TheArpitkoberoi great, that's good to know

[@WilsonVelez]

Jim, thanks for your videos. I would like to know if it would be possible to run OPNSense on multiple hosts with pfsync using Proxmox SDN instead of multiple switches. I'm really new to home labbing.

[@Jims-Garage]

Yes, it should be possible with vxLAN. I'll try to cover this soon.

[@fastsloth-ef4pq]

@@Jims-Garage This would be awesome

[@NekoCentral]

Just followed this my self, works great except one thing, DNS is not working for me, it tries to set proxmox as the dns server but is not responding to requests from the lxc containers, havent found out why.

[@JoaquinVacas]

How does a router/firewall machine work under an SDN? I have a single machine that I use for homelab testing and I would love to test SDN but also letting some VMs like OPNSense ones work on a SDN VNet, does it treat it like another virtual network bridge under Proxmox? Thank you for your video! I've just discovered your channel :)

[@cheebadigga4092]

great video, thanks!! I have a weird issue though. On PVE 8.1.3, I created a vnet with dhcp and a valid range, however, the UI keeps saying "State: Changes", and when I hover over it, it says "Pending changes: my-ip-range-here". Clicking on Apply doesn't change that State, and rebooting the node doesn't either. I tried messing with the MTU to no avail. Did anybody experience that as well?

[@Jims-Garage]

Interesting. Have you looked in the syslog?

[@rahulkundu4u]

MTU Auto works for me.

[@Jims-Garage]

Ok great, thanks

[@KristianKirilov]

I watched this video a few months ago, I tried to replicate the setup but I've got random issues. Today I tried again, I can't say there is a progress on that - the bridge interface isn't created, so dnsmasq complains about missing interface, ifreload sometimes exit with error code 1, because of missing interface in /sys/class... because of the missing interface the dhcp is not working correctly. So I will wait another few months and hope it will get sorted.

[@Glatze603]

After applying the settings, I get the error "Could not run before_regenerate for DHCP plugin dnsmasq cannot reload with missing 'dnsmasq' package" with a fresh 8.1 installation...

[@Jims-Garage]

Interesting, perhaps 8.1 doesn't come with everything you need as per the documents. There are typos all over the notes so I think it was rushed out (quest instead of guest). Have you tried installing the dependencies?

[@lmontei]

just do: apt update apt install dnsmasq systemctl disable --now dnsmasq nano /etc/network/interfaces, insert at the end of the file: source /etc/network/interfaces.d/* and it will work. thanks @Jims-Garage for the geeky stuff :) and i did not "fix" the MTU, the default works for me.

[@Jims-Garage]

@@lmontei glad to hear, thanks 👍

[@sebasdt2103]

Can't really grab the use of SDN in a homelab where you already have a router like opnsense. Is it like for more granular network control?

[@Jims-Garage]

It's likely of limited use for most within a homelab environment, however, it does restrict inbound traffic. It's useful in the cloud if you only have limited IPs available.

[@FTLN]

@@Jims-Garage Its usefull when you are using ISP Router box, and you need some parts of your network sealed off from your main lan :)

[@ltonchis1245]

I'm still trying to figure out the purpose on this, so correct me if I'm wrong but this is a way to create a proxmox network for your nested opnsense network firewall to live in?

[@fahmi8999]

Hi, could you provide step for PowerDNS setup in Proxmox?

[@ripaire]

Amazing video 🎉 thanks for the great explanation, good work 🫂❤️

[@Jims-Garage]

Thanks 👍

[@ilyavakhtinskikh3513]

you need insall dnsmasq even if you doing fresh installation

[@MarkConstable]

I tried this on a 3 node cluster and only one node works, the other two have a yellow triangle exclamation mark icon. You didn't show us what happens on your other proxmox-asus node so I'm not sure if I messed up my multiple attempts or your guide only works for a single node?

[@Jims-Garage]

Simple SDN only works on a single node. You need vxlan for multi nodes.

[@MarkConstable]

@@Jims-Garage Right, thanks for the heads-up. So how about an evpn + vxlan SDN multi-node tutorial at some point 🙂

[@Jims-Garage]

@@MarkConstable yes, it's on the list. Just need to work it all out first 😂

[@MarkConstable]

@@Jims-Garage Heh, "work it all out first". I've spent the last week googling and searching for any step by step guide that applies to a Proxmox cluster and... nothing. It's too complex to just throw stuff and see what sticks. My first few attempts just destroyed my working cluster, so I need a "simple" guide to follow. Like anything, once I have a working example in front of me, I'll be able to tweak it as needed and apply it elsewhere. Perhaps consider using a virtual 3-node VM cluster within Proxmox so the rest of us can follow along without destroying a real cluster.

[@XFallenOlympusX]

It wont let me apply the Network Device to the VM. This is the error VM 301 qmp command 'netdev_add' failed - network script /var/lib/qemu-server/pve-bridge-hotplug failed with status 512 I have a linux bond using my 4 ethernet ports on the server for link aggrogation. Is that the problem

[@zyghom]

so wait, if you created SDN, what is the speed of it? does it go through the physical NIC if it has to go out of Proxmox machine? what if not outside - then what is the speed?

[@Jims-Garage]

The speed is probably more of a software/CPU issue as no interface is used (eg. NICs). It only goes through the NIC for external access.

[@zyghom]

@@Jims-Garage I thought so, thx for confirmation. But, what is then the difference between SDN and using another vmbrX with separate subnet? we can have many vmbr as well as many SDN, right?

[@beprivatecdblind7831]

I had to use MTU 1490 to get SDN to work, not sure what is going on

[@Sulaimanzai]

Hello Jim again, I created a simple zone named `purple` within that zone i have two vnets (DMZ(192.168.1.0/24), SecOps(192.168.10.0/24)). I have a VM with IP 192.168.1.100 which can ping another VM in SecOps vnet 192.168.10.100 and also network where my proxmox is 192.168.16.0/24 but I cant ping 192.168.1.100 or 192.168.10.100 from 192.168.16.20? any hint or help is appreciated. thanks

[@Sulaimanzai]

thanks, just had to add route "sudo ip route add 192.168.10.0/24 via 192.168.16.20" , 192.168.16.20 is the proxmox IP.

[@hujosh8693]

not working at all. vms get ips but cannot ping each other.

[@Jims-Garage]

Check your firewall rules on the VMs. If you follow the guide they're on the same subnet so it's the only possibility. What OS are you using?

[@hujosh8693]

@@Jims-Garageubuntu 22.04 lxc containers on different nodes. on same node ping is reachable. on different nodes ping not reachable.

[@hujosh8693]

@@Jims-Garage using pve 8.1 and assigning sdn vmnet to difference lxc containers on different nodes. not reachable

[@Jims-Garage]

@@hujosh8693 that's because simple SDN is node specific. You need vxlan for internode (I'm still to cover this)

[@hujosh8693]

@@Jims-Garagethank you.

[@pusheen483]

Does it not support IPv6? I’m not seeing it at all in the GUI here.