How to Setup a Proxmox Cluster

[@TechTutorialsDavidMcKone]

If you want to learn more about Proxmox VE, this series will help you out kzbin.info/www/bejne/qXm6ioiqZbtgmZo

[@subpixel2234]

Thank you for the detailed discussion of CPU requirements and network interface design!

[@TechTutorialsDavidMcKone]

Good to know the video was helpful I like to know the why not just the how So that's why my videos are so detailed

[@subpixel2234]

Agreed! Why, is critical. In order to solve more complex problems one must understand the why (root cause) of their current problem. I see too many people get lucky with guess-and-check, their problem solving ability maxed out quickly.

[@ltonchis1245]

Thanks for the videos Mr. Open Source Matrix Architect 😁

[@TechTutorialsDavidMcKone]

Now that's a catchy title

[@bhupindersingh3880]

Precise and Clear and informative. A Short Quesiton 1. Can a cluster running on three nodes support a Ceph Cluster with its independent from the cluster machines and be attached to the Cluster for shared Block storage and File Store ? Will it hinder the cluster performance ? 2. What NIC card speed in needed for the Cluster. -- Is the cluster using this NIC for corsync activity and I assume this is not bandwidth hungry? Thanks in advance for the guidance

[@TechTutorialsDavidMcKone]

I haven't touched on Ceph yet for PVE but it is on my to do list Corosync just depends on latency so the cluster NIC is best put in its own subnet. As long as the physical interface isn't overwhelmed by bandwidth, it could be set up as a sub-interface

[@bhupindersingh3880]

@@TechTutorialsDavidMcKone a followup: What I understand that the Corsync should be in separate VLAN with its own NIC. Is my understand correct?

[@TechTutorialsDavidMcKone]

@@bhupindersingh3880 Ideally a separate interface as well but just VLAN is fine as long as the physical interface doesn't have too much traffic

[@bhupindersingh3880]

@@TechTutorialsDavidMcKone So If I dedicate a physical interface that is 1GB for Mgmt, another physical interface 1 GB for Corsync, Separate Physical interface for VMs, Separate interface for Storage. The Corsync will be a VLAN dedicated to it. Does VLANs require any special precaution. Shall appreciate your valuable thoughts on this approach.

[@TechTutorialsDavidMcKone]

@@bhupindersingh3880 You would still need separate VLANs as a computer can't have more than one interface in the same subnet If you use physical interfaces though then you only need to configure them on the physical switch ports

[@pavlovsky0]

I've used 3 (actually 4) dell optiplex's for a proxmox cluster. the 4th box I setup as a dedicated proxmox backup server (pbs).

[@TechTutorialsDavidMcKone]

I haven't used their backup server How does it compare to Proxmox VE storing backups on a NAS shared folder?

[@pavlovsky0]

@@TechTutorialsDavidMcKone yeah, so it can obviously do that. But, the real beauty of pbs is you setup a backup datastore and you can use zfs just like on your TrueNAS. So, I setup a zfs backupstore and then you register the bps to your proxmox cluster and it shows up just like other shared storage. Bps can replicate to other bps's and it has a hierarchy of data/organization. You can enable pruning can deduplication. You can install a proxmox_backup_client on your other linux boxes to backup them up too. Lastly, it has tape drive support to backup that datastore to tape (please note bps backs up to local datastore and then you backup to tape). ElectronicsWizardry is a good YT channel to follow, he's a proxmox enthusiest I follow.

[@TechTutorialsDavidMcKone]

Thanks for the update as that sounds very useful The option for Linux servers sounds great

[@pandukautsar1375]

nice vid, have you ever get the ceph cluster with existing cluster? and is it better than if we had 6 node in one cluster with 3 nodes for ceph? i think i need your smart advice, regards

[@TechTutorialsDavidMcKone]

I'm happy using shored storage for now as it's easier to backup, but at some point I might start using ceph As for cluster size though, the more you have the more likely something will go wrong So unless you have a really big network you would have 3 nodes with lots of CPU and memory After that you scale to 5 You mentioned 6, but clusters should have an odd number of nodes when all are working to avoid the risk of a voting tie

[@AlonsoVPR]

Thanks for the explanation, I just have a question. What if I want to connect my 3 nodes directly instead of using a switch for the Cluster HA network? Should I use 2 Cluster Networks on each node? Thanks in advance!

[@TechTutorialsDavidMcKone]

It's not something I've tried so I don't know what the outcome would be For a lab, I just used a cheap retail switch to connect the servers together

[@AlonsoVPR]

@@TechTutorialsDavidMcKone Understood, my goal is to use minimal gear on my cluster so connecting the 3 nodes directly through their nics (I know that it's possible but don't know how) I save space on that switch. I guess I'll have to get the gear and my hands dirty first :P

[@jimscomments]

Mr. McKone I have two NAS servers running TrueNAS and I rsync the files from NAS 1 to NAS 2 as my raid backup. My first Node has a NFS share to NAS 1 set up in preparation to cluster the three PVE computers. I will eventually set up HA as well. The only single point of failure is my NFS share on NAS 1. Is there any way I can have the cluster connect to a redundant NFS share on NAS 2 or is just creating snapshos of the NFS share on NAS 1 and rsync it to NAS 2? All three videos you made covering the topics discussed in my first paragraph sure were helpful.

[@TechTutorialsDavidMcKone]

Nothing springs to mind in terms of automation if NAS 1 fails, or at least not within the scope of a typical hypervisor The HA option for Proxmox only covers hardware failure of a node It does add a bit of complexity but you could have an NFS share to both NAS devices As long as NAS 1 syncs to NAS 2 then if NAS 1 goes down, you should in theory be able to manually spin up a VM over the other NFS share without too long an outage The risk though is data corruption and so you have to fall back to restoring from backup You could reduce the risk by splitting VMs over both NFS shares; Business do that sort of thing for "cost justification" purposes But it then means NAS 1 would need to sync to NAS 2 and vice versa and isn't ideal if say NAS 1 has a 10Gb NIC but NAS 2 has only 1 Gb

[@jimscomments]

Thanks for the thoughts. I plan on running my two TrueNAS servers as VM's on two nodes and look at PBS. I've heard that PBS is more sophisticated than PVE's backup system so maybe it has a better solution with snapshots that's an easier solution to get things restored faster.

[@TechTutorialsDavidMcKone]

@@jimscomments PBS would make sense as it will backup a host as well but it needs a dedicated host and storage solution PVE does have its own backup system to schedule regular backups for VMs and it also has retention/version options which is very useful

[@jimscomments]

I've read Proxmox's opinion on not running PBS as a VM and I see there point. I have a HP Thin Client I was planning on using as the dedicated PBS system but I'm wondering since I'm going to be running HA I could use my two TrueNAS systems for the storage. Having PBS as a HA VM seems to overcome the negative aspects Proxmox discusses. Thoughts?

[@TechTutorialsDavidMcKone]

@@jimscomments I think the reasoning against it is because it creates a dependency; The backup system relies on the very thing you want to be able to restore If PVE breaks and you can't access PBS as a result, you'd have to find another way to fix PVE or rebuild it from scratch Even if you have two nodes, a configuration change or software upgrade for instance could break the cluster So it's better to keep PBS and PVE independent of each other

[@jimscomments]

Mr. McKone I'm aware that when migrating VMs in a cluster the computers in a cluster should all have the same processors for migration. I have read that the third computer can be set up to just be the third vote but not have VMs migrated to them. I have a HP Thin Client I was going to use as the third computer in the cluster but it has an AMD CPU. Since it's not going to have VMs migrated to it, am I thinking correctly that it will function in the cluster as the third voting system? Very nice videos that you create. They are very helpful.

[@TechTutorialsDavidMcKone]

Yes, you can have a 3rd computer in a PVE cluster that's just doing the voting I use a Raspberry Pi as a QDevice because I can run all of my VMs on one server, but I have a second one to cover against patching and hardware failure A third server with that compute power would be a total waste of money I did a video on this if you're interested in it, but you just need something that can run Debian and the corosync software kzbin.info/www/bejne/oHLPq3WjaZWipMU

[@jimscomments]

@@TechTutorialsDavidMcKone Thanks for the video. The text link I found was a little vague. Looking forward to viewing it tonight.

[@jimscomments]

In case your looking for ideas on additional videos here is one to think about producing. If I understand Proxmox correctly it will send notifications internally but not to an external email address. Proxmox has a forum document but it uses a gmail account with special passcodes. Plus it looks like to get SMART notifications requires a totally separate setup. I find your videos your videos organized and presented very well and this topic might be of interest to other people as well.

[@a3-82]

can we joint cluster with different version of proxmox ? i have proxmox version 6 and 7. can I joint them ? thanks before

[@TechTutorialsDavidMcKone]

It's better to upgrade systems first so the version is the same on all nodes Besides the changes in Proxmox itself, the underlying operating system is Debian and there can be breaking changes in there as well

[@pavlovsky0]

Did you plan to have two 100 nodes when you did the cluster join to demonstrate a common failure?

[@pavlovsky0]

you can't have two 100's. if you had a 100 and a 101 it would have probably worked.

[@TechTutorialsDavidMcKone]

Yes I just wanted to demonstrate common problems

[@TechTutorialsDavidMcKone]

Well spotted

[@virtualhermit]

TY

[@TechTutorialsDavidMcKone]

Thanks for the feedback Good to know the video was helpful

[@tinfoil-hat-qc1rk]

Can I use this Cluster without an Storage Network Interface?

[@TechTutorialsDavidMcKone]

You can run a cluster on a single interface But ideally it's best to split the traffic over different interfaces

[@zyghom]

I am still thinking: only HA will support failover, while cluster not really. So, apart from common management point: what is the other value added to have a cluster?

[@TechTutorialsDavidMcKone]

Well the main reasons I'd want a cluster is for things like software maintenance and to cover against hardware failure If you have shared storage you can migrate your VMs off a node, upgrade its OS then rinse and repeat, all without interruption Now if all you have is one node, then if anything in that server fails, it's probably game over And although HA on some hypervisors can restart a VM on the same node that it crashed on, I feel it's more for automated recovery in the event of a hardware failure Rather than having an outage for maybe hours, HA will restart a VM on another node in maybe under a minute It can do a lot of other useful things mind, but I think those are more for enterprise networks

[@zyghom]

@@TechTutorialsDavidMcKone ok, that is clear: I build 2x proxmox on 2 machines, use for everything but Proxmox OS the shared storage from TrueNAS and then, I can take 1 PVE down almost without interruption. That is clear. So 2x Proxmox it is ;-) I have 1 on AMD and second on Intel - lets see how that works. I am yet to try it. So far I tried only... PVE being the test instance in... PVE (just to check how that works, and yes, it works). Thank you David.

[@TechTutorialsDavidMcKone]

@@zyghom Mixed CPUs can cause problems if you want to do live migrations but at least it gives you hardware redundancy

[@zyghom]

@@TechTutorialsDavidMcKone I know, but I am not running surgery room - just home automation - I am ok with 30s break - live it does not have to be ;-)

[@ierosgr]

Now that I ve watched it I have some recommendations/observations / questions to make. -The part where you are talking about requirements and one of them is the time. What if, the nodes are on different sites in areas with different local time? -Since Clustering could benefit not only enterprise companies but small/medium as well where the budget is lower for sure (my area of expertise if you ask), that means the quorum pc could be a nuc/rasberry or whatever else low powered machine and even though it will be used just for the votes, a video needs to be made showing what happens when one of the other 2 main servers goes down for good. How would you clear the second's info from the main server (the one on which the cluster created) or is it the same if the server upon which the cluster created to go down like if any other joined node had to be shuttled down for good? -Also a resonable question and something you could show is, during creation of the cluster certain conf files have been created and updated with new lines like corosync key to /etc/corosync/authkey corosync config to /etc/pve/corosync.conf and probably others I am not aware of right now Did you check if those lines are also available to the other 2 servers as well? -Even though the next logical step would be a video for HA, my recommendation would be more info about troubleshooting the cluster. Point is not only being able to create it (you already mentioned this is easy) but being able to maintain and troubleshoot it as well.

[@TechTutorialsDavidMcKone]

Those are good points and over time I will create more videos covering Proxmox I don't know of any cluster deployments spread over different time zones as I usually work with large corporations So typically there are two Data Centres in a country/region each running its own cluster There's no requirement to migrate a virtual machine from one DC to another Smaller companies I originally worked with operated a Disaster Recovery strategy They had one active Data Centre and in the event of a site outage would restore data from backup at another site Having said that, in theory the cluster servers could be managed under the same timezone The VMs though would be better synched to a separate NTP server so that there own time is accurate But if these servers are so far apart, latency would likely be a concern

[@ernestoditerribile]

I have 4 x3650 M5 servers all with 2x Xeon E5-2620 v4 processors, Nvidia Quadro K2000 GPU’s and 192 Gb of 2166 DDR4 RAM. Can I run 1 Kali instance to use all processor cores and GPU’s for bruteforcing clients in my network.

[@TechTutorialsDavidMcKone]

Not that I'm aware of because a virtual machine runs on a single node

[@ernestoditerribile]

@@TechTutorialsDavidMcKone would love to have virtual machines running in sync on multiple nodes.

[@ierosgr]

Like before even check it because it would be thorough

[@TechTutorialsDavidMcKone]

Can you clarify what what you mean?

[@ierosgr]

@@TechTutorialsDavidMcKone hahaha I just read it and seems weird indeed. With Like meant I pressed the like button before even watch the video because I knew the description of the procedure of custering would be thorough like mentioning all the necessary steps and not just a quick review....it meant to be a positive comment but as it seems I failed :)

[@TechTutorialsDavidMcKone]

Now I understand 😀 I thought I'd messed up the video so I checked it again and it seemed fine