Proxmox Cluster QDevice Raspberry Pi

[@TechTutorialsDavidMcKone]

If you want to learn more about Proxmox VE, this series will help you out kzbin.info/www/bejne/qXm6ioiqZbtgmZo

[@kymnyth]

This was very helpful. I have come back to this video a number of times. Thank you!

[@TechTutorialsDavidMcKone]

Good to hear the video was helpful. Thanks for the feedback

[@Farang_Lifestyles]

very usefull tutorial David,,, I have been running with two bare-metal Proxmox VE using it to migrate VM back and forth when doing maintenance, etc... was wondering why sometimes things would not function .. you explanation of the quorum and votes makes sense and having a RasPi to be the 3rd party to vote is a perfect for a low energy solution

[@TechTutorialsDavidMcKone]

Good to know the video was helpful I only need two servers myself, so it was great to find you could use a Pi as a qdevice

[@discovery_af]

Thanks for the help! I have a mini pc along with a server build that I clustered. Didn't realize that quorum was a thing until someone pointed it out. Thankfully this video exists.

[@TechTutorialsDavidMcKone]

It's definitely useful in a small network So good to know the video was helpful

[@colinkhalid1]

many thanks, just got this working. instructions were spot on👍

[@TechTutorialsDavidMcKone]

Good to hear I found it very useful for saving on costs and energy

[@905jay]

great content as usual - thanks for posting this it's exactly what I needed :)

[@TechTutorialsDavidMcKone]

Good to know it was useful I found it's a very useful way to save money

[@jsg9582]

Thank you for the clear explanation and instructions! Bookmarked for future reference :)

[@TechTutorialsDavidMcKone]

Glad to hear the video was helpful

[@sinisterpisces]

This is an awesome video. I was completely unaware that I could set up a third device to provide a quorum for my cluster. I was about to go through the trouble of setting up a whole third Proxmox node. :) You saved me a complex and apparently unnecessary adventure. I did have a few questions: 1) Are there any stability issues with installing on top of the full Raspberry Pi OS with GUI (as opposed to the minimal/no GUI version)? 2) I noticed that it copied SSH keys when you ran through the setup, but I didn't see you set those up. Does that happen automatically, or did you configure your Proxmox node to use SSH keys in another video? 3) Would you feel comfortable running any other services on a Pi 4b 8GB acting as your Qdevice? (I'm assuming Corosync doesn't use a ton of CPU.) Thanks again!

[@TechTutorialsDavidMcKone]

Good to know the video was helpful The desktop OS might be stable enough, as long as the Pi has the resources, but I can't be certain as I no longer use it You don't have to create any keys yourself for this setup process as it's copying over a public key for the root user and I didn't supply that If you install anything else on the qdevice you could introduce a security risk; Any other software probably means other devices will need access to the Pi and that could then be used as a possible jump point to a hypervisor Having extra software means a computer is more likely to have a bug and crash or misbehave And you could run into the problem of having too much network traffic on the NIC, or in the network it's connected to, which results in the corosync traffic being dropped and that leads to clustering problems I just had a Pi lying around and as it's low power I'm quite happy to let it just be a qdevice This way it sits in an isolated network and aside from management access, only the hypervisors can talk to it and vice versa

[@radekw-gh6ln]

Thank for this video. It helps a lot. However for me this rule of quorum it's strange. It means you can lost only one device at the time - one of proxmox clusters and qdevice. But what to do when you lost both proxmox and qdevice at the same time? Then cluster is unfunctional. Maybe runing qdevice on node itself is some solution for that scenario?

[@TechTutorialsDavidMcKone]

When it comes to designing networks and clusters, even very few Enterprise companies plan for a double failure When a computer fails, it's very rare for a second to fail at the same time, so the additional cost isn't worth it Yes, you get mechanical failures of hard drives for instance or fans, but usually it's a software problem that causes a computer to fall over Motherboard failures do happen, but they're rare With Proxmox VE, even if you're down to one node, the virtual machines on it would carry on working So as long as you're running redundant VMs in your cluster, the impact is maybe a loss of performance, or you can't make configuration changes to an application Now you can't make changes to the cluster either, unless you bypass the rules, but it's not advisable Instead, it's better to recover the cluster Running a qdevice on a hypervisor wouldn't be advisable because then you're at greater risk of a double failure The qdevice is only really intended for a cluster with two nodes So, if one of those nodes is running the qdevice, then when the node crashes it would take the qdevice with it and all you'd have left is one node It's also very easy to reboot a node by mistake, which would result in the same issue Because the cluster can't achieve quorum, the node can't use HA to spin the qdevice back up, unless again you bypass the rules And you can only have one active qdevice as well, so you can't have one physical one and a virtual sitting on standby

[@philsson]

Thank you for this great video. A question for you. I have added a Qdevice to my cluster and it shows up as "Qdevice" under membership information. Is there a way to give these a better name? I would like to name them after the name of the devices they are on. Also I cannot add more than one node as it says that "QDevice already configured" Thanks in advance

[@TechTutorialsDavidMcKone]

I haven't tried it but there are advanced settings and I think this one is what you're looking for; votequorum_device_name www.systutorials.com/docs/linux/man/8-corosync-qdevice/ There's only meant to be one qdevice in a cluster The idea is, you only need two nodes for redundancy and a 3rd server would be a waste of money so you add a qdevice to provide the needed 3rd vote Once you go beyond 3 servers you would be expected to use just servers anyway

[@philsson]

@@TechTutorialsDavidMcKone Ah okay, I was thinking that more devices would make it more solid. I don't have network redundancy and thought that breaking my network would cause one node to "take over" even though both servers are running fine, just separated. My idea was that to prevent this I could have multiple QDevices and increase the quorum so I could be sure I had voters from key parts of the network present. Maybe this sounds like I would be using it wrong?

[@TechTutorialsDavidMcKone]

@@philsson The goal is to prevent a cluster break in the first place by giving nodes network, disk and power redundancy But you can run into software bugs anyway If you did end up with a "split brain" though you don't make any changes and try and get the servers talking again ASAP Things fall apart when nodes have different configurations leading to a disagreement that they cannot resolve

[@8bitkid408]

Great video. Can I run pihole and QDevice on a PizeroW?

[@TechTutorialsDavidMcKone]

I wouldn't recommend it as it uses Wi-Fi The cluster needs fast i.e. low latency responses so you need a wired computer

[@richardfrancis9620]

@@TechTutorialsDavidMcKone so if i was to have an usb to ethernet adapter a pi zero would work?

[@TechTutorialsDavidMcKone]

@@richardfrancis9620 It could be worth trying in a lab But I don't know if the software will be available as the OS is different or if the hardware could keep up

[@ShoruKen]

since it is just a home lab and not a live/production environment, the qdevice allows me to take down a server in the cluster without killing the whole cluster!

[@TechTutorialsDavidMcKone]

I've found it to be very useful A 3rd server would just be a voting machine in a home/lab environment so I'd rather have a Pi do that

[@GaryInnerarity]

Can we have multiple qdevices - to act as redundancy to the 1 qdevice? Knowing raspberry pi's, it would be nice to have an extra as a backup.

[@TechTutorialsDavidMcKone]

Normally you don't have extra redundancy in IT, you design to handle a single failure Proxmox's recommendation is to only use a qdevice when you have 2 nodes It makes sense for a small cluster because if a node fails, the qdevice allows changes and so on to still go ahead And if the qdevice fails, well you still have your two nodes Having 2 qdevices would also result in a cluster of 4 voting machines and you need an odd number in normal conditions to avoid a tie breaker

[@bangan9570]

First, Thanks! However, i got errors on starting the qdevice, and spent a couple of hours researching. I could change user to root and get it to run, but that is bad practice from what i learned. :D running '''chown -R coroqnetd:coroqnetd /etc/corosync/qnetd /var/run/corosync-qnetd''' solved my problem, as the owners of both directories was root. Is it something missing in the corosync-qnetd package, or did i do something the wrong way?

[@TechTutorialsDavidMcKone]

Out of curiosity what did you install this on as I used a Pi and it's still running the older Bullseye code? The software was simply installed as shown and it took care of the folders and permissions The user in the service is showing as coroqnetd so what you've run into seems odd Not sure if something has changed, but for me, the ownership for the folder /etc/corosync/qnetd/nssdb/ is root:coroqnetd So the user account has access to it via the group Ownership for the parent folders is root:root I'm seeing a symbolic link for /var/run pointing to /run I do have a folder /run/corosync-qnetd and the ownership for that is coroqnetd:coroqnetd Makes me think there may have been a bug in the installation package It makes no sense to run this software as coroqnetd but not allow access for the user

[@bangan9570]

@@TechTutorialsDavidMcKone I had a fresh install of latest version Raspberry Pi OS 64bit. For testing, i just tried with Raspberry Pi OS (Legacy, 64-bit) (Bullseye) and it worked as soon as i readded the qdevice to proxmox. I am too new to this to know if i made a mistake or if there are differences. :) Again, huge thanks for making these video tutorials for us 'newbies'. Next up I'll try to get mail notifications (gmail) working.

[@squareeyesproducer]

Tried to install on a docker container on the Pi but even with 5405 forwarded the remote root execution fails. Seems part of the script provisions things on the Pi instead of the container (maybe?). Then it occurred to me. Why not install Proxmox on the Pi and have it natively participate in the quorum? Or even have a CT on Proxmox on the Pi that runs barebones corosync?

[@TechTutorialsDavidMcKone]

I hadn't considered a container as the Pi I had was just gathering dust, so I just put it to use as a qdevice Presumably there must be something it needs access to and maybe AppArmor is blocking access to it? I'm aware of a version of Proxmox for the Pi, but it's not official and I think it's still on v7, using 32-bit Debian Mind you, the devs of Pimox suggest you could mix a Pi running Pimox with x86 servers running Proxmox So that might be a better idea for a lab as in theory you'd have 3 nodes in a mixed cluster You wouldn't need a qdevice, but you'd be limited to 32-bit VMs on the Pi I wouldn't run a qdevice as a container on a node though, or even a VM The qdevice is only recommended when you have only 2 nodes and if the node running it fails, the qdevice is lost and HA won't work

[@Neo8019]

Thanks, great video. I was getting "Host key verification failed" when trying to setup the QDevice on the first PVE. After some search I found that I had to run the following command on the PVE ( I run it on both) pvecm updatecerts

[@TechTutorialsDavidMcKone]

I've never ran into that issue myself, but good to know Thanks for sharing

[@jacksukerman5150]

How do I check the cluster status on the pi?

[@TechTutorialsDavidMcKone]

I'm not seeing anything you can check on the pi itself But on a PVE node you can run this command pvecm status As long as things are setup and working it should mention the qdevice

[@fatal3713]

I set a password and user in Pi imager before installing into Pi Zero 2. I went and changed the root password as you demonstrated. When I go to setup device it says that root password is wrong.

[@TechTutorialsDavidMcKone]

That's puzzling Other than mistyping a character, and it is possible if you type too fast for instance, I wonder if the issue is keymapping If the password contains symbols that can cause a problem if the keyboard map doesn't match the keyboard layout If that's the case, you could try pasting in the password to see if that resolves the issue or experiment with different keys to find the correct symbol Unfortunately you can't change the password without knowing it and then you have to reinstall the OS If you have access to the CLI you can type things out as well to what symbol you get back versus what the keyboard shows

[@fatal3713]

@@TechTutorialsDavidMcKone thanks for responding, will try your suggestions

[@fatal3713]

@@TechTutorialsDavidMcKone I finally got it to work, don't ask me how because I don't remember. Thank you