Setting Up My First Proxmox High Availability Cluster

[@chromerims]

👍This is the best video of the year on all of KZbin. Cannot thank you enough!

[@DBTechYT]

Wow, thanks! I really appreciate your comment!!! Glad the video was helpful :)

[@chromerims]

@@DBTechYT I am keeping an eye out for the 3 Steam Decks ceph cluster that 45 Drives channel will be releasing later this week.

[@DBTechYT]

That sounds amazing!!

[@chromerims]

@@DBTechYT It nicely bookends 2022, somewhat the year of all things Steam Deck. Their vid drops on Thursday I think. I gotta watch your latest video migrating to zfs tonight when I have time. Kindest regards.

[@chrisumali9841]

Thanks for the demo and info, have a great day

[@DBTechYT]

Thanks for watching!!

[@donaldwilliams6821]

Nice video! If you haven't already setup Proxmox Backup Server, do so! It's amazing. Does the obvious thing of backing up VMs, and containiners but also does dedup, replication to another backup server and tape support. But I also like that it supports backing up bare metal Linux PCs too.

[@DBTechYT]

I've definitely got a backup server set up and configured to back up once per day and verify the backups 🙂

[@dominick253]

Yes. Really great setup. I made a PBS VM on my windows TV PC. It's always running anyways so perfect for a backup server.

[@brainamess2979]

Thank you so much for this video :) Lots of help. Subscribed!

[@dominick253]

Honestly proxmox makes it so easy! Only problem I ran into was using just two nodes. Ended up giving my main node two votes and everything worked fine. Getting a pi 5 to use as the third node.

[@connor7439]

I didn't think Proxmox supported ARM architecture? Only x86?

[@the_latinist]

@@connor7439 But any Linux machine can run a QDevice - a small service that communicates with each node via SSH and whose sole purpose is to break a tie to ensure a quorum.

[@jumpman1823]

Looking good with the cap!

[@DBTechYT]

Thanks!! I got it from www.lamoodbighats.com/ Had to find a place that sells decent looking caps for big heads like mine lol

[@ahyi9350]

This is really interesting and you made me wanna buy more hardware just to play this 😂

[@BrianThomas]

yeah, same here. I don't blame him though. I've had issues for a while when it comes to buying too much tech. haha.

[@HSVEOCT]

Great Video Loved It. Thanks for the inspiration.

[@DBTechYT]

Thanks for watching!

[@CJ-vg4tg]

Great explanation - been running 3 nodes on ceph but am planning to move over to zfs. Thanks

[@martinbower2915]

Great video, proxmox is a really nice setup for a homelab. I like the lightweight containers, but I also run kubernetes and traeffic

[@donaldwilliams6821]

You can run LXC/LXD/Docker containers and K3B on Proxmox. No reason I can think of that traeffic wouldn't work also. Proxmox is much more capable than just for "homelabs"

[@drbyte2009]

Great video again David, i love proxmox!

[@DBTechYT]

Me too! Thank you so much!!

[@JuanCintron]

Excellent tutorials and walkthrough! thank you for sharing.... I watched several of your videos and do have a questions. I see you talked during the wrap up about CEPH and watch your other videos and also see you move away from CEPH due to the overhead (similar to your reason for leaving OMV and moving to ProxMox). Have your found a sweet-spot on storage implementation for proxmox? balanced resource usage and failover time in HA? thank you again!!!

[@dimitristsoutsouras2712]

Hi. Nice video. Seems that after 8:57 where you create the storage for each node (with the same name as you ve mentioned) you ve forgot to show something, else I can t explain the different behavior in my cluster. What I mean by that is, if you keep checked the box on the first node only and not the others, then whichever node you re on gui you ll see only the Storage of the first node. All the others will be like they have no storage (even from each node's gui, it can t be seen). The same goes if you go individually to node 2 or 3 and try to create a VM. At the tab where you need to choose storage space for the VM you can only choose local. No Storage. So something you missed to show I guess. PS By the way you didn t check thin provision option for the Storage at Datacenter->storage->Storage->properties New edit: The only way for the storage to be seen to other nodes too is go to Datacenter->storage->Storage->properties and inside that window list down at option Nodes all 3 nodes and check them. Then automatically that storage is visible to all other nodes too. Even though, what would be the purpose for all 3 nodes to see only the first one's storage and not theirs. In case of first node failure his Storage (which the other two see) would be absent so where is the HA and replication to this scenario?

[@DBTechYT]

The way I set things up works. I don't know what to tell you. I followed multiple other tutorials for my setup and it has been up since I made this video.

[@acselman995]

This is the MOST CRITICAL STEP that was completely missed in the video! If you don't do what dimitri says above, REPLICATION will NOT WORK! I love DBTech's vids. This comment saved me after hours of messing around. Thank you. This is the part to pay attention to: -> "The only way for the storage to be seen to other nodes too is go to Datacenter->storage->{name of your storage ie: Storage}->EDIT and inside that window list down at option Nodes all 3 nodes and check them."

[@craiguk1973]

I ran into the same issue (storage not showing on other nodes until updating datacenter-storage area to add other hosts). Many thanks for sharing! Great video btw

[@andrewjohnston359]

If all 3 nodes are pointed to the first zfs pool, are they not just using this storage? Where does it show defining in replication to use storage on node 2 or node 3? Or is the act of labelling it the same tricking it into using the same namespace so it just doesn't know any different? Also if the author of the video states something is 'very important' but doesn't explain why (turns out it's because he doesn't know), then he shouldn't be making the tutorial IMO

[@GiovanniAcosta]

@@andrewjohnston359 Kinda harsh. At least dbtech's video's are opening up insights and the opportunity to investigate more. Though, it would be nice to get a definitive answer to your questions, as I also have the same. Please keep making the videos dbtech, we're all learning here.

[@rklauco]

Great video, thanks! It seems I cannot replicate unless I have ZFS on both ends... But still very usable, thanks!

[@ca8746]

Thanks for this video!

[@PrimalDashes]

Incredibly well done, as always, and I see the where we're headed 😀. I noticed you're using Omada, I just started with the line. Have the router and waiting to pick up the AP to replace the FiOS router I'm using as an AP for now

[@DBTechYT]

Thanks!! Loving the Omada setup quite a bit! Just got it and am still learning, but it's a nice learning curve :)

[@PrimalDashes]

@@DBTechYT are you using the Omada router too? Just curious if you're running a separate firewall in front of it or letting it do the firewall/load balancing on its own. I'm in the same boat of just getting into the setup

[@DBTechYT]

I am using the router, too. I'm letting it handle everything and haven't had any issues yet :)

[@briansmith8235]

Always interested Dave.

[@posalab]

Hi, great video... just a suggestion: in a set like yours I will try definetively with a true network storage system like gluster. Is not ceph but works very well with proxmox and with HA. The migration from a host to another is blazing fast and recover VM from a failed node faster than a replica solution like this. It's a viable solution between ceph that require nore resources and little cluster like this. In you case you can allocate the second ethernet to handle gluster.

[@damiendye6623]

Ceph doesn't use much resources you can set the ram amount but you need 10gbit ethernet

[@kklip]

Hi David, any idea why it throws the error "no replicatable volumes found (500)" when trying to add a VM to replication? I'm using NFS storage in my cluster if it helps. Thanks

[@Whiskey7BackRoads]

Do you know if this works via remote VPN or tunnel? I know it would be slower but would it work? I have 3 locations with internet. Was thinking of Tailscale tunnel on all three but have them in cluster and use HA?

[@mauriciovillatoro9361]

Wow that was fast.

[@tdragon87]

Just have 1 proxmox server, so haven't try this out yet, but how would one go about doing this with ISCSI?

[@katsumoto_ishida]

hi i have a qustion please do you know how to connect or join two proxomox locally seperated like one proxomox in a city and the other is in an other city i don't know how t connect them to create a cluster ?

[@yuriw777]

Thx for the video! Can I achieve what you describe here regarding Storage on zfs with only one disk? TIA

[@DBTechYT]

You can definitely set up proxmox zfs storage with just one drive

[@agalbulescu]

thank you so much for this video, it really helped me out setting up a three node cluster. I have one question though: let’s say that the first (master) node goes down and the lxc/vm’s move to the other nodes. you mentioned that after the first node comes back online the lxc/vm does not automatically move back, but continue to run on node 2 or 3. but how does replication work from that point on? asking as we set-up the replication from node 1 > node 2 and node 1 > node 3… if we continue to add data to the lxc will that only be available on the current node or should we set up replication for example from node 2 > node 1 / node 3? thanks so much for a great tutorial again and have a great weekend!

[@DBTechYT]

The replication should keep working and the nodes may even load-balance themselves as necessary and move containers around to make things more efficient.

[@agalbulescu]

@@DBTechYT probably the fastest response ever! thank you so much! you're the best!

[@markgordon4524]

Excellent video. I really like your pace. I did not have to fast forward a all! If you have an application like postgres db running on your server how does that work when moving to a new server? So if a disk failed you would be at most 15 minutes old?

[@DBTechYT]

Thanks! Moving to a new server will result in losing any data that isn't backed up. So if you plan on moving to a new server, I would spin up the new server, then run a backup of the database on the old server, then restore it to the new server and shut down the old server. Shouldn't have much data loss at all that way

[@BrianThomas]

This is a great video. Could you show us how to configure a CEPH cluster? I've been looking into it and I haven't been able to find a good tutorial at all.

[@DBTechYT]

Thanks! I've actually talked to a few people who aren't keen on CEPH storage and have recommended using NFS storage to help with keeping overhead and complications to a minimum.

[@BrianThomas]

@@DBTechYT Interesting. I know you can't get to all the questions, so I thank you very much for your time in replying. I've been using NFS for a while now and I've heard so much good things about CEPH. So much so that it was worth doing a lab, but what you're saying makes sense. It's recommended that you have at least a 10G connection for the management due to the amount of chatter. That's a lot of overhead.

[@BP-777]

Can you do the same thing with the local storage if you set them up with ZFS at the initial install?

[@DBTechYT]

You should be able to buy you might have issues with high availability

[@ierosgr]

At 9:17 you mentioned the need to click add storage only on the first node and not the others but you didn t explain why is that

[@DBTechYT]

It's just the process for this setup

[@PaoloLaitempergher]

Nice clip, thank you! a question: must the cluster nodes have the same hardware configuration or can I have servers with different cpus, quantity of RAM?

[@DBTechYT]

You can mix and match

[@enderst81]

I tried doing this setup back on 6.x and failed, completely missed unchecking Add... ended up using Ceph. Looking forward to seeing how you're going to migrate from a single node to the cluster.

[@fbifido2]

please do this demo again but with the ceph storage !!!

[@DBTechYT]

Definitely going to look into ceph

[@kavehvahidinaeini8338]

Great Thanks Could you please do the same process with xcp-ng?

[@DBTechYT]

Maybe. I've never used it, but I may look into it :)

[@donaldwilliams6821]

FYI: There are many videos on clustering for XCP-NG The YT channel kzbin.info Has videos for just about all aspects of XCP. I used to run that but found Proxmox a better overall solution.

[@TehShawns]

@DBTechYT I have 3 nodes with a single internal SSD with Proxmox installed on. When I create a cluster, if i want to create a VM on node 2 there are no local drives showing. I'm at a loss at what to do here? I read it is not best practice to install proxmox to a usb thumb drive but you have and that would allow me to work around the physical limitation of the machines single SSD port and create the ZFS pool for HA, but is there any way to achieve what you done in this video using a single SATA SSD in each node with proxmox installed to it? Thank you!! Very helpful video!

[@jenniferw8963]

What did you send up doing? I am in a similar situation. Others hav used a USB thumb drive as well I see. Perhaps ifyou get a high quality USB thumb drive with decent read/write speeds maybe it'd fine? I learned from a Proxmox forum that ceph needs the entire internal nvme, and wont run off a partitition unforuantely.

[@alexitanguay]

same question here, what have you guys done?@@jenniferw8963

[@jimholloway1785]

I see your 3 Zimaboards on your server rack, Do you have a link to the shelves you have these Zimaboards on that are attached to your Server rack? I am looking for something to rack my Lenovo M900 tiny server and maybe more Lenovo servers in my future. This shelf looks like it is just the right size for the Zimaboards and I am looking for something to put my Lenovo M900 on...thanksjJim

[@DBTechYT]

So what I did was used some double-sided tape and stuck them to one of these: amzn.to/41bHlEm (affiliate link) Then I just put that assembly on a standard shelf like what you might find here: amzn.to/43oloUo (affiliate link) Some of the shelves I'm using in my rack came with the rack and the others came out of an EchoGear rack

[@jimholloway1785]

@@DBTechYT thank you very much. Jim

[@vijeyparthasarathy]

I am still little confused .. Need to know if the below scenario will work. I will install proxmox in my server .. install portainer and in each container inside portainer I will install wordpress .. will it work .. Please guide

[@DBTechYT]

Once you install Proxmox, you can create a container and install Portainer in the Proxmox container and then install WordPress that way if you want. That's basically how I have all of my Proxmox containers set up.

[@vijeyparthasarathy]

@@DBTechYT thanks I understand 👍 now

[@bluesquadron593]

Running similar setup with three HP elitedesks. Why not CEPH right from the get go?😊

[@DBTechYT]

Honestly because I haven't learned how to do it yet lol

[@bluesquadron593]

@@DBTechYT Dead simple. Will be good content for the next video 😜

[@crimsionCoder42]

So does a proxmox cluster increase availability and stability or reduce the time for task completion by separating out the tasks, like, let's say, machine learning and data science calculations? Or is it both tasks are separated across the node, and high availability is there by design?

[@DBTechYT]

Proxmox is just a hypervisor. You can configure it to do all kinds of things, but it doesn't do any of the stuff you've talked about without multiple nodes working together in some capacity

[@asraelvudogel]

Hello David, I don't know if this is the right way to place a video request, but I'll just do it now. Many people around me are currently switching from twitter to fediverse and I'm thinking about hosting my own instance of mastodon with Docker/NPM/Cloudflare, but unfortunately I haven't really found a good tutorial on the internet.

[@BrandonGroves84]

Awesome

[@DBTechYT]

Thank you so much!!!

[@cruxwire]

If something was missed in the video why dont DB Tech pull down the video and add the edit?

[@DBTechYT]

What are you talking about being missing? I can't think of anyone else who has had any issues with something being missed. Also, you can't just "add the edit" after a video has been uploaded. That's not how KZbin works.

[@quiquos]

BTW, just in case there are more with 2 nodes cluster, and I don't know if this was the reason, but when I was creating the ZFS on the 2nd node, and with the "Add Storage:" flag unchecked, it just won't add the storage to the node. Replication wont work, error: 2022-11-11 18:29:00 500-0: (remote_prepare_local_job) storage 'VMS' is not available on node 'pvematrix' 2022-11-11 18:29:00 500-0: end replication job with error: command '/usr/bin/ssh -e none -o 'BatchMode=yes' -o 'HostKeyAlias=pvematrix' root@--192.168.1.199-- -- pvesr prepare-local-job 500-0 VMS:subvol-500-disk-0 --last_sync 0' failed: exit code 255 Once again I'm not sure if this happening because the cluster is only with 2 nodes.

[@jayday4759]

I ran into the same issue/error on my 3 node cluster. Once you have created the ZFS pool on all nodes edit the storage properties on the datacenter level and select all nodes on which you have the pool.

[@meroxdev]

The storage disks and size should be the same on every node?

[@DBTechYT]

I don't know that they HAVE to be, but I think it's best practice

[@meroxdev]

One more q, it s doable to make a proxmox cluster with only 1ssd per node ?

[@DBTechYT]

@@meroxdev Absolutely. That's how I had mine set up until this morning :)

[@StanyCarteny]

woooow ^___^

[@tyrant-rex4126]

The pain of pve cluster is when 1 cluster is down you cant login via web interface

[@DBTechYT]

Do you mean when one of the nodes is down that you can't login the web interface? If so, log into a different node's interface

[@tyrant-rex4126]

@@DBTechYT well, i set up a cluster with two pve nodes. when A is down, i cant login from B via web interface. You have to ssh to B and reduce the number of expected node to login

[@quiquos]

@@tyrant-rex4126 Hi, you can, you didn't set the cluster correctly For 2 nodes you don't have a voting system, which proxmox is based on, you must change the quorum. # this is only made on one node, because it will be replicated to the other edit /etc/pve/corosync.conf # add this two lines in quorum "space" expected_votes: 1 two_node: 1 after provider: corosync...

[@jekob_pl]

Domain 1.1.1.1??????

[@DBTechYT]

Yeah. I misspoke. I realized it right as I said it, but didn't clarify. I should have said IP or DNS. It's for CloudFlare DNS

[@abdullahazad]

Why are you talking so fast? English is not the native language of everyone watching the video.

[@DBTechYT]

Unfortunately, I'm not talking fast. This is me slowed down as much as I can without feeling like I'm trying to talking like I'm drunk. I guess I would suggest playing the video at a slower speed? dbtechreviews.com/i-talk-fast

[@Felix-ve9hs]

You can always set the video playback to 0.75x or even 0.5x speed if he talks faster than you can understand :) And IMHO telling someone else how they should talk like in a Video you get to watch for free feels kinda rude :/

[@DBTechYT]

@@Felix-ve9hs This has been going on for years. Long enough and frequently enough that I put that "I talk fast" page on my website and refer people to it. I've always just talked fast. When I try to force myself to talk slower, it feels like I'm being patronizing to myself and my audience. Thanks for your comment and support!!

[@BrianThomas]

@@DBTechYT Yeah there are a few people on KZbin that I have to slow the video down and it works perfectly, but you're not one of them. I certainly don't want to hear you sounding drunk haha.

[@Miguel-gy3qq]

have you explored eks anywhere? @dbtech