Setup Active-Passive Cluster with Keepalived & HAProxy (Two raspberry pis)

UPDATE:
I made a mistake on 14:24 to the config thanks for everyone for letting me know. All instances should have the same virtual_router_id. So the secondary should have 101 instead of 102. I got lucky and worked because I assigned different priorities. Sorry for the confusion.
Thanks to JR E and Parth Patel for catching the mistake
Red hat doc access.redhat.com/documentati...
Keepalived is a routing software written in C that provides simple and robust facilities for loadbalancing and high-availability to Linux systems. In this video we will talk about this technology and implement it along side HAProxy as a high availability (failover)
Chapters
0:00 Intro
2:10 What is failover?
4:30 What are we building
7:20 KeepAlived Example
* What is Keepalived?
* Setup Keepalived with HAProxy
* VRRP/VIP
* Two rapsberrypis with keepalived and haproxy
sudo apt-get install keepalived
sudo vim /etc/keepalived/keepalived.conf
vrrp_instance pi1 {
state MASTER
interface eth0
virtual_router_id 101
priority 100
authentication {
auth_type PASS
auth_pass 1234
}
virtual_ipaddress {
192.168.254.100
}
}
vrrp_instance pi2 {
state BACKUP
interface wlan0
virtual_router_id 101
priority 200
authentication {
auth_type PASS
auth_pass 1234
}
virtual_ipaddress {
192.168.254.100
}
}
Red HAT Doc
access.redhat.com/documentati...
Stay Awesome,
Hussein

[@hnasr]

I made a mistake on 14:24 to the config thanks for everyone for letting me know. All instances should have the same virtual_router_id. So the secondary should have 101 instead of 102. I got lucky and worked because I assigned different priorities. Sorry for the confusion. Thanks to JR E and Parth Patel for catching the mistake Red hat doc access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/load_balancer_administration/ch-initial-setup-vsa

[@Obsi995]

Really good stuff man. I came here to watch a vid on KeepAlived and ended up going down the rabbit hole of your video suggestions till I eventually came back to this one and watched it. I'm glad I did that, I definitely understand it well. I love your energy too. Thanks Hussein!

[@hnasr]

Obai Alsamadi thank you Obai for taking the time to leave a comment! Really appreciate it and glad you enjoyed the content ❤️

[@arghyl]

You have one seriously easy way to explain the config file. Thank you sir!!

[@zeenmc]

@Hussein, first thank you for your content. Second on side your knowledge, also you have nice English, and way how to share your knowledge with us. I am interesting in Nginx and HAProxy, and you gave me good basis to continue exploration. I am moving from Networking to DevOps track, also I have strong experience in Linux, but always I had some issues with Nginx and HAProxy. Stay safe. Thank you again.

[@ricardohincapie1537]

Love your energy. I was just smiling each time you said "puppy" again and again. Great content, this works!

[@hnasr]

😊 thank you

[@Gunzy83]

Awesome video man. Exactly what I need for my homelab.

[@ArpanDasS]

Great video, thanks Hussein!

[@lookback6314]

what a man without complex, Thanks!

[@rafael.torquato]

Thank you for the excellent content, simple, objective and functional.

[@palaniappanrm6277]

Totally new to this High Availability and Switch over. Thanks a lot 👍

[@hnasr]

Hope you enjoy it! take your time its a deep topic and always have an open mind that you can always learn more..

[@mx338]

If you have a SMTP Relay running you can also setup keepalived to send mail notifications with just a few more lines directly in the config file.

[@isayasadhanom5481]

Hussein - you are awesome!

[@greg6094]

Awesome video! Please make a health check video too 😁

[@danydanger]

Just felt the Eureka moment of understanding KeepAlived. Thanks Hussein for it, Can u also give a Tutorial for MariaDB Galera Cluster(4-Nodes) as well ? And how to achieve HA if using MultiMaster(3-Nodes) in a single cluster ?

[@artasheskhachatryan4804]

Great video, thanks for it. I have a question about Keepalived and VRRP protocol. Is it possible to configure a Virtual IP address between 2(or more) nodes which are in different Geolocation datacenters with different subnets?

[@jamallmahmoudi9481]

Hi , Hussein cool & perfect Thanks a lot .

[@vivekatbitm]

Another great video, thanks!! 1 question around selection of VIP address: if both haproxy are not in same local n/w, which ip to use for VIP? Can we use any public IP as well?

[@hnasr]

That is a very good question that I am afraid I don’t know the answer too (which is awesome it means research time) It really depends on whether the VRRP protocol is supported across different networks or not. Need to search that

[@ricardohincapie1537]

Any news on this? I'm stock with it too...

[@noahwilliams8918]

Thanks Hussein - as usual, you simplified a concept I had been stuck on for a while now down to a working example. One question that came to mind while watching was: Is there a secure (emphasis on secure) way to do this over the internet without setting up a site-to-site VPN? Can we get some TLS on this puppy for georedundancy, hah?

[@hnasr]

Noah Williams thanks Noah, interesting question and loaded and need to do some more research .. The only security im worried about is VRRP in keepalived and this stinking user/password could be weak and could be controlled by anyone with access .. For TLS you absolutely need it on whatever reverse proxy is running on your keepalived cluster in my case I used HAProxy (I made a video showing that) so encrypting the traffic itself isn’t a problem the VRRP passes traffic blindly .

[@user-yw7eg2li4o]

veryyyyyy interesting video!!!!!

[@patrickconrad2874]

Hey Hussein how would you recommend implementing kubernetes? Would it be better to add it in to my proxy server if I have it pointing to different domains. Or would it be better to attach to each application? My thinking is that I can add it on the raspberry pi and be able to spin up anything from there. But kubernetes is pretty new to me

[@donaldrais3915]

Thanks you very much for this video, it's really helpful! Just 1 question about using Keepalived for a floating ip address - I found other tools such as Pacemaker (with corosync) for this purpose and I wondered if is there a reason you chose Keepalived instead? I need to choose which tool to use and I'm not sure what should be better in terms of fast response, simplicity and reliability. My limitation is not using a loadbalancer for this task, but only use 2 master-slave servers with 1 ip address. Thanks again!!

[@hnasr]

Hey Donald. No particular reason, When I see a technology I implement it to see for my self the pros and cons. Some people did suggested I check out Corosync which I will as well. As of know I don’t know which one is better. I know keepalived works perfectly. The only beef is it works only linux , pacemaker works on windows that will be an advantage I guess

[@donaldrais3915]

@@hnasr Hi again! After reading more about HA solutions and keepalived, it turns out the split-brain problem can cause issues when both nodes think they are the master. If you heard/thought of a way to handle this issue it will be really helpful, maybe as a advanced next video :)

[@srkoenma2947]

^^ Thank you!

[@omersucuoglu8469]

Great video very helpful thanks! I have a slight issue followed instructions to the letter and it worked until I did a reboot test and from there the failover does not work anymore. tried with the id being different and the same... not sure what I'm doing wrong. trying this on 2 virtual machines running PiHole. any help is appreciated

[@vibekdutta6539]

Coooooooooooooooooooooooooooool stuff

[@maximefromspace3001]

Hi, and thx. Question, when the configuration is done, how to permanently synchronize the lamp software and databases between the servers ?

[@CoDeC__]

Hi and thanks, I have noted that you have used different virtual router id, what happen is there are other HA pairs?

[@danydanger]

Can u show the Ifconfig before and after the HA came into picture from P1 & P2 ?

[@huseyinyolalmaz3193]

Can you provide both haproxy configurations?

[@ranu__12]

Do i need to install HAproxy too ? I have two Linux system with nginx installed in it ...And after configuring keepalived in both the machines , all the settings that you have mentioned , when i am hitting the VIP i am getting a message that this page could not be reached .

[@priyashreeshetty708]

Hello. I am using 2 debian 10 version machines where I have installed haproxy and keepalived on both the machines. The setup is working fine. That is when haprxy is stopped on one machine say A the failover IP is moved from machine A to B. However, I am unable to access the stats page using the failover IP which is moved from A to B. Also, ping on the failover IP is not happening even though the IP is moved to B. The same issue is occurred when the failover IP is moved from B to A. Could you please help

[@trollingday7124]

Nice video ;-) I want to create a VIP for two Active/Passive servers but the problem is the console of this App (Dollar Universe) works with specific 4170 port... Do you know how could I specify to my VIP that works with this port or redirect to this IP_servers:4170 ports???

[@nateshsharan1844]

Does the IP show up in ifconfig ? Where can I find the IP is UP. I want to know the Master and slave

[@azouaouhamouimeche1453]

we have to take two deffrent interface ??

[@ssteva]

Hi Hussein, thanks for video, it surely helped me. Can you check priority config value in vrrp_instance section, because manpage says: # for electing MASTER, highest priority wins. # to be MASTER, make this 50 more than on other machines. priority 100 According to this, priority should be 200 for pi1 and 100 for pi2?

[@hnasr]

ssteva thank you ! Really I haven’t noticed the numbers should matter. Thanks for sharing and correcting the mistake 👍

[@optimiserlenergie1094]

So why do we need to put MASTER or BACKUP in the conf file ? if the highest priority is the master ?

[@optimiserlenergie1094]

If you want to use this on a real server from a hosting, the virtual IP can be the public IP of the server ? Or you need first a HAProxy listening on the public IP that redirects to a local address on the server like 192.168.254.100, where serveral keepalived are listening ?

[@hnasr]

Correct you have the VIP point to the servers directly HAProxy here is just acting like a reverse proxy which is a best practice (in case you want to make changes to your backend without bringing the whole site down

[@maheshmahesh-oo9hf]

Hi, I am from database team. From db end linux team configured the keepalived with load balanacer with two database servers . But when we are trying to connect to db any of the master or backup server we are able to connect. But when the application team using the vip from app to db getting error. We have opened all required db ports. But no luck. Do we need to open any specific ports from network rules for this vip (vrrp keepalived)

[@akakop]

what will be the configuration setup if don't wat to use HAproxy, there are only two servers hosting services.?

[@hnasr]

It should be the exact same thing KeepAlived config has nothing to do with HAProxy

[@dineshrhel7898]

Thanks for the video. Having doubt, is that possible, to add multiple web services with its ports to configure on single ha proxy and keep alived For example: 1. Apache 2. DB work bench

[@hnasr]

Thanks Dinesh, yes for sure you can. In HAProxy have a rule that says acl (access control list) condition /webserver go to backend “apache” which have all servers running apache web server.. But if /db you can go to the “workbench” backend and that will have all servers running db workbench To learn more about ACL check out my haproxy video

[@dineshrhel7898]

Thank you so much of your reply, I saw that ha proxy crash course and now I got it. I became of your fan the way of you presenting content stuff along with engaging the viewers without boring

[@nikunjbhartia2222]

It's weird that you did not have even a single failed request, does that mean the client is making an ARP request every time to get the Mac address of the VIP ?

[@nikunjbhartia2222]

Aah I just watched the other video about possibility of a failed request until the local client ARP table getting updated when the backup sends a broadcast about Mac update

[@PapipopCOD]

I do a curl "ip virtual" and i get 503 service unavailable no server is available to handle this request. .. pls help

[@PapipopCOD]

with everything configured

[@hnasr]

Papipop that means your backend is not available (anything behind haproxy) haproxy is available but no backend services.. check that

[@PapipopCOD]

@@hnasr pls check my haproxy i can't find the solution #--------------------------------------------------------------------- # Example configuration for a possible web application. See the # full configuration options online. # # haproxy.1wt.eu/download/1.4/doc/configuration.txt # #--------------------------------------------------------------------- #--------------------------------------------------------------------- # Global settings #--------------------------------------------------------------------- global # to have these messages end up in /var/log/haproxy.log you will # need to: # # 1) configure syslog to accept network log events. This is done # by adding the '-r' option to the SYSLOGD_OPTIONS in # /etc/sysconfig/syslog # # 2) configure local2 events to go to the /var/log/haproxy.log # file. A line like the following can be added to # /etc/sysconfig/syslog # # local2.* /var/log/haproxy.log # log 127.0.0.1 local2 chroot /var/lib/haproxy pidfile /var/run/haproxy.pid maxconn 4000 user haproxy group haproxy daemon # turn on stats unix socket stats socket /var/lib/haproxy/stats #--------------------------------------------------------------------- # common defaults that all the 'listen' and 'backend' sections will # use if not designated in their block #--------------------------------------------------------------------- defaults mode httpchk log global option httplog option dontlognull option http-server-close option forwardfor except 127.0.0.0/8 option redispatch retries 3 timeout http-request 10s timeout queue 1m timeout connect 10s timeout client 1m timeout server 1m timeout http-keep-alive 10s timeout check 10s maxconn 3000 #--------------------------------------------------------------------- # main frontend which proxys to the backends #--------------------------------------------------------------------- frontend main *:80 acl url_static path_beg -i /static /images /javascript /stylesheets acl url_static path_end -i .jpg .gif .png .css .js use_backend static if url_static default_backend app #--------------------------------------------------------------------- # static backend for serving up images, stylesheets and such #--------------------------------------------------------------------- backend static balance roundrobin server static 127.0.0.1:4331 check #--------------------------------------------------------------------- # round robin balancing between the various backends #--------------------------------------------------------------------- backend app balance roundrobin server app1 127.0.0.1:5001 check server app2 127.0.0.1:5002 check server app3 127.0.0.1:5003 check server app4 127.0.0.1:5004 check # HAProxy Load Balancer for Apache Web Server frontend http-balancer bind 10.5.5.60:80 default_backend web-servers backend web-servers mode http balance roundrobin stats enable stats auth admin:123 server cluster01 10.5.5.31:80 check server cluster02 10.5.5.32:80 check

[@PapipopCOD]

@@hnasr node01 ! Configuration File for keepalived global_defs { notification_email { [e-mail] } notification_email_from [e-mail] smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id LVS_DEVEL } vrrp_instance keep.com { state MASTER interface eth0 virtual_router_id 51 priority 101 #used in election, 101 for master & 100 for backup advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 10.5.5.120/8 } } node02 ! Configuration File for keepalived global_defs { notification_email { [e-mail] } notification_email_from [e-mail] smtp_server 127.0.0.1 smtp_connect_timeout 30 router_id LVS_DEVEL } vrrp_instance VI_1 { state BACKUP interface eth0 virtual_router_id 51 priority 100 #used in election, 101 for master & 100 for backup advert_int 1 authentication { auth_type PASS auth_pass 1111 } virtual_ipaddress { 10.5.5.120/8 } }

[@PapipopCOD]

@@hnasr I'm desperate sorry for my behavior but I don't know what to do

[@kirillvk3171]

MASTER should has higher priority

[@default_youtube_profile]

why can't you just use option allbackups in haproxy to load balance in case of failover of 3001 and 3002. we don't need keepalived in that case .

[@hnasr]

What if haproxy failed?

[@default_youtube_profile]

@@hnasr good point because we have 2 instances of keepalived with same ip high availability is served.

[@hnasr]

👍👍

[@default_youtube_profile]

@@hnasr i have cross continent vpn using openvpn , i have 2 instances of haproxy, then should i use eth0 in keepalived config or tun0 ? All web server and haproxy and keepalived are inside same vpn.

[@earthling_parth]

PEOPLE BEWARE: This is a wrong Keepalived configuration. We have 10+ HAProxy/Keepalived clusters running in the same subnet in our company LAN and virtual_router_id being different is what even allowed Hussein's demo to be kinda successful was his priority understanding is also wrong. Priority of 200 > 100 and if you had kept the same virtual_router_id, Pi2 would've been elected the master from the get-go.

[@hnasr]

Thanks for catching this, I updated the video description and pinned comment. It was my luck with different priorities as you said that caused my config to work.

[@earthling_parth]

@@hnasr I didn't add this to the comment thinking you'll never see this but thank you very much for all your other backend engineering videos and DevTools series. I highly appreciate you Hussein ♥️

[@trollingday7124]

Nice video ;-) I want to create a VIP for two Active/Passive servers but the problem is the console of this App (Dollar Universe) works with specific 4170 port... Do you know how could I specify to my VIP that works with this port or redirect to this IP_servers:4170 ports???

[@maheshmahesh-oo9hf]

Hi, I am from database team. From db end linux team configured the keepalived with load balanacer with two database servers . But when we are trying to connect to db any of the master or backup server we are able to connect. But when the application team using the vip from app to db getting error. We have opened all required db ports. But no luck. Do we need to.opem any specific ports from network rules for this vip (vrrp keepalived)