RabbitMQ on Kubernetes for beginners

[@MarcelDempers]

For those who missed out RabbitMQ Basics: kzbin.info/www/bejne/npe4eoqbZtl-bqc RabbitMQ HA Replication: kzbin.info/www/bejne/fKvUm6eLZ91pj6M

[@magrag1987]

Hi@That Devops Guy , this was very helpful in understanding the concept. Can you help me in an situation I'm facing. How to avoid loss of messages in a time when a pod is down and a new pod is assigned. It takes few seconds fr kubernetes to assign new pod. And in this time, there are loss of messages 🙏 please help how to tackle. Thank you

[@eniac78]

Thanks! I was following the example on the Rabbitmq website in the post entitled "Deploying RabbitMQ to Kubernetes: What's Involved?", but they left out the part about setting env vars for nodename, use long name, and k8s hostname suffix. I was about to give up and move back to operator until I concluded it was probably a dns name resolution issue and set those env vars from your example. Then it all clustered together! Thanks! :)

[@ryanshannon7703]

Ran into similar issue when setting up DevOps build agent targeting a VM we hosted in Azure that required SQL Server access permissions. Turns out the agent uses a network service identity and I needed to create that as an actual user on the VM itself to enable access to the actual disk of the VM, rather than just use it as a zombie for powershell scripts. It's like, bruh, Microsoft really needs to at least add a caveat or something to the documentation.

[@SV-tc8cu]

Thank you! You have a talent for explaining things, and this is pretty rare thing to find on the youtube.The format is very succinct but the code very well organized and super easy to follow. I have not had a single "what the heck is this for" moment. I like the fact that you explain things that I already know but need reminding. Instead of stopping video and googling, I am reminded about it there and then or just given a quick idea about the concept so I can research it later. This is a great way to avoid frustration and context switching.Fantastic job! I only had a single issue where LoadBalancer would be stuck in pending state. I followed LoadBalancer setup section on kind website and managed to get it working at the end.

[@DominicTrix]

Wow, your hard work setting these tutorials up is fantastic and appreciated. Dense amount of information but so well presented I feel like I had all my questions answered and more. Thank you.

[@robwaldram1275]

Because of you and @fireship I have been pushed to non-stop learn new skills. Right now slowly catching up to the main developers of the project I was an UX / UI designer in. Thanks for broadening my reality of what is possible. Keep up the amazing work!

[@isaacndarwa]

Thanks for the video! It's very helpful. I have one issue though: I can't get the port-forward to work. I am using a self-hosted cluster with OVH based servers, and the port-forward command just isn't working for me. Do you have any idea what the issue could be?

[@DrMicr0b]

Thanks for the video. Helped me a lot. There is a minor discrepancy in video and github commands - in video you use the port 5672 to connect to the UI but in the github README it is 15672

[@MarcelDempers]

Yeah you're right. 💪🏽 it should be port 15672 in the video. I must have gotten an error and reshot the scene and looks like the original scene made it into the video instead of the fixed scene. 👀

[@NaorPeretz]

HI @MarcelDempers few questions & requests for updates: 1. you use in post-forward command, after you delete rabbitmq-0, the UI management get lost and I need to re-run it, is that for you too? look like you don't run it again or you remove it in edit 2. why you don't show use in namespace inside the yaml files, once you move to complex systems with few namespaces, this is very useful. 3. add how you declare your queues on start the cluster inside the yaml files(probably more then 1 will be welcome), usually you will need more then 1 queue, at least for the hard stuff. 4. how you upgrade your versions? I found some information on it, but it didn't work me, and because I just create the cluster I don't need now, but it will be useful, Especially when the version you are showing has deprecated. 5 look like set_policy about the mirroring deprecated in 3.11.0 version, I think a lot of people will want too see it on the new version. Thanks you:)

[@RebeliousSapien]

how is persistency maintained in Rabbitmq when it's Kubernetes ?? ... when you delete a rabbitmq pod, what happens to the queues items?

[@이찬진-o1g]

i run m1 docker desktop k8s , kubectl get storageclass. and get name hostpath ... and change storageClassName : "hostpath" it works! nice video one question,,, is there any reason mirroring without rabbitmq-3 node?

[@dave-ventura]

stable upper arm! And a good tutorial, thanks

[@allinonevlog9169]

Hello sir, I followed all the steps you taught. But still i not getting rabbitmq dashboard.

[@Sushic.]

Hi, I am encountered an error "Bad characters in cookie" while rabbitmq deployment, I am doing same things shown in video. But I can't deploy rabbitmq, could you help me?

[@brockhard]

Im curious. Is there a reason you would deploy it this way over using a helm chart? Ive really embraced gitops and can use argocd to deploy both ways. I wanna try it this way woth rabbit mq to see how it feels to manage it this way. Would love to hear your opinion on managing deployments and when you would use helm or create the yml yourself. Excited to see you making new videos. Youve been a major help learning kunernetes and pushed me to manage my own k3s cluster.

[@MarcelDempers]

Thanks for the kind words 🙏🏼 I've always used raw YAML when building and putting things together that i have to maintain as its one less layer of complexity i have to deal with. When it comes to picking open source things off the shelf or making things available for others to use, helm would be better as you generally dont change the overall manifests etc and only set values per environment. For my clusters i use helm for ingress, monitoring, observability, keda, and platform level components. For microservices and their components like queues, caches etc i just use yaml

[@rajendragosavi2233]

Very good and informative video :) Subscribed. Watching and implemeting all your tutorials. Thanks

[@kaustuvchatterjee4117]

How livenessprobe & readinessprobe can be implemented for rabbitMQ?

[@roymollenkamp991]

Whatever you make... it's not enough. Thank you.

[@jm9116]

Thanks for to tutorial. I'm stuck on the publisher though: At about the 15:47 mark in the video, you use postman to test-submit a message, I get error: connection refused. I have verified with kubectl get pods that the rabbitmq-publisher is running. I'm not sure what I'm doing wrong. Any hints?

[@MarcelDempers]

Had a quick look. If you're using the kind cluster - it does not have support for service type=LoadBalancer, which is used on the publisher service where we post messages to. To post messages to this endpoint, I'd suggest running `kubectl port-forward` to that service and then proceed to post messages. kubectl port-forward -n rabbits svc/rabbitmq-publisher 80:80

[@jm9116]

@@MarcelDempers Thank you very much -- I was using Kind, so that explains it.

[@zheniachubarov3384]

Hi! Thank you for the awesome video? But could you make an update because environment variable for erlang cookie is deprecated and newer versions use different approach

[@sumer9999]

Gem of a video, life saver for many people

[@Misanthrope84]

That's a great video, thanks for creating it! However, I have deployed all of the yaml files you used in this tutorial into my cluster and I think you have an error: in your video you access the rabbitmq-management interface by using port-forward to port 5672, while in fact this does not work. It only works with forwarding to port 15672. Thanks again mate!

[@MarcelDempers]

You're right, it's been fixed in the readme.md 💪🏽Thanks for the kind words 👍🏽

[@babu_bangalore]

Followed all the steps, but the pods are in pending state, and it shows Warning FailedScheduling 9m40s 0/5 nodes are available: 5 pod has unbound immediate PersistentVolumeClaims. Warning FailedScheduling 9m40s 0/5 nodes are available: 5 pod has unbound immediate PersistentVolumeClaims. What should I do ?

[@MarcelDempers]

I'd suggest to watch the videos on statefulsets and persistent volumes. You will need to troubleshoot your volumes and storage class to see what storage is available in your cluster and if its working

[@DouglasNaCl2]

For those facing this issue, I found the solution, but first let's go to the cause. When running the command kubectl apply -n rabbits -f kubernetes/rabbit-statefulset.yaml following the tutorial, I didn't pay attention to the section storageClassName: "standard". After running kubectl get storageclass, I noticed that in my case the volume was called hostpath, so I modified the file to storageClassName: "hostpath". However, the PVC had already been created and I didn't notice it. It was only after debugging and finding the command kubectl get pvc -n rabbits that I realized the error. The volume created in the first attempt is persistent as expected, and I would have to manually remove it for a new one with the correct settings to be created. The command used was kubectl delete pvc data-rabbitmq-0 -n rabbits. After that, I just had to run kubectl apply -n rabbits -f kubernetes/rabbit-statefulset.yaml and everything went as expected. Other useful commands are: To identify that the problem was in the PVC: - kubectl describe pod rabbitmq-0 -n rabbits To see what we have in the PVC: - kubectl describe pvc data-rabbitmq-0 -n rabbits

[@budi0580]

Awesome video ❤️. Auto subscribe.. and waiting for postgres cluster on kubernetes 👍

[@tynshjt]

RabbitMQ has Kubernetes Operator now. Do you recommend using that or this manual way is still better?

[@nabarundebjani]

Could you please provide the steps if ingress need to configure for management UI

[@hectorfonseca1002]

Thank you so much, you have positively impacted my life. Thank you!

[@amitkumdixit]

port forwarding as to be done again one we delete any pods.

[@g3ar75]

Really loved the previous 2 videos, this one was really nice too but next level for someone like me who rarely works with kubernetes. Fault is at me for being a noob but I would love to know if you have any other kubernetes tutorials. Final question, which would you recommend most? Minikube or Kind?

[@MarcelDempers]

Thanks for the kind words, I really recommend Kind, its much more light weight and faster to bootstrap and you can run multiple clusters side by side. I do have a Kubernetes playlist, its quite old, but it still holds up :)

[@g3ar75]

@MarcelDempers thank you for the quick reply, I've been going deeper down the rabbit hole and was trying to combine this with ha-proxy for loadbalancing, still having a bit of a rough time setting it up since I'm not finding much documentation on how to make the amqp call go to the loadbalancer instead of rabbitmq

[@g3ar75]

​@@MarcelDempers I just managed to finish the setup with docker-compose and included ha-proxy for loadbalancing, I do think there's little documentation on haproxy for mq if you don't really know the details about tcp, are there any tips I can find on your channel? I'm gonna move on to kubernetes next

[@concay8872]

Thank you so much. I tried to use your yaml script and I used persistence storage however, If I define the storage less than 500mb, it didn't work. CRASH REPORT Process with 0 neighbours exited with reason: {{could_not_write_file,"/var/lib/rabbitmq/mnesia/[e-mail]. so I increased them to 10Gi and worked. I don't understand what is the storage for? for the production what is the recommended size? what situation the storage will be full? thanks again

[@hellitelli]

Thank you for the hint with increasing the disk size. I run into the same problem today.

[@RebeliousSapien]

do you still need the rbac and the plugin to find other rabbitmq instances if you only plan on having just the one ?

[@mzw8374]

Hello, What if we setup the rabbitmq on EC2 instance not with containerization (k8s) for serving all the microservices on kubernetes cluster, is it possible?

[@Kushagra105]

Thanks for this wonderful tutorial!!

[@johnnanong8756]

Any alternative than kind? It seems like kind is very "go" dependent when installing. Probably an alternative tool that you dont have to install other compiler? Thanks!

[@MarcelDempers]

Kind is a portable tool to test K8s. You don't need go installed to run it. It's an alternative to heavyweight minikube. There is k3s too, which is more geared for production workloads.

[@johnnanong8756]

@@MarcelDempers Thanks! Your 3 series video is really helpful in fully understand clustering/mirroring and now deploying to kubernetes... :)

[@kaustuvchatterjee4117]

Can you share the URL of the github repo you are referring?

[@MarcelDempers]

description 😎💪🏽

[@magrag1987]

hi, thnak you, i was looking for this lesson from longtime. in my organisation we have a cluster and i have some data loss, when i delete a pod. i am guessing this may be due to some policy, like HA-fed could be set to all nodes, and i want to check the status of the fed, CAN YOU PLEASE HELP ME HOW TO CHECK THE FEDERATION-STATUS FROM A NODE, .. i used rabbitmqctl rabbit@NODE NAME federation-status but i didnt get any json file. thank you

[@thangtrananh622]

I have configured PVC to be 50Mi. It seems that it isn't enough. The rabbits usually crash. Then I decide to increase it to 5Gi, and everything is OK.

[@edmoretshuma8311]

hie there , thanks for a very insightful video. I am getting an error with the erlang cookie when the containers startup : CRASH REPORT Process with 0 neighbours crashed with reason: "Bad characters in cookie" in auth:init_no_setcookie/0. I set the cookie in bash with dd if=/dev/urandom bs=30 count=1 | base64 . Is there something else Im missing?

[@ananyasahoo8552]

This video is really amazing and detailed. Have you made any videos regarding setting up rabbitmq in nomad? It would be really nice if you make one🙂

[@vinaygowda7413]

Could you please try to cover "microservice logging" using Grafana Loki.

[@brandonfang524]

Thanks for the great tutorial for RabbitMQ. I wondered if this also work without RBAC? Cause we may not have namespace admin authority at work.

[@abhilashhazarika6714]

Thank you for this. I am using this to deploy the cluster to AWS EKS. Can you please explain how to access the management dashboard of rabbitmq over nginx ingress load balancer? Thanks. I have nginx setup that I can access other deployments but I am not sure how to configure nginx for statefulset.

[@abhilashhazarika6714]

Nevermind, I found the solution. Create a service targetting a particular rabbitmq pod and target port should be 15672. Then add new redirect rule in nginx for this service.

[@DutGi]

Thanks for the great video! can i clarify that (14:04) 4 pods are formed for a rabbitmq cluster? is it also possible to create a rabbitmq cluster on one pod (i.e. 4 rabbitmq node containers on a pod = one rabbitmq cluster) ?

[@MarcelDempers]

Correct, a cluster is a distributed system to give you high availability. The more pods the better so you can distribute them across nodes. Although you can run many processes in 1 pod, its an anti pattern and not recommended.

[@DutGi]

@@MarcelDempers thank you so much, learning alot from you channel!

[@uhjarnaa]

Thank you very much for a very instructive video. I'm getting one issue with the publisher pod... Whenever I try to post a message to the pod it says connection refused. Any thoughts on what to check to get it working? I have successfully test the publisher image when checking out the two other videos in the series.

[@hirenpatel2006]

Hi I am facing the same issue

[@helgardwagener]

Not sure if you keep an eye on comments for old videos, this is invaluable thanks for explaining so clearly how to implement RMQ to K8S. How would you update the RMQ config if the cluster was deployed via Helm?

[@MarcelDempers]

Thanks for the kind words 🙏🏽 You would have to consult the helm chart documentation, they should provide you relevant overrides via the values file.

[@helgardwagener]

@@MarcelDempers I can't believe you actually replied! I'll take a look at the Helm chart for reference on overrides. I'll definitely be pointing people to you content not even the kubernetes docs do as good of a job, I am now half way through the K8s playlist, thanks again, Lekker dag verder!

[@vagifgafar2946]

Is it possible to combine all these yaml files into one?

[@overthinker-tk1zn]

Awesome ! Thank you!!

[@davepazooki3891]

Can I push rabbitmq:3-management docker image from my docker desktop to Azure, e.g. in a container register? If yes how? Thanks

[@MarcelDempers]

rabbitmq:3-management is a public image and resides on docker hub. For Azure registry, you need to follow their docs on logging in to the registry, then you could tag rabbitmq:3-management as /rabbitmq:3-management using the docker tag command, then docker push to push it to your ACR.

[@davepazooki3891]

@@MarcelDempers Yes that is the correct way to do it. What you also need to do is to create and run an instance of the pushed image under Services, Repositories of ACR.

[@Kushagra105]

Thanks for the video, I was facing some issues with autoscaling my workloads as per the queue length, But now I have solved that problem, All thanks to Prometheus Operator Tutorial, This video also proved really helpful to me as now we can migrate our Rabbit workloads to Kubernetes. Thanks Once again!!

[@davoswain]

Great video and series on RabbitMQ. I've followed your tutorial using GKE but also added an Ingress on port 80 with a domain name pointing to a loadBalancer pointing to port 1883 to receive MQTT messages. When I use the loadBalancer IP and port 80 to send a message it works but when I try and go through Ingress the message doesn't get through. The Python code sending the message doesn't throw an error. Just wondering if you have any words of wisdom to help resolve this issue? I want to be able to connect to the RabbitMQ Kubernetes cluster for all message connections using the Ingress and domanin name. Thanks.

[@MarcelDempers]

Using ingress is not as straightforward but possible depending what controller you use since Rabbit is not plain HTTP and most ingress controllers are mainly using HTTP. There are ways to do it with NGINX: medium.com/cooking-with-azure/tcp-load-balancing-with-ingress-in-aks-702ac93f2246 And also official docs: github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/exposing-tcp-udp-services.md

[@davoswain]

@@MarcelDempers thanks for the prompt response. I'll check out the NGINX links you sent. I was hoping I could get away with a standard GKE ingress configuration but am realising that I need to build a deeper understanding of some different controllers. Once again thanks for your help.

[@mattkirkevold7437]

first off, great videos! What I wonder is if you use mirroring can you setup the pods without persistent volumes?

[@MarcelDempers]

Sure you can. But they are all trade offs Replication in most data stores allow multiple copies across multiple instances. So you dont always need to persist the disk, however, doing this you greatly increase risk of losing all your data. At the end of the day it all depends how important your data is, how your services will react to data loss and how your system will recover. For message queues, it may be different to databases, since messages may be short lived so impact of losing a queue's data may not be as serious, but that depends on your design and importance of your data. For example, How would customers be impacted, if the queue lost all its data, etc

[@TcheloJG]

Hi, Marcel! First of all thanks a lot for this tutorial. I am learning a lot with it. I have question for you or maybe one of the other followers. In the video you passed rabbit-0, 1 and 2 , as ha-params and in you GitHub repo you changed it to 1, 2 and 3. And beyond that, you changed the node names. Why these changes?

[@ashseth7885]

You are gem, thank you

[@MrVnelis]

Thank you so much for your videos!

[@hcgaron]

Your videos are freaking amazing! I'm debating currently whether or not to run rabbitMQ in my k8s cluster or to go with a cloud SaaS solution. I can't tell what kind of Persistent Storage to use if I host it on my own. I'll probably deploy this to DigitalOcean, so I could use their storage. It sounds like using the actual droplet storage (node storage) would be bad because it won't be available across nodes, correct? What kind of storage do you recommend? I'm doing a bit of cost analysis, since it sounds like I'll have to purchase some amount of storage - and it might be a wash to just use a cloud service and have it managed for me. Thanks for all your great content!

[@MarcelDempers]

It depends what you're trying to achieve and reasoning for persisting messages. If it's for audit reason and you want history for a year's data, you could have an audit queue where you send copies of the message to where a consumer writes it to cheaper storage like S3 or some cheap bucket store. For cross node storage (HA) you need to use mirroring to achieve replication for high availability. (RabbitMQ feature, see my clustering video for more) Depending on your system, persisting data for longer than few hours may not be needed so you could just use a local node folder to save cost

[@pietercoetzer5537]

Lekke Man...!! Thx for the informative vids. Really helped me get my mind around the world of RabbitMQ. If only all docs was so easy to follow :-) KZbin 4 the win, cause who presses F1 these days??

[@ryanshannon7703]

F1 -?

[@vtvvnha2230]

awesome video

[@yijencheng7641]

Hi, thanks for your AMAZING tutorial. I just started understanding RabbitMQ clustering, and have went through your three videos. However, I'd like to know what's difference between using Kubernetes cluster versus the previous video(config file)? Cause using kubernetes seemed wayyyy more complex, but the two ways seemed to acheive the same outcome :( Thanks!

[@MarcelDempers]

Yeah they're both achieving similar outcomes. The main difference is the docker solution is more simplified because it implies that the containers are running on fixed hosts with fixed host names. In a cloud solution, servers scale and host names are dynamic. This is the case with Kubernetes as the containers may move between hosts and you'll also want to be able to scale up\down. The K8s solution uses a discovery plugin to achieve clustering between containers VS the docker one which simply has fixed configs.

[@yijencheng7641]

@@MarcelDempers just want you to know this is really helpful to me, thanks!!

[@9819960437]

This video's are realy informative. Please create videos on mongodb replicaset cluster on kubernetes and elasticsearch multinode HA cluster on kubernetes.

[@markjaysongonzaga1131]

I hope this is the same as kubernetes in raspberry pi 4GB

[@markjaysongonzaga1131]

I think it works but i'm just having hard time doing port forward 192.168.1.150:7000/ it is displaying garbled text

[@markjaysongonzaga1131]

i see it now.. i just have to expose 15672 port from the container :)

[@thatinstant]

This video falls apart when that devops guy adds mirroring by exec'ing into the pods instead of doing it right and ensuring that the mirroring setup is repeatable and declarative.

[@SimenRussnes]

WHY IS THIS GUY SO HUGE?=

[@MarcelDempers]

Gotta get huge to code huge 💪🏽😎

[@gianlucadivincenzo1956]

Simply

[@jackmurak]

Beginners??? I think not

[@patrickjustice7371]

mirrored queues are deprecated feature, you should use quorum queues for replicated queues

[@phaniindukuri5873]

Awesome articulation, thanks for the session... But It is not working for me when i post from publisher, getting error "Error: connect ECONNREFUSED 127.0.0.1:80" - -- Further here is my local machine netstat output - i think ishould have the port 80 forwarded right? -- $ netstat -ant|grep LISTEN|grep 80 tcp 0 0 127.0.0.1:8080 0.0.0.0:* LISTEN -- I think this is for management tcp6 0 0 ::1:8080 :::* LISTEN

[@phaniindukuri5873]

Resolved it.. somehow the published is not exposed to port 80.. I could able to connect with port 31669 and working fine.