Golang Microservices: Events Streaming using Apache Kafka

Рет қаралды 31,185

Mario Carrion

Күн бұрын

Пікірлер: 20

@huseinnashr 3 жыл бұрын

Thanks for making this video. Really helpful

@MarioCarrion 3 жыл бұрын

Thanks Husein. Take care.

@arturfil 11 ай бұрын

Thank you for all this content!

@MarioCarrion 11 ай бұрын

Thanks for watching! Cheers

@arturfil 11 ай бұрын

@@MarioCarrion I was wondering if you would do another step by step from scratch tutorial for Kafka & microservices

@MarioCarrion 11 ай бұрын

That's not a bad idea, a lot of things have changed since I recorded this video.

@tigaron27 6 ай бұрын

Hi Mario! Thank you for this content, it's very well explained. Would it be possible for you to make a tutorial about adding gocron package in this microservice series?

@MarioCarrion 6 ай бұрын

From a quick search "gocron" seems to emulate the original cron; I don't know you're use case but I'd keep it simple and try using whatever containerization option you currently use for running scheduled jobs, that's typically the simplest way to do things.

@weitanglau 3 жыл бұрын

Great video! Will you be making a video on how to implement SAGA pattern for microservices?

@MarioCarrion 3 жыл бұрын

Thanks Lau. Yes, I'm planning to covert different patters and Sagas are going to be included.

@weitanglau 3 жыл бұрын

@@MarioCarrion nice! Looking forward to it :)

@najamawan 2 жыл бұрын

Hi mario, i have a use case. Imagine our web server is exposing some rest endpoints (gorila mux) that have long time to process (external systems) so we publish a message on kafka or nats and just give result back to user that we promise when we have result available u will see it in web ui. Cosumers that are console apps written in go consume event, process and publish a new event with result. Now we want to consume those result events inside same gorilla web server and through websocket or server sent event wana update user that their long async task is completed. My question is if we are running a infinite loop and our gorilla mux router endpoints still work ? i am guessing since this loop is not sleeping request will not be handled and user will face timeouts

@MarioCarrion 2 жыл бұрын

Hi! You're correct, your client will eventually time out; however what if you approach it in a different way? Let your customers know (like you said) "we are processing your request come back later", and in the backend do what you said (consume and process of the event) and then let the customer know "your result is available" maybe using an email or other alert mechanism. Other way would be the frontend to poll the status of the process from the backend from time to time, to indicate the progress being made, but I don't like that approach because the client could leave a browser open all day, and if the process takes long time like you said you are going to be wasting backend resources.

@nenadsmiljkovic2793 2 ай бұрын

@@MarioCarrion Hi Mario, thank you for investing so much energy and passion to help us inexperienced developers learn something highly valuable. I am trying to learn from this video like many others, and would like to build upon this comment. Which type of Kafka producer would you use for the above scenario? Async or Sync one? I am still new to Kafka, I can't make this choice confidently because all those Kafka config options confuse me (wait all/fire and forget/etc.). Please allow me to clarify further with pseudo-code, as we developers communicate the best that way: //**** some HTTP endpoint handler code // extract request data, let it be some JSON // KEY MOMENT: publish synchronously or asynchronously into kafka here? // return HTTP 202, something else will return the processed result to the client //**** so many things to consider for the KEY MOMENT: - can i afford to lose message or not? in my case i can not - therefore should i go with sync producer (this is a blocking call, which may hurt performance)? - or can I somehow use async one without losing the message because it will unblock the handler and keep my server responsive? - which one to choose for high throughput applications? - how would you approach this? maybe use Redis instead of kafka, and then read from Redis, dump into Kafka (processing architecture stays the same) ? Nobody on YT and other tutorials discusses this very common scenario, with usual tradeoffs to be faced. I think I speak in the name of all of us who are learning that we would be in great debt to you if you could shed some light about the above questions. Thanks again for everything.