Nest.js Kafka Microservice Tutorial

In this video, we create a Nest.js Microservice using Kafka as our transporter. We use both an event-driven and message-driven style of communication.
Github Repo: github.com/mguay22/nestjs-kaf...
Kafdrop: github.com/obsidiandynamics/k...
Get my highly-rated Udemy courses at a discount here: michaelguay.dev/udemy/

[@mguay]

Get my highly-rated Udemy courses at a discount here: michaelguay.dev/udemy/

[@renends7615]

In a short time, something complicated was simplified. Great tutorial.

[@maksymbaranovskyi8362]

Amazing. Your tutotials are concise, clear and packed with essential info. Thank you!

[@thangapandianponnuchami4473]

Wonderful content man! You are one of few creators who make short and clear. I am watching all your videos even if I aware of the topics to see your perception. Eager to wait for more content :-)

[@mguay]

Thank you so much! This is greatly appreciated.

[@michielsw]

Thanks for taking the time to code out our suggestion from the previous video, appreciated!

[@mguay]

Sure, thanks for the great suggestion!

[@GuilhermeSilva-vn6fw]

You simplified a lot of things and have wonderful teaching skills, thank you

[@funnyclipz520]

Awesome tutorial.... finally found a good tutorial of Kafka with example using NodeJS

[@moylababa8196]

I found your channel today and subscribed immediately. your channel is full of awesome tutorial. keep up the good work. Besh wishes

[@abhikbanerjee3719]

Awesome tutorial! Just a heads up - for anyone using NestJs 10+ and @nest/microservices ^10, the data field contains the value itself (data.value will be undefined)

[@mitchross2852]

Hi, Can you follow up this video to dead letter queue, error handling, message repeat, load balance etc. Alot of people who use kafka use it at a enterprise-level for log replication, backups, redundancy. Can you make a another simple video like this for handling real world scenarios?

[@Varuag89b]

Great content as always 👏 Keep up the good work 👍

[@mguay]

Thank you very much.

[@lord4791]

Awesommeeeeeeee ! Please make more Microservices with nest & react

[@mohammedriyadh120]

Hi Michael, Thanks a lot for the great channel, i will appreciate it if you can go much deeper with kafka clients and consumers options

[@digvijaysinghthakur2903]

Tutorial is really helpful for beginners

[@ninabohm1535]

Great tutorial, merci!

[@zhangshiyucao]

this is gold, thank you

[@mkroven]

thank you this is a great tutorial.

[@abhishekgn2626]

Great content 🤩

[@muhammadnaveedashfaq2991]

Amazing content. Thanks

[@AmirMo_Razmi]

you save me from my nightmares from my project, bro

[@tonyfrenzy]

pretty nice. Big ups 👍

[@shubham8550]

great content 😊👍

[@martinduong6944]

It's amazing tutorial.

[@farzadthev1554]

Cool !!! Thanx a lot

[@michaelmontero2902]

amazing.

[@Trans_Canada_Highway]

Could anyone comment pros and cons between setting up kafka consumer as a microservice vs as a Nestjs module as in previous tutorial?

[@codeheaven559]

Nice one

[@NilsMoller]

Hello, future man speaking here. `data.values` is not a thing anymore, the `data` object is now directly the DTO. Nest version: 9 KIafkajs version: 2.2

[@yatindrajain6828]

Hi, Is there any way to get messages based on offset or get offset of last consumed message?

[@MrVjfloyd]

Hi Michael, thank you for sharing, for some reason .Everthing is working well but I'm getting an undefined when a consume the event, probably my kakfa configuration is different

[@samereldadah1911]

Great

[@Abhishekn._]

thanks

[@MartinPerez-mi1ty]

New sub

[@saurabh9446]

I am getting this error - "The group coordinator is not available". I am using docker-compose with WSL2 on windows home

[@carlosspark2756]

Hi great content , would you show a server-frontend mvc architecture using nest-next npm package

[@arslananjum416]

Can anyone tell, how to create API with implementation of kafka in nodejs?

[@10_manhcuongnguyen87]

Hi, first of all thank you for this video. I implemented according but only get empty memberAssignment as {}

[@kirstypollock6811]

Ha. Not what I expected. I'm used to microservices communicating via pub sub using various cloud message buses. This tutorial looks to me awfully like the dreaded "distributed monolith' - the microservices in this video all seem to have to know about each other, rather than just emitting an event and not knowing or caring what other services might consume those events...The 3 services are tightly coupled - this ends up messy, hard to understand when you have many microservices. Kind of misses the whole point of microservices, for me, anyway. Fair comment or did I miss something? I'm surprised. NestJs has always seemed better thought through in it opinions than this... is it really encouraging direct service to service comms as the default ? Or is that just how this example does it?

[@amarwavare711]

How to keep nodemodules common for all services ?

[@hoseinmirian]

thanks amazing video. dont u have a course for this on udemi or somewhere? second, the concept of microservice is to have each service based on one language if team wants, and the way nest js works seems its fit for whole communication through js files? third!, Its a bit confusing when u are sending any type of request to the same root route like localhost:3000. whilst having route for each service might makes more sense what do u think? I mean u are sending all type of request to it and it dispatch events! fine! but there is no actual route so front-end can call in different end point.. can u clarify on that, tnx

[@leonelkahameni5315]

Nice video, is there a way to do that using Fastify instead od Express?

[@mailmohsin]

Noob Question: Shouldn't the API Gateway also implement a message pattern while communicating with the billing service. An Order would go to Gateway -> Billing -> Auth and it would need to return as response back to the user via same Gateway so that the billing was successful?

[@Piratacapitan]

Hi amazing video! I have a issue, I have a macbook M1 and tried to use docker-compose for intall kafdrop but I had a error. Can you have a mac M1? Do know how to install kafdrop in M1 machines? Thanks!

[@permanar_]

I have a question, lets say like we want to share the same Data Types between Billing and Auth for the handleOrderCreated from billing microservices. Like when we send a message right, then subscribe to it and get a callback with a params inside it, how do we want to make it like predictable? Like what is the "inside" of this particular param? How do we do this?

[@permanar_]

You called the `user.stripeUserId` because you "know" whats inside the object. But then how do we make it like type-safe and predictable?

[@tuanphamanh62]

Hi. In my case, I just got the undefined value just because my payload in eventPattern does not have value property. I don't know why it is different with the video . Please share if you know the difference with me.

[@darrenhudson1033]

I had the same issue and just used the data object without calling data.value. may be a change in the JSON.stringify maybe?

[@oyinlolaolasunkanmi]

Hi Michael, thanks for this tutorial. It is worth every minute of my time. I have a very important question. How do we send back a response to the API gateway, which is intended for the frontend application ? Lets say billing was successfully and frontend application needs a feed back in order to proceed or cancel the order. How do we implement this ?

[@mguay]

Thanks! Continually polling the backend, Websockets, or a refresh button are some common approaches.

[@sajankumarvijayan811]

Wondering if is it possible to achieve this using mono repo? Because I want to avoid code duplication and manage single node_modules. The way you set it up works but I wonder wouldn't be a bit difficult to maintain the project code base in the long run?

[@eunhakoo2124]

Also wondering about this

[@crossfitiswack1147]

Yeah, this seems like not the right approach tbh.. Pretty sure nest is supposed to manage this.

[@chaouchsofien7988]

hello , thx for the video , what is the difference between this approach and the other approach you did it in the other video using only kafka js , is it the same ?

[@user-yr1df4ko6u]

me too . its not clear for me .

[@abdullahajibade7210]

Does the event pattern has a return type as message pattern. I asked because of the subscription you did.

[@mguay]

It returns an Observable the same as the message pattern, however I'm not sure what data would come back, if any.

[@xisssss]

great tutorial! really sorry for the stupid question.. but what vscode theme are you using?

[@mguay]

No such thing. I’m using Material theme and icons.

[@kisstamas6675]

I learned about microservices it is the very best when the services are fully independent from each other. That means they are not communicating directly with each other, but only with events. In this video you are useing the both way to communicate with services, it's weird for me. The independency between microservices is crucial, because they can not effect each other processes (when one service terminated unexpectedly the other services have to work fine). If it necessery you have to store data redundant on each service. What do you think about that? Actually this is a very good video :)

[@mguay]

I agree this approach can come with some baggage. Check out my other Nest.js + Kafka video for a more decoupled approach.

[@kisstamas6675]

@@mguay thank you, i'm following you, so i will check your videos ;)

[@wayneyu3031]

How can I set up the VS Code theme as yours, can u give me so reference? Again, I love your video so much, great work, looks cool! keep going bro!

[@mguay]

Thanks! I use Material Theme + Icons which are extensions.

[@mridulmishra8166]

-21deg Dude do u live in siberia ? :p ... great content BTW.

[@mguay]

I'm in the USA.

[@misternether8602]

Nice tutorial, but too much code in one place, making it hard to comprehend

[@marcovignotto1559]

Thanks! I literally love your content! But you're too fast, I think in 5 minutes I paused 100 times!

[@mguay]

Thanks! And I appreciate the feedback.

[@yacinemathurin2013]

Hi, I'm currently on nest@9.x.x and kafka@2.x.x and had to call the toString() prototype method on the ClientKafka emit() method. The NestJs documentation needs update.