Kafka Streams using Spring Cloud Stream | Microservices Example | Tech Primers

This video covers how to leverage Kafka Streams using Spring Cloud stream by creating multiple spring boot microservices
📌 Related Links
=============
🔗 Github code: github.com/TechPrimers/kafka-streams-microservices-example
🔗 Kafka setup: docs.confluent.io/platform/current/quickstart/cos-docker-quickstart.html
🔗 Public Domain API: domainsdb.info/
📌 Related Videos
===============
🔗 Spring Boot with Spring Kafka Producer example - kzbin.info/www/bejne/hJuriop7i8R4ldE
🔗 Spring Boot with Spring Kafka Consumer example - kzbin.info/www/bejne/f5_GeGOVjbiGeMk
📌 Related Playlist
================
🔗Spring Boot Primer - kzbin.info/aero/PLTyWtrsGknYegrUmDZB6rcqMotOFZKvbn
🔗Spring Cloud Primer - kzbin.info/aero/PLTyWtrsGknYeOJHtd3Ll93GRf28hrjlHV
🔗Spring Microservices Primer - kzbin.info/aero/PLTyWtrsGknYdZlO7LAZFEElWkEk59Y2ak
🔗Spring JPA Primer - kzbin.info/aero/PLTyWtrsGknYdt079e1pyvpgLrJ48RQ1LK
🔗Java 8 Streams - kzbin.info/aero/PLTyWtrsGknYdqY_7lwcbJ1z4bvc5yEEZl
🔗Spring Security Primer - kzbin.info/aero/PLTyWtrsGknYe0Sba9o-JRtnRlkl4gXMQl
💥 Join TechPrimers Slack Community: bit.ly/JoinTechPrimers
💥 Telegram: t.me/TechPrimers
💥 TechPrimer HindSight (Blog): medium.com/TechPrimers
💥 Website: techprimers.com
💥 Slack Community: techprimers.slack.com
💥 Twitter: TechPrimers
💥 Facebook: TechPrimers
💥 GitHub: github.com/TechPrimers or techprimers.github.io/
🎬Video Editing: FCP
---------------------------------------------------------------
🔥 Disclaimer/Policy:
The content/views/opinions posted here are solely mine and the code samples created by me are open sourced.
You are free to use the code samples in Github after forking and you can modify it for your own use.
All the videos posted here are copyrighted. You cannot re-distribute videos on this channel in other channels or platforms.
#KafkaStreams #SpringCloudStream #TechPrimers

[@cheshvikaraj7769]

nice explanation. Thank you

[@dheereshsingh6415]

tremendous effort to make this stuff so good and understandable

[@TechPrimers]

Glad you found it useful

[@kylemarcus1152]

Wow awesome video! Learned a lot, Kafka Streams API is very powerful

[@jaineshmodi]

man this is the best I have seen on Kafka stream.

[@williandougherty6674]

Great tutorial! Good to see in practice the difference between Kafka and Kafka Stream!

[@melwin-dmello-dev]

Just what I was looking for!!! Thank you for posting this helpful content!

[@TechPrimers]

Glad it was helpful!

[@ljka]

Cool. I like how you explain and go directly to what is needed to make things work. No too many speeches; because we can find that in the documentation ourselves.

[@TechPrimers]

Cheers bud #TimeIsPrecious

[@diogennesthesinope61]

Never seen this explained so clearly. Great job!

[@TechPrimers]

Glad it was helpful

[@cultOfApple]

Great , this is super awesome and v complete and realistic one

[@TechPrimers]

Thank you so much 😀

[@panndey]

Great video explained both the ways of vanilla Kafka way as well as Kafka Streams using spring cloud binder .

[@balachandran7777]

Really liked it. Great efforts made for clear understanding.

[@TechPrimers]

Glad you liked it

[@PrashantZombade]

Nicely explained, thanks for such a great tutorial!

[@SandeshReddy]

Great session👍nice use case with kafka stream processing with spring

[@reachkrishnaraj]

Another concise and helpful AV. Appreciate it, buddy.

[@ankitjain6437]

Ajay !!You are simply incredible...Thanks a ton for sharing such a valuable Kafka usage pattern..You Rock !!

[@TechPrimers]

Cheers Ankit. Glad it's helpful

[@daccu8633]

thanks a lot. it is very helpful for kafka beginner

[@narendrakumariitb]

Thank you for such an easy explanation 🙏

[@TechPrimers]

Glad it was helpful!

[@JameS00989]

Very 🔥 nice video clearly explained waiting for more such video and thanks for amazing content 🙏

[@ambarishkapil8004]

Damn! you really coded it all up under 30 mins. Sr. Dev spotted I guess XD

[@udit9337]

Excellent Kafka tutorial thanks

[@riteshsingh1245]

Very Nice tutorial for Kafka Streams

[@SaiKumar-vf9lz]

Much needed .. Stay safe Ajay .. thank you for the awesome work

[@TechPrimers]

You too stay Safe Sai. Thank you👍

[@anandhnarendra9230]

Awesome explaination !! Very clear

[@TechPrimers]

Glad it was helpful!

[@manis8483]

Great flow of explanation. Loved it ❤️

[@kappaj01]

Great stuff sir. Excellent video.

[@allaboutmusicanddance5108]

Great tutorial thankyou 🧿

[@manognajoshik8465]

Hi Ajay..I am a regular follower of your tutorial.. Could you please make a complete video on project reactor (Spring webflux) covering every concept? If already available, kindly provide me the link.

[@nicolaslopez-82]

Very useful video! Thank you for your amazing work.

[@TechPrimers]

Glad it was helpful!

[@maheswarbehera1]

Great educational video bro. It will be great if you can add Kafka Connect and ksql db too.

[@siddheshlatkar2165]

Great tutorial. Thanks!

[@TechPrimers]

Glad it was helpful!

[@haohonglin466]

Thank you!!! Very useful tutorial!!

[@rakeshroshan829]

Thank you Ajay. This is very informatic and easy to comparable with kafka and kafka stream. Please Can u make a video on Kafka connect with other messaging system like activemq and with other data source like oracle ..

[@teminoah2960]

awesome!!, great tutorial from a great teacher

[@arungore6578]

Thank you for the tutorials. What is the difference between spring kafka and spring cloud stream , spring stream kafka. When we should go for spring Kafka and Kafka Stream ?

[@RealityCoupleVlogs]

How to do error handling for publishing or consuming by using Kstreams

[@TechPrimers]

Great question Palani. I will try making a video with examples

[@RealityCoupleVlogs]

@@TechPrimers Thank you..looking forward for ur video..! Would help for me to implement ... currently I m not using streams ..since don't know how to handle error

[@aswarthanarayanareddyb176]

Awsome content and great effort. Thank you so much Ajay. ♥️

[@TechPrimers]

Thank you aswartha. Glad you liked it

[@tianyuli3015]

Hi Ajay, would you mind explaining why you are using the spring kafka + spring cloud stream combination instead of the reactor kafka?

[@TechPrimers]

It's just for the ease of use. If we are using spring boot, the first obvious option is to try spring native libraries due to jar dependencies and their version maintenance.

[@alfred.g]

Best educational video! But what if there is a deserialization error? How to pre-log a message that caused the application to crash?

[@TechPrimers]

You can add a Spring interceptor to log those.

[@balajiarumugam1876]

Clean 💥👏👏. Can you provide list of the Mac shortcuts you used in intellij .. I couldn't able get from the bottom hints?

[@vinodhreddy6227]

Can you do one video for nats with complete end to endu message brocker

[@ed.d8065]

thank you! but I have a curious question can you explain what you did on 8:30?

[@jcrshankar]

Good one buddy.. please add error/exception handling

[@lifesamazinginindia]

Hello, great tutorial. One question: Did we define consumer groups, how is it handled in ksp ?

[@dheerajsachan5808]

Hey Thanks for the explanation, I really learnt something today :) , I have a question what if I wan to read from two kafka topics and send it to two kafka topics, in that case how will be my application.yaml. Thanks in advance.

[@test-nature]

Thank you very much

[@skullwise]

great video, as always. One question, if we were to replace the Kafka producer(service 1) with Kafka cloud streams(just like the other 2 services), should we have to create a method that returns a Supplier of type Supplier Please confirm :)

[@TechPrimers]

Nope. Supplier takes KStreams as input, it won't work for Producer. Kafka streams doesnt come at that microservice since there is no Kafka Stream usage at producer side

[@skullwise]

@@TechPrimers thanks for your reply. Does that mean kafka streams can be used only when we wanna consume data? And that why we are not able use in the producer service?

[@riteshyt8170]

Thanks a lot 😊👍

[@DhirenDash]

This was good. Although I am confused. Why would we use Kafka Streams for consuming a message? Should we not be using a normal consumer? Kafka Streams would be used only when we are consuming from and producing to a Kafka topic. Particularly useful when we need Exactly Once.

[@_ravi_kumar_gupta]

Great video. In extension to this can we get error handling, retry, rollback, transaction. It would be really helpful 🙏🙏🙏

[@chaithanyaprakash5070]

@Hi, did you get any chance to check error handling, retry , rollback etc.,

[@chaithanyaprakash5070]

@ Ravi Gupta

[@_ravi_kumar_gupta]

No

[@ArjunKumar-zu2kl]

Very nice explanation. What if we have so many Autowired classes then how can we replace it with Constructor Injection, do we need to keep adding those classes in class constructor, if yes then it will be too long when we have more classes autowired?

[@TechPrimers]

If there are too many that means the class needs to be broken down

[@maixuandai]

Very useful demo

[@rashmiacharya4669]

Nicely explained.

[@ayyappag7097]

excellent .. :)

[@SaravanakrishnanPk]

What's the tool that you used for creating arch visualization? It looks clean and nice.

[@TechPrimers]

Google slides

[@SaravanakrishnanPk]

@@TechPrimers Thanks, Can you share the google slide link (or download link) for this architecture diagram.

[@rahuldeshmukh4662]

Any example reference for spring cloud stream kafka binder producer which send stream to topic. As we don't have producer in kafka stream binder I need producer from Kafka binder to send data in key value pair.

[@ajit1105]

Hi Ajay, how can we send the inactive domain to inactive topic. Do we need to streambridge

[@jv88at17]

Hi, Can you explain how you will handle exceptions as default behavior is FAIL if there is a exception while processing.

[@adhithyanms3]

awesome

[@ILuvBilli]

public domain API is not up

[@shaikfazeel6812]

Hi... Just having a small doubt.. In my project scenario.. For example if there is a Café where purchase transactions take place.. Any transactions, then they are immediately published using kafka.. Now suppose there are multiple different Cafés in different regions.... So in that case how much topics should I use and how shall I split the data based on different regions?and How do I design my kafka architecture using spring boot? Kindly provide ur answer..

[@TechPrimers]

If you are segregating by regions, you decided that data from one cafe region cannot be shared with another region, then you will need separate kafka topics and processes for consuming and persisting them

[@sandeshrana7225]

How can we trace the messages in this Kafka architecture using Spring Cloud Stream? I want to store every messages in my own database i.e. do not want to use Zipkin or Jaeger.

[@omarkhalil4944]

Thank you very much!

[@debkr]

Does kafka streams pull one message at a time from the topic?

[@venkateshcharala2204]

Great job.. Is there any video or source code link for same kafka cloud stream consumer class with retry and dead letter queue? If any can you please share with me.

[@sanjeevprasadkotapati5783]

Thank you very much Ajay.i am a big fan of your content and this is one topic I have been looking for.is it possible to explain why we should use kafka streams when we can use traditional kafka consumers aka kafka listeners instead of kafka streams? Any advantage?

[@TechPrimers]

Like any other framework/library, it's the ease of use for transforming data once it's consumed. Kstreams price rich SDK which helps in performing the transformation much simpler and readable

[@acronis536]

how about a KTable example and a web controller serving requests from the KTable directly? :)

[@sanjaykantheti4002]

Hi Ajay, Very informative. As u already shown us kinesis so can u try an example using Google cloud dataflow??

[@TechPrimers]

Sure will try soon sanjay

[@joemonta1702]

SUPER & Thanks...

[@cleyxds]

it is possible to do this in a single project ?, instead of creating distinct projects for each cases. Thank you for the video.

[@TechPrimers]

If you do this in a single project why do we need queues for interaction then? If you are referring to source code, that's upto you. You can have it in a single repo

[@himanshubhatt9912]

Hi Ajay can you make some videos examples like this one on hexagonal pattern.

[@TechPrimers]

Sure Himanshu

[@DevOpsConnect]

Kafka streams cannot deserialized if there are two processor are active how to fix this ??

[@RaveendraGuntupalli]

Awesome Thanks

[@pruthirajmohanty3520]

Please create a video to address kafka stream with sleuth integration like standalone application which can pull the data and insert into DB

[@craftsmanengineering]

Many thanks :)

[@gouravkumar-or6jy]

This is a feedback bro, from next time before to show any demo related to spring please show its version because sometime we try to do but its deficult to create the same project at our end.

[@kunalsharma-zc2ho]

IntelliJ in action after a long period :)

[@TechPrimers]

Yes kunal. 😁

[@benedictdixon4622]

I keep getting the following on the processor? The class '[B' is not in the trusted packages: Any ideas? The consumer config also never seems to get the deserializer to be the one specified in the config (JsonDeserializer), so I think it is because of this? value.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer

[@TechPrimers]

Yes. Serialization is the key Benny. I had faced similar issues while trying it out before recording the video. You hav to Serialize and deserialize with same library

[@benedictdixon4622]

@@TechPrimers I believe I am? I am using the same Serializer and Deserializer that you are specifying in your application.yml files? It is just not picking them up to be used and so it is falling back on the ByteArrayDeserializer?

[@janamsoni3185]

Hey bro, i have added Cachable Static Assets with Spring MVC, but now i want to give button to user by which he can clear cache of website

[@TechPrimers]

Get the CacheManager instance and your cache from it and clear(). Take reference here stackoverflow.com/questions/56458748/reload-refresh-cache-in-spring-boot

[@JaNaMSoNi]

Are you sure it'll work with MVC ? Because here I'm talking about static resources browser's caching

[@TechPrimers]

Oh you mean the images and stuff. That will not work with the above solution. You need to clean up your storages in the client side via react/angular whichever you use

[@subrahmanyagurumurthy5083]

Nice one thank you. But I think you hacked the Get method to perform some operation!

[@TechPrimers]

Yes subbu. POCs are always hacked 🤓

[@aqibishaq885]

Sir one video for play framework of java please

[@sanjaykantheti4002]

Hi Ajay, Could you make a video on Kafka Connect and ksql db??

[@TechPrimers]

Sure Sanjay. Will do

[@sanjaykantheti4002]

@@TechPrimers Thanks Ajay for making a video on Kafka connect. It would be great if you could explain how to create custom connectors with a hands-on example & how to create custom transformers and use them in Kafka connect.

[@Virat77799]

Also my other question: in crawler you said you could use kstreams, how can kstreams read data apart from topic ( kstreams reads data to and from topic) Pls provide any example if kstreams could read from different source Correct me if my question are wrong ❌

[@TechPrimers]

Nope. Crawler doesnt read from kstreams. It can't use kstreams. Kstreams is a kafka dependant library from Kafka. Only processor n service uses kstreams

[@Virat77799]

So the value what you get after possessing the data could be saved in db right

[@TechPrimers]

Yes you are right

[@Virat77799]

However with plain kstream apis saving to db is not possible, and spring kafka streams is solution. Am I right?

[@Virat77799]

How do you save the data on consumer side to DB? In kstreams it's not possible right. Kstreams is all about topic to topic If so can you tell us the way to save data to db?

[@TechPrimers]

You have to leverage spring jpa or traditional way of storing to DB. But you can hook those methods into the kstream consumer like any other class implementation

[@JaNaMSoNi]

Intro ❤️

[@pitchireddyb646]

❤️❤️❤️

[@lenguyenkhoik14dn72]

Do you publish this repo on your github guys ?

[@TechPrimers]

Yes. The link is in the description

[@cinemaworld7931]

Use white color for your background bro

[@digisecureagent7679]

Another point, don't use same domain. Microservices should be independent. If same domain changes it forces all of the other depended models to apply changes and this blocks the whole product build pipeline.

[@manindersingh6371]

Thanks for making the video. Kindly choose those website who is active in world-wide. Trying to use your domain Website to access the domain but this website is not working in Canada. Anyway content was fine. Thank you

[@manindersingh6371]

and also share the proper link of branch in Github. Your github link was for home page of github.

[@TechPrimers]

Did not notice the missing link Maninder. Thanks for notifying. I have updated the github code in the description. here is link for Github code: github.com/TechPrimers/kafka-streams-microservices-example

[@saranyarupini5932]

Awesome work 👍🏻 Could you please try playing with multiple streams like configuring, merging, joining, processing etc? Because i had issue after merging 2 KStreams, records weren't flown to processor. cdcStreams.merge(edcStreams).process((ProcessorSupplier) () -> new CCCPStreamProcessor(config, esRepo), ConstantUtils.CCCP);

[@ishanisarkar8854]

❤️❤️