Spring Boot | Kafka Schema Registry & Avro with Practical Example and Implementation |

[@sureshnamala6776]

Excellent tutorial on real time scenarios. Avro serialization is one of the real time scenarios used in the projects and also an important topic in the Kafka interviews. Thanks a lot Basant for the efforts put in, much appreciated!!

[@gopisambasivarao5282]

Thanks Basant. Appreciate your efforts. God Bless you.🙂🙏 Every week waiting for your videos. You are my Guru. 😊 The way of your explanation is awesome.

[@kalaiselvankesavel2971]

Thanks for making such a detailed video about schema registry!!

[@2RAJ21]

Thank you. i learned more about kafka docker compose.

[@ManishSingh-dj4yu]

Thanks for this tutorial.. your contribution to the developer community is invaluable.

[@aadiraj6126]

Thank you Basant bhai.. Lucid explanation, loved it👍

[@rahulrajsaini5670]

super video .. such a important topic explained so simply and so well .. thank you sir for all your efforts

[@rajyahoob]

Nice explanation, lot of other things to learn as well such as Docker, etc. Thank You as always.

[@malleswarrao3887]

Excellent sir no words to describe your knowledge sharing

[@user-0987-a]

Surpringly I had this kind requirement using avro object, many thanks

[@Javatechie]

Glad it helped!

[@ilronin804]

Wowww Awesome Thank you very much, so helpfull!

[@858871]

Bro your content is exceptionally great..

[@sgr7ss]

Nice explanation

[@mysavingclub]

Nice video you are putting lot of effort

[@Imjamalvali]

thanks bro, pls do like this videos, pls keep ur good work

[@AAlphonse682]

Thank you for a wonderful tutorial

[@anushbabu5023]

Thanks, looking for this one. Please cover for Kafka connect also😊

[@Javatechie]

Sure buddy

[@manjunathbabu6609]

"Let's get started" has a separate fan base..

[@itsmeibrahimm]

Excellent tutorial

[@suman8528]

great content ,waiting kafka connect and kafka stream

[@Javatechie]

Kafka connect i do struggled like anything but no results . Once successfully i will ran one source and sink poc then i will update buddy 👍

[@ivanhomziak]

Thanks!

[@Deepakblg97]

❤❤ Thank you so much sir 🙏

[@armandoruizgonzalez]

Thank's Basant

[@avisulimanoff1231]

Thanks for your useful tutorials. Short question please: what is the difference between you first change to avro scheme, that when you send Employee information without age/dob was validate successfully , to your second change, when you added the middle name, the validation failed. I didn't notice any change in the process for the two changes...

[@Javatechie]

Check I have explained in control centre ui the difference in schema evolution

[@amsfuy]

From 1:00 on you explain that in the plain Kafka world we need to build a new producer and consumer app once a class, whose objects we send to Kafka, changes. From 41:00 on you generate a new Employee class based on the changed schema and then rebuild producer and consumer as well. From my understanding, using schemas we still need to build a new producer and consumer app; so this is not an advantage of the whole schema overhead. The only advantage I see is that we do not manually change classes. Do I overlook something?

[@РоманПивоваров-ф7ш]

i'm stuck in the same, still dont get it..

[@Javatechie]

No your understanding is correct. It means we no need to rely on data like pojo/events even producer modified any payload structure still my consumer will not break

[@kevinameda2711]

@@Javatechie I used everything same as you but when it came to evolving the schema and dropping some of the fields, I got change in schema error 409. I am wondering because I used same properties as you. Also, another thing is, I have the following dashboard I am not seeing the messages even though the schemas and the topics are there. I am talking about port 9092/clusters/** but the 8081/subjects/** is showing

[@kevinameda2711]

This are the errors org.apache.kafka.common.errors.InvalidConfigurationException: Schema being registered is incompatible with an earlier schema; error code: 409 2024-04-23T09:29:46.988+03:00 ERROR 12236 --- [apache-schema-registry] [nio-8888-exec-1] o.a.c.c.C.[.[.[/].[dispatcherServlet] : Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed: org.springframework.kafka.KafkaException: Send failed] with root cause org.apache.kafka.common.errors.InvalidConfigurationException: Schema being registered is incompatible with an earlier schema; error code: 409

[@tusharrawal885]

Hello sir, I already followed you from last two years and you are awesome. Could you please help me for one interview question on threading. In you corrent project where you are using thread with example?

[@kanaillaurent526]

Very helpful!

[@utbhargav]

Thank you So much. why AvroSchema generated DTO has @Deprecated on the attributes ?

[@dattatraybharde2902]

Nice video ❤

[@mohammadturabali3870]

awsome

[@suprajainamadugu7083]

Thanks for the detailed information, But how to use different versions at consumer side while deserialising.

[@arunvijay2279]

If consumer in different microsevice, I need to generate avro class for all the consumers right ?. In future, if we need to add new field, first we need to add default as empty, is my understanding correct ?.

[@Javatechie]

Yes that’s correct but if your consumer is different project then you need to only get the updated schema from producer that’s it

[@saravanakumars52]

Yes exactly because whatever version of avro Object producer is sending consumer is going to consume that object as per version in schema registry when deserializing

[@santhoshg5342]

​@@Javatechieif producer sends new schema, does consumer need to generate the dto files from consumer end before it consumes. Until the new changes deployed to prod, is it uses old schema ? How exactly schema versioning works for consumer without any break if consumer is a separate external service ?

[@vverma011]

Could you please discuss more on Forward and Backward compatibility configurations.

[@deekandau4596]

This is awesome content thanks. One question , how do you share the schema across multiple micro services.

[@Javatechie]

It should be store in registry

[@deekandau4596]

@@Javatechie Sorry i didn't get which registry you talking about. Could you elaborate further. Can we store it in like a config server? Just thinking..

[@arunabhtiwari4771]

Hi Basat , what is default compatiblity for avro schema registry .

[@rathinmaheswaran]

Hi Sir , I need to convert an input xml file to avro schema dynamically instead of having it under resource folder and then build the Avro object and send it to Kafka topic .can you please help me with this.

[@pradeeptamishra9225]

Sir I have one question. If we put producer and consumer in separate project. Then the .avsc file we should keep in consumer project. Then any changes in payload structure in producer side the same we need to change in consumer side avro file. So at the end the consumer jar file needs to be re deployed to the server. So what is the benefit? Of schema registry. Please answer

[@Javatechie]

Actually this file should be place on some common or centralize place like config server or s3

[@letsCherishCoding]

Nice explanation. Just have one doubt, can we use this when we are storing employee info in database like cassandra as database schema is fixed there. And if we want to accomodate new fileds we will have to manually add them.

[@Javatechie]

But in the db schema you need to modify the field.

[@letsCherishCoding]

​@@Javatechie yes, I just wanted to know couple of real life use cases of using avro schema

[@Javatechie]

@@letsCherishCoding purpose of avro to avoid failure in schema evolution. Next you can think of any e-commerce or food delivery app who frequently do changes on their payload

[@letsCherishCoding]

@@Javatechie got it. Thanks

[@LakshmiprasadKota]

nice tutorial but if u do this concept by creating two differnt projects(producer and consumer) then it wil be more clear and prefect

[@Javatechie]

Yes I agree but to reduce the length of the video i kept both in the same project but what's the challenge here to separate consumers into different projects?

[@mithileshchandra2072]

@@Javatechie i think we need to generate the beans class using avro schema in both producer and consumer application and then need to run both app. I thought if we separate producer and consume and if we change schema in producer it should reflect in consumer app without re run but seems not possible we have to re run both app. pls correct me if I'm wrong?

[@Javatechie]

You are correct only need to build your app again that's the things you need to do

[@supriyakavuri1988]

Hi Sir, How can we achieve this kind of version in open-source Kafka ?

[@Javatechie]

Not possible supriya

[@srinivastadimalla1232]

Any help on this is appreciated Sir.. I subscribed to your channel now and would like to continue to have the membership and thank you very much for all the knowledge you are sharing to the community. How to make it work for build.gradle file Sir?

[@Javatechie]

Hello srinivas , Thank you for following javatechie. What's the problem are you facing with gradle ?

[@CanalGeekDev]

@@Javatechie hello, can you please demonstrate how to set this dependencies using gradle instead ?

[@aditvikramsingh1929]

Can you also cover Kafka connect and stream

[@Jothianand-g1v]

Great tutorial ! One question : Do we need to add any configuration related to SSL when using confluent cloud schema registry (https) ? I'm getting below error when my spring boot app is trying to connect to the schema url. SSLHandshakeException - PKIX path building failed: sun.security.provier.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

[@Javatechie]

Yes you need to add keystore and truststore certificate in producer and consumer configuration

[@Jothianand-g1v]

@@Javatechie Thanks for the quick response

[@shivamkhare-p5z]

which data type we can use for date in schema , Pls suggest ?

[@riyakumari8377]

I have a doubt what if we are not sure what kind of schema is coming from my database as my app might be handling different types of json data in that case Do i need to define schema for every json data one by one or is there any other way?

[@Javatechie]

Schema is nothing is the contract between producer and consumer buddy. For example to work as IAS you must need to clear the UPSC exam that's the rule or you can say contract set by the government. If you have multiple exams then you need to prepare for different subjects like schema

[@SurajKumar-l8d8y]

everything's working well, but my control centre often shut down itself after some time, need to start it again from docker desktop, why can be the cause?

[@ramanav3139]

can you guide me how to do kafka clustering in spring boot application how to handle

[@sudheerkumar-tp1mg]

1. In confluent dashboard we can able to see the sesitive data, how to secure it in one interviewer asked me this question, please suggest, also confluent is free or paid. 2. If consumer is down producer is producing messages continuously and the hardware resources are exhUsted kafka how can handle this situation

[@Javatechie]

Use secure Kafka cluster and yes it’s open source

[@sudheerkumar-tp1mg]

@@Javatechie Thanks for the reply basant sir, actually content ex: credit card numbers, social security numbers etc. which should display in amsked manner, please suggest

[@GMSGAANAMANI]

consumers are configured with appropriate heartbeat settings

[@gauravjaiswal9808]

hi sir can you please create a video on employee biometric login using Spring boot

[@attrayadas8067]

Can you please make a video on how to make a class immutable in java?

[@2000riddick]

I need the same in gradle build

[@talhaansari5763]

Currently we are using in our application.

[@GR-gk8dh]

14:44

[@el_yisusT]

Thanks a lot

[@gauravjaiswal9808]

hi sir can you please create a video on employee biometric login using Spring boot