Create a Kafka Cluster Using AWS MSK And Stream Data - Full Coding Demo

[@theamithsingh]

great tutorial mate, would love to see more 'hands-on' tutorials like this on youtube.

[@li14lax]

I keep getting timeout errors:" WARN Client session timed out, have not heard from server in 10027ms for sessionid 0x0 (org.apache.zookeeper.ClientCnxn)" when I try to create the topic. Is there a certain permission I need to grant on either the MSK service or the EC2 instance I'm currently running? Both are in the same VPC. TIA!

[@prithviboinpally2138]

Great tut. Audio is really quiet though

[@JohnnyChivers]

Thanks for watching Prithvi. First person to mention the audio. I usually edit using headphones so it wasnt noticeable to me, but I'll keep it in mine for future vids.

[@SacrificialGoat94]

Instead of opening kafka up to everything or using the public ip you can just specify the security group of the ec2 instance ?

[@JohnnyChivers]

Yes, and absolutely advisable. I tend to make everything as open as possible on these demos, just so there are less chances of something going wrong for those following along. However, in the real world. Lock it down with security groups and Nacls.

[@nikhilmittal2469]

Hi guys, getting below error when trying to write message on the kafka topic. Any help would be appreciated. ERROR Error when sending message to topic AWSKafkaTutorialTopic with key: null, value: 14 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) org.apache.kafka.common.errors.TimeoutException: Topic AWSKafkaTutorialTopic not present in metadata after 60000 ms. [Resolved] Its working for me now, had to change few security settings, "access control method"= unauthenticated access Encryption= TLS encryption

[@Quivun]

Thanks! These demos help learning immensely!

[@JohnnyChivers]

Thanks for watching and the comment Quivun.

[@Explore_with_Azizul-maqsud]

Awesome & Very nice content... but, unfortunately, the lowest sound ever in the youtube videos I experienced here..?? Please record with loud speaking. Thank YOU

[@nozzom82]

Hi Johnny, I am not getting the kafka bootstrap brookers - aws kafka get-bootstrap-brokers --region ap-southeast-2 --cluster-arn "arn:aws:kafka:ap-southeast-2:xyxdfasdfasf". For some reason I do not get any response. When I run the describe kafka command I do get the full response with Zookeeper id and cluster details.

[@JohnnyChivers]

Hi Nozzom, Thanks for watching. What version of the CLI are you using? There was an issue with this exact thing on 1.16 and going to 1.17.9 or higher seems to solve it.

[@nozzom82]

@@JohnnyChivers I have not checked the version. Thank you for the suggestion.

[@jimbrannlund4677]

Upvoted partly for the accent. :)

[@JohnnyChivers]

Thanks for watching Jim.

[@user-rr3cc3uv2h]

Nice post, very straight foward!

[@JohnnyChivers]

Thanks very much.

[@Chavoosi]

Great tutorial mate. Thanks for your efforts

[@JohnnyChivers]

Thanks very much.

[@manasupadhyay8101]

Hey your video is awesome can you teach how to connect MSK from my local i am running spring boot application

[@SwapnilGangrade1]

Did you get any leads? I am also looking to connect to MSK using spring boot application running locally

[@matheusmandotti6121]

Just incredible, congratulations !! Great video

[@JohnnyChivers]

Thanks for watching Matheus.

[@PoojaGupta-bi1kw]

how can i connect it with springboot producer app?

[@sridharshan1614]

we can do it in private to public like bastion host

[@ajaykumarveldhi6385]

Thank you for the Demo ...

[@JohnnyChivers]

No problem Ajay.

[@samjoshuaberchmans3901]

Do you have any tutorials for Kafka/Spark streaming data ? Would like to view them. EMR will be having Spark , and we should be running spark from there. Consumer/Producer will be available in EC2. Would like to start a HBASE EMR to insert the values final into the HBASE tables. If you have any sample Videos for the same post it.

[@JohnnyChivers]

I don't at the moment unfortunately. It is on my future lesson plans list for a series on EMR using both Spark and Flink streaming, but they won't be completed for another couple of months as it stands. Presently, there is a lot of Kinesis on the channel - but in future I will be addressing this imbalance with other streaming technologies.

[@rostislav_engineer]

thanks for this video!

[@johnsaviello6240]

I am getting the following error when I go to create the topic " Exception in thread "main" kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING". Any way to resolve this?

[@JohnnyChivers]

Hi John, It could be a couple of things. Firstly double check that the zoo keeper connection string is correct. It will time out if it can't reach zoo keeper. If the connection string is correct then it's probably a security group or ACL issue preventing the connection. In the demo i open up the security group to all traffic. This is a bit excessive but my aim is to make there less issues when following along. So, ensure you can reach zoo keeper and you're not being blocked by in bound rules in a security group. After that, you may have an ACL issue depending on how your aws account is configured. There is a great article on stacked overflow about it here. stackoverflow.com/questions/58173102/aws-msk-timeout-when-creating-kafka-topic-with-acl-turned-on Hope this helps. Johnny

[@johnsaviello6240]

@@JohnnyChivers hey Johnny. Turns out you were right about the security group - I was trying to adjust the main SG I was using to accommodate my local machine IP. Then I added in an inbound rule for said SG and it worked. Thanks!

[@JohnnyChivers]

No problem. Good news that it is working. Thanks for watching.

[@nainaarabha9186]

I have one doubt. Can we host multiple kafka producers in one ec2 instance?

[@ravindharanshanmugam8853]

Thanks for the nice demo. one question from my end, how to stream data from filebeat to AWS MSK then to logstash? FileBeat => AWS MSK => Logstash (both file beat & logstash is in same vpc as of AWS MSK) I tried to connect as usual but hard luck. do we need to do any special config change?

[@JohnnyChivers]

I am not overly familiar with FileBeat, but there is a config file which let's you specify Kafka brokers and suspect this needs updated to reflect the MSK information. Being inside the same VPC should make this easier. www.elastic.co/guide/en/beats/filebeat/master/kafka-output.html

[@ravindharanshanmugam8853]

Johnny Chivers thank you. Will try it out.

[@arrjay3814]

Looks like the path in the wget command to download Kafka is now invalid. Can you share the latest command to download Kafka? Thanks for supporting the community Johnny!

[@maksymkoval1754]

Thank you! It was a very helpful)

[@JohnnyChivers]

Thank you very much

[@nishokganesan6804]

Can we connect this EC-2 instance from out local and send messages to topic?

[@mqguitar06]

Fantastic video, Thank you so much!

[@JohnnyChivers]

Thanks for watching.

[@AkashGupta-sy4pi]

Hi, I am getting following issue Messages are rejected since there are fewer in-sync replicas than required" Can u please inform me what might be the case

[@JohnnyChivers]

Sounds like your minimum sync replicas have gone below the limit for the topic. It could be a configuration issue, but if your working to the exact settings I have? Then the cluster maybe unhealthy due to disc space running out. There is some debugging information about disc space and sync issues here docs.aws.amazon.com/msk/latest/developerguide/troubleshooting.html#troubleshooting-offlinepartitions-outofsyncreplicas

[@AkashGupta-sy4pi]

@@JohnnyChivers All brokers have enough disc space, still it is throwing same error

[@JohnnyChivers]

It may be one of those annoying cases where deleting the MSK cluster and rebuilding from scratch could be the best solution. I've had a couple of issues with EMR today where deprovisioning and reprovisioning solved the issue.

[@melsaied101]

That was AWESOME

[@JohnnyChivers]

Thanks for watching Mohamed.

[@jkho8365]

could you do 1 more on how to use consumer groups? Realise currently i get all. from beginning :)

[@JohnnyChivers]

Yes. I'll add it to the list of future requested videos. I hope you found this one useful for getting started.

[@jkho8365]

@@JohnnyChivers yes it was, managed to do it

[@danteiva8482]

Thanks! All working properly!

[@JohnnyChivers]

@dante thanks for watching and following along.

[@adrianomorsa8547]

Amazing tutorial

[@JohnnyChivers]

Thanks adriano

[@yovelb131]

hey man great video! im trying to run the "aws kafa describe-cluster ....." command but im getting the following error: An error occurred (NotFoundException) when calling the DescribeCluster operation: The requested resource doesn’t exist. any chance for help?

[@JohnnyChivers]

Hi Yovel, thanks for watching. At what point are you running into the issue? Is it after the cluster has become available on the console?

[@yovelb131]

​@@JohnnyChivers hey man thanks for the response, well i got that error when i tried to use the aws kafka command to get the MSK ARN in the terminal after the cluster became available. i managed to just use the information displayed in the aws console to complete the video though

[@mharongundayao6242]

Amazing tutorial Thank you! Can I connect my spring boot app to amazon MSK cluster? using kafka connect?

[@JohnnyChivers]

Thank you Mharon. Yes, you can use kafka connect. Unfortunately, there is no MSK managed service for this. There is however a great stack overflow post on this exact topic and a link to a video that is really helpful on the best answer post. stackoverflow.com/questions/59194786/kafka-connect-with-amazon-msk

[@mharongundayao6242]

@@JohnnyChivers Okay Thank you for this Do you suggest using Amazon MSK When you want your spring boot app to produce and as well as consume using the @KafkaListener annotation in the Spring boot app? from the amazon msk cluster?

[@JohnnyChivers]

It certainly can be done. It's really a personal thing. I've always found that third party offerings of kafka remove the headache of managing your own cluster. The great thing about MSK is the abstraction. However, the trade off is you don't have as much control compared to building your own cluster from scratch. But, building one from scratch and then maintaining it is not a simple task.

[@zheng5152]

I love your desktop background image，how to get it！

[@JohnnyChivers]

It's one of the desktops on Mac Dark Mode. If you Google "dark mode mac ocean" the background is available as a wallpaper on the 5/6th hit on image search. You can download it and use that as a desktop on Windows or Linux.

[@Hypnock1]

Hello i have a question. Can i do this on AWS Free Tier? I want to try this and AWS Glue ETL Streaming jobs but cant find any information about it. Nice video!

[@JohnnyChivers]

The smallest MSK broker instance is a T3 small which isn’t in the free tier unfortunately. The pub/sub EC2 I spin up could be a micro instance in the free tier. It will just take slightly longer for the updates and installs than this demo. I had the larger instances in this demo spun up for 6 hours and it cost me 5 USD, so by using the free tier EC2 for pub/sub plus the smallest broker instance it shouldn’t be anymore that a few dollars. In terms of glue I have a getting started video on this channel kzbin.info/www/bejne/rWaXlnycfs99iNk

[@Hypnock1]

​@@JohnnyChivers Thank you very much! Im trying to do some jobs using MSK and Glue ETL Streaming jobs into S3 (Parquet format). This will come in handy!

[@diegobayona9785]

Great post!!! Thanks!!!

[@JohnnyChivers]

Thanks for watching Diego.

[@adrianomorsa8547]

Hi Johnny, Connect Amazon MSK cluster to on-premises web server?

[@JohnnyChivers]

Yes, just be careful with latency.

[@adrianomorsa8547]

@@JohnnyChivers Which way do you suggest doing ?

[@raghupathym25]

Nice video man, can u help me on this error

[@JohnnyChivers]

It's a bit of a hard one without more detail. If you had your zoo keeper server up, and the client connected it maybe a case you need to restart your zookeeper server and reconnect the client. Make sure you do this from separate terminal windows.

[@raghupathym25]

@@JohnnyChivers thank u, will try

[@raghupathym25]

@@JohnnyChivers can't find commands to stop zookeeper on windows

[@JohnnyChivers]

Are not using msk and shhing into it? Same as the tutorial? If so, you'll be on Linux even if your using a Windows machine.

[@raghupathym25]

@@JohnnyChivers yeah i am following the steps, but getting WARN Client session timed out, have not heard from server in 10026ms for sessionid 0x0

[@johnrbl6458]

nice

[@JohnnyChivers]

Thanks

[@ramachandraraopamidimarri1578]

Video Volume is very low

[@kevinmilner2072]

I really wish I could hear what you are saying.

[@nareshrenukuntla7806]

Great, if you can increase the font size for next videos it would be helpful

[@JohnnyChivers]

Thanks for watching Naresh. I’ll increase the font size.

[@c1i2s3c4o5]

yes, you could have maximized your ec2 putty windows , audio could have been little louder, also windows flipping was little faster, btw great content all in all!!!

[@hoddybhaba6704]

audio quality is not good

[@srinivasamaddipati356]

Too low audio and font size.

[@samarthkulkarni6947]

aws kafka get-bootstrap-brokers --cluster-arn arn:aws:kafka:us-east-1:022942928579:cluster/demo-cluster-1 --region us-east-1 after hiting this command not getting output