great tutorial mate, would love to see more 'hands-on' tutorials like this on youtube.
@li14lax2 жыл бұрын
I keep getting timeout errors:" WARN Client session timed out, have not heard from server in 10027ms for sessionid 0x0 (org.apache.zookeeper.ClientCnxn)" when I try to create the topic. Is there a certain permission I need to grant on either the MSK service or the EC2 instance I'm currently running? Both are in the same VPC. TIA!
@prithviboinpally21383 жыл бұрын
Great tut. Audio is really quiet though
@JohnnyChivers3 жыл бұрын
Thanks for watching Prithvi. First person to mention the audio. I usually edit using headphones so it wasnt noticeable to me, but I'll keep it in mine for future vids.
@SacrificialGoat943 жыл бұрын
Instead of opening kafka up to everything or using the public ip you can just specify the security group of the ec2 instance ?
@JohnnyChivers3 жыл бұрын
Yes, and absolutely advisable. I tend to make everything as open as possible on these demos, just so there are less chances of something going wrong for those following along. However, in the real world. Lock it down with security groups and Nacls.
@nikhilmittal2469 Жыл бұрын
Hi guys, getting below error when trying to write message on the kafka topic. Any help would be appreciated. ERROR Error when sending message to topic AWSKafkaTutorialTopic with key: null, value: 14 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) org.apache.kafka.common.errors.TimeoutException: Topic AWSKafkaTutorialTopic not present in metadata after 60000 ms. [Resolved] Its working for me now, had to change few security settings, "access control method"= unauthenticated access Encryption= TLS encryption
@Quivun3 жыл бұрын
Thanks! These demos help learning immensely!
@JohnnyChivers3 жыл бұрын
Thanks for watching and the comment Quivun.
@Explore_with_Azizul-maqsud Жыл бұрын
Awesome & Very nice content... but, unfortunately, the lowest sound ever in the youtube videos I experienced here..?? Please record with loud speaking. Thank YOU
@nozzom823 жыл бұрын
Hi Johnny, I am not getting the kafka bootstrap brookers - aws kafka get-bootstrap-brokers --region ap-southeast-2 --cluster-arn "arn:aws:kafka:ap-southeast-2:xyxdfasdfasf". For some reason I do not get any response. When I run the describe kafka command I do get the full response with Zookeeper id and cluster details.
@JohnnyChivers3 жыл бұрын
Hi Nozzom, Thanks for watching. What version of the CLI are you using? There was an issue with this exact thing on 1.16 and going to 1.17.9 or higher seems to solve it.
@nozzom823 жыл бұрын
@@JohnnyChivers I have not checked the version. Thank you for the suggestion.
@jimbrannlund46773 жыл бұрын
Upvoted partly for the accent. :)
@JohnnyChivers3 жыл бұрын
Thanks for watching Jim.
@user-rr3cc3uv2h3 жыл бұрын
Nice post, very straight foward!
@JohnnyChivers3 жыл бұрын
Thanks very much.
@Chavoosi3 жыл бұрын
Great tutorial mate. Thanks for your efforts
@JohnnyChivers3 жыл бұрын
Thanks very much.
@manasupadhyay81012 жыл бұрын
Hey your video is awesome can you teach how to connect MSK from my local i am running spring boot application
@SwapnilGangrade12 жыл бұрын
Did you get any leads? I am also looking to connect to MSK using spring boot application running locally
@matheusmandotti61213 жыл бұрын
Just incredible, congratulations !! Great video
@JohnnyChivers3 жыл бұрын
Thanks for watching Matheus.
@PoojaGupta-bi1kw10 ай бұрын
how can i connect it with springboot producer app?
@sridharshan1614 Жыл бұрын
we can do it in private to public like bastion host
@ajaykumarveldhi63852 жыл бұрын
Thank you for the Demo ...
@JohnnyChivers2 жыл бұрын
No problem Ajay.
@samjoshuaberchmans39013 жыл бұрын
Do you have any tutorials for Kafka/Spark streaming data ? Would like to view them. EMR will be having Spark , and we should be running spark from there. Consumer/Producer will be available in EC2. Would like to start a HBASE EMR to insert the values final into the HBASE tables. If you have any sample Videos for the same post it.
@JohnnyChivers3 жыл бұрын
I don't at the moment unfortunately. It is on my future lesson plans list for a series on EMR using both Spark and Flink streaming, but they won't be completed for another couple of months as it stands. Presently, there is a lot of Kinesis on the channel - but in future I will be addressing this imbalance with other streaming technologies.
@rostislav_engineer6 ай бұрын
thanks for this video!
@johnsaviello62403 жыл бұрын
I am getting the following error when I go to create the topic " Exception in thread "main" kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING". Any way to resolve this?
@JohnnyChivers3 жыл бұрын
Hi John, It could be a couple of things. Firstly double check that the zoo keeper connection string is correct. It will time out if it can't reach zoo keeper. If the connection string is correct then it's probably a security group or ACL issue preventing the connection. In the demo i open up the security group to all traffic. This is a bit excessive but my aim is to make there less issues when following along. So, ensure you can reach zoo keeper and you're not being blocked by in bound rules in a security group. After that, you may have an ACL issue depending on how your aws account is configured. There is a great article on stacked overflow about it here. stackoverflow.com/questions/58173102/aws-msk-timeout-when-creating-kafka-topic-with-acl-turned-on Hope this helps. Johnny
@johnsaviello62403 жыл бұрын
@@JohnnyChivers hey Johnny. Turns out you were right about the security group - I was trying to adjust the main SG I was using to accommodate my local machine IP. Then I added in an inbound rule for said SG and it worked. Thanks!
@JohnnyChivers3 жыл бұрын
No problem. Good news that it is working. Thanks for watching.
@nainaarabha9186 Жыл бұрын
I have one doubt. Can we host multiple kafka producers in one ec2 instance?
@ravindharanshanmugam88534 жыл бұрын
Thanks for the nice demo. one question from my end, how to stream data from filebeat to AWS MSK then to logstash? FileBeat => AWS MSK => Logstash (both file beat & logstash is in same vpc as of AWS MSK) I tried to connect as usual but hard luck. do we need to do any special config change?
@JohnnyChivers4 жыл бұрын
I am not overly familiar with FileBeat, but there is a config file which let's you specify Kafka brokers and suspect this needs updated to reflect the MSK information. Being inside the same VPC should make this easier. www.elastic.co/guide/en/beats/filebeat/master/kafka-output.html
@ravindharanshanmugam88534 жыл бұрын
Johnny Chivers thank you. Will try it out.
@arrjay38142 жыл бұрын
Looks like the path in the wget command to download Kafka is now invalid. Can you share the latest command to download Kafka? Thanks for supporting the community Johnny!
@maksymkoval17543 жыл бұрын
Thank you! It was a very helpful)
@JohnnyChivers3 жыл бұрын
Thank you very much
@nishokganesan68042 жыл бұрын
Can we connect this EC-2 instance from out local and send messages to topic?
@mqguitar063 жыл бұрын
Fantastic video, Thank you so much!
@JohnnyChivers3 жыл бұрын
Thanks for watching.
@AkashGupta-sy4pi3 жыл бұрын
Hi, I am getting following issue Messages are rejected since there are fewer in-sync replicas than required" Can u please inform me what might be the case
@JohnnyChivers3 жыл бұрын
Sounds like your minimum sync replicas have gone below the limit for the topic. It could be a configuration issue, but if your working to the exact settings I have? Then the cluster maybe unhealthy due to disc space running out. There is some debugging information about disc space and sync issues here docs.aws.amazon.com/msk/latest/developerguide/troubleshooting.html#troubleshooting-offlinepartitions-outofsyncreplicas
@AkashGupta-sy4pi3 жыл бұрын
@@JohnnyChivers All brokers have enough disc space, still it is throwing same error
@JohnnyChivers3 жыл бұрын
It may be one of those annoying cases where deleting the MSK cluster and rebuilding from scratch could be the best solution. I've had a couple of issues with EMR today where deprovisioning and reprovisioning solved the issue.
@melsaied1013 жыл бұрын
That was AWESOME
@JohnnyChivers3 жыл бұрын
Thanks for watching Mohamed.
@jkho83654 жыл бұрын
could you do 1 more on how to use consumer groups? Realise currently i get all. from beginning :)
@JohnnyChivers4 жыл бұрын
Yes. I'll add it to the list of future requested videos. I hope you found this one useful for getting started.
@jkho83654 жыл бұрын
@@JohnnyChivers yes it was, managed to do it
@danteiva84823 жыл бұрын
Thanks! All working properly!
@JohnnyChivers3 жыл бұрын
@dante thanks for watching and following along.
@adrianomorsa85472 жыл бұрын
Amazing tutorial
@JohnnyChivers2 жыл бұрын
Thanks adriano
@yovelb1313 жыл бұрын
hey man great video! im trying to run the "aws kafa describe-cluster ....." command but im getting the following error: An error occurred (NotFoundException) when calling the DescribeCluster operation: The requested resource doesn’t exist. any chance for help?
@JohnnyChivers3 жыл бұрын
Hi Yovel, thanks for watching. At what point are you running into the issue? Is it after the cluster has become available on the console?
@yovelb1313 жыл бұрын
@@JohnnyChivers hey man thanks for the response, well i got that error when i tried to use the aws kafka command to get the MSK ARN in the terminal after the cluster became available. i managed to just use the information displayed in the aws console to complete the video though
@mharongundayao62423 жыл бұрын
Amazing tutorial Thank you! Can I connect my spring boot app to amazon MSK cluster? using kafka connect?
@JohnnyChivers3 жыл бұрын
Thank you Mharon. Yes, you can use kafka connect. Unfortunately, there is no MSK managed service for this. There is however a great stack overflow post on this exact topic and a link to a video that is really helpful on the best answer post. stackoverflow.com/questions/59194786/kafka-connect-with-amazon-msk
@mharongundayao62423 жыл бұрын
@@JohnnyChivers Okay Thank you for this Do you suggest using Amazon MSK When you want your spring boot app to produce and as well as consume using the @KafkaListener annotation in the Spring boot app? from the amazon msk cluster?
@JohnnyChivers3 жыл бұрын
It certainly can be done. It's really a personal thing. I've always found that third party offerings of kafka remove the headache of managing your own cluster. The great thing about MSK is the abstraction. However, the trade off is you don't have as much control compared to building your own cluster from scratch. But, building one from scratch and then maintaining it is not a simple task.
@zheng51523 жыл бұрын
I love your desktop background image,how to get it!
@JohnnyChivers3 жыл бұрын
It's one of the desktops on Mac Dark Mode. If you Google "dark mode mac ocean" the background is available as a wallpaper on the 5/6th hit on image search. You can download it and use that as a desktop on Windows or Linux.
@Hypnock14 жыл бұрын
Hello i have a question. Can i do this on AWS Free Tier? I want to try this and AWS Glue ETL Streaming jobs but cant find any information about it. Nice video!
@JohnnyChivers4 жыл бұрын
The smallest MSK broker instance is a T3 small which isn’t in the free tier unfortunately. The pub/sub EC2 I spin up could be a micro instance in the free tier. It will just take slightly longer for the updates and installs than this demo. I had the larger instances in this demo spun up for 6 hours and it cost me 5 USD, so by using the free tier EC2 for pub/sub plus the smallest broker instance it shouldn’t be anymore that a few dollars. In terms of glue I have a getting started video on this channel kzbin.info/www/bejne/rWaXlnycfs99iNk
@Hypnock14 жыл бұрын
@@JohnnyChivers Thank you very much! Im trying to do some jobs using MSK and Glue ETL Streaming jobs into S3 (Parquet format). This will come in handy!
@diegobayona97853 жыл бұрын
Great post!!! Thanks!!!
@JohnnyChivers3 жыл бұрын
Thanks for watching Diego.
@adrianomorsa85472 жыл бұрын
Hi Johnny, Connect Amazon MSK cluster to on-premises web server?
@JohnnyChivers2 жыл бұрын
Yes, just be careful with latency.
@adrianomorsa85472 жыл бұрын
@@JohnnyChivers Which way do you suggest doing ?
@raghupathym253 жыл бұрын
Nice video man, can u help me on this error
@JohnnyChivers3 жыл бұрын
It's a bit of a hard one without more detail. If you had your zoo keeper server up, and the client connected it maybe a case you need to restart your zookeeper server and reconnect the client. Make sure you do this from separate terminal windows.
@raghupathym253 жыл бұрын
@@JohnnyChivers thank u, will try
@raghupathym253 жыл бұрын
@@JohnnyChivers can't find commands to stop zookeeper on windows
@JohnnyChivers3 жыл бұрын
Are not using msk and shhing into it? Same as the tutorial? If so, you'll be on Linux even if your using a Windows machine.
@raghupathym253 жыл бұрын
@@JohnnyChivers yeah i am following the steps, but getting WARN Client session timed out, have not heard from server in 10026ms for sessionid 0x0
@johnrbl64582 жыл бұрын
nice
@JohnnyChivers2 жыл бұрын
Thanks
@ramachandraraopamidimarri1578 Жыл бұрын
Video Volume is very low
@kevinmilner2072 Жыл бұрын
I really wish I could hear what you are saying.
@nareshrenukuntla78062 жыл бұрын
Great, if you can increase the font size for next videos it would be helpful
@JohnnyChivers2 жыл бұрын
Thanks for watching Naresh. I’ll increase the font size.
@c1i2s3c4o5 Жыл бұрын
yes, you could have maximized your ec2 putty windows , audio could have been little louder, also windows flipping was little faster, btw great content all in all!!!
@hoddybhaba6704 Жыл бұрын
audio quality is not good
@srinivasamaddipati3562 жыл бұрын
Too low audio and font size.
@samarthkulkarni6947 Жыл бұрын
aws kafka get-bootstrap-brokers --cluster-arn arn:aws:kafka:us-east-1:022942928579:cluster/demo-cluster-1 --region us-east-1 after hiting this command not getting output