Apache Kafka on AWS (Amazon Managed Streaming for Apache Kafka / MSK) by Frank Munz

Hot off the press:-). I published a new article about streaming data pipelines with Kafka and Delta Live Tables. More advanced material once you seen this video here. That article comes includes a demo with a Twitter live steam and a bit of ML, aka Hugging Sentiment analysis: www.databricks.com/blog/2022/...
but now back to the Kafka basics:
Apache Kafka is one of the most popular open-source projects for building messaging and streaming applications. Kafka takes data from various sources, makes it available to different applications, and therefore helps to eliminate daily batch jobs.
Kafka plays an important role for Change Data Capture (CDC) and in the world of microservices. This presentation gives an overview of the new Amazon Managed Streaming for Kafka (Amazon MSK).
Based on knowledge gained from several Kafka implementation projects I will explain some of the technical underpinnings first. You will learn about brokers, topics, and Zookeeper. Then I will explain what makes Kafka special, analyse major pain points in on-prem Kafka projects, critically analyse how Kafka differs from Kinesis, and why the cloud is the best way to use Kafka.

[@sheungching7079]

Clear and Easy to understand. Thank you

[@wendelledgard3411]

I'm learning about Apache Kafka and I have to tell you that this is the best video I've seen so far. Thank you for the content!

[@frankmunz4262]

Thanks 👍

[@Raiseren]

Nice I can now get my ASMR boost while learning stuff.

[@sejalss7608]

Excellent articulation of the Kafka and AWS Managed Kafka service. Thank you for your simple and clear explanation.

[@frankmunz4262]

Thx to you Sejal for your feedback

[@saravmca]

This is best video for Understanding of Kafka and MSK . Very nice presentation and the explanation

[@frankmunz4262]

Thx 🙏

[@vasu840]

Nice Presentation. Clear and easy to understand about options provided in AWS for Kafka.

[@frankmunz4262]

Thanks - I appreciate your feedback! :-)

[@volvos60bloke]

Thanks for sharing this, great content, much appreciated.

[@frankmunz4262]

Thanks - I appreciate your feedback! :-) pls subscribe for more tutorials.

[@bholanathchowdhary9143]

Nice content Helped me a lot!!

[@vocebotafe]

Amazing presentation. Thanks!

[@frankmunz4262]

😊

[@RonakZala]

Thanks a lot for this. You made it so simple and clear!

[@frankmunz4262]

Thanks - I appreciate your feedback! :-) pls subscribe for more tutorials.

[@frankmunz4262]

thanks for sharing this!

[@gyansheelkumar7451]

great Content , so Helpful ...Thanks for making life so easier in a single video ..much appreciated man

[@frankmunz4262]

Thanks - I appreciate your feedback! :-)

[@frankmunz4262]

thanks for sharing it!

[@melsaied101]

Awesome, Thank you

[@frankmunz1659]

thanks a million for your feedback Mohamed!

[@sujitkumarpatel5105]

Very nice presentation, video and very good explanation/comparison on Kafka(AWS MSK) Vs. AWS Kinesis. Many thanks for creating this video and sharing. Appreciated.

[@FrankMunzAndMore]

I love your feedback Sujit. I am @frankmunz on twitter by the way. Feel free to share the vid.

[@ksjadon1]

Nice content. For better understaning, play with 1.5 X speed.

[@ersingharjot]

Hello Munz, Thanks for sharing.. I am looking for Live PM via Kafka data and apply for ML . you have any tutorial on this..

[@nambir]

Hi Munz, we have EKS and MSK in same vpc. .net container cannot access msk. how to debug the issue. Do you any video of EKS and MSK.

[@p.a8750]

The messages appeared in a different order from what you entered. Is that normal?

[@elijahgavrilov1686]

Great demo. Question is how to make Kafka available outside of private VPC, but for authorized clients?

[@nainaarabha9186]

I have one doubt. Can we host multiple kafka producers in one ec2 instance?

[@semalsherathia1034]

Why did the messages appear in different order when consumed than it was during producer? Is it because we partitioned the topic and hence the messages went to different partition and read in incorrect order?

[@frankmunz1659]

exactly. well explained!

[@camilas4067]

What kind of producer partition strategy do you have in MSK? I am looking for this information but I can't find it.

[@subashkannudass5950]

You should create more videos on Kafka & MSK, thanks

[@letslearnjava1753]

Hello Buddy , if you are interested to know MSK Kafka in-depth, you can refer the below videos -- Getting Started with AWS Managed Streaming for Kafka with in-depth service setup kzbin.info/www/bejne/eHeunoR3lst4lcU Creating a Serverless Apache Kafka(MSK) publisher using AWS Lambda kzbin.info/www/bejne/q53IhYmPbLGeg68 Using Amazon MSK as an event source for AWS Lambda kzbin.info/www/bejne/npDCapqfeb2Jm6c End to End Streaming Data Pipeline Using AWS MSK & AWS Serverless Services kzbin.info/www/bejne/ombcoYp4fbKmhLs

[@manasupadhyay8101]

Hey I am unable to connect my kafka from msk

[@arpankhetani7818]

Once we open Session manager , how did you went to kafka folder to run the kafka-topics and other commands in session ?

[@frankmunz4262]

see reply above pls. also MSK is a managed service. not EC2 with kafka tools. I used that EC2 instance just to explain how to interact, e.g. create a topic with MSK. Could have done the same with running create topic from my laptop.

[@jaimeduncan6167]

Thanks for the video, I fail to see how this is a fully managed service. The configuration changes one can do are pretty limited, the interface looks dated, the documentation is limited or not current, and worse: the basic operations can't be done from the interface. A system can not be fully managed Kafka when it can't create or delete a topic, can't set an expiration date, can't list the topics, can't create a sink. Basically is just a create cluster with nice defaults instead of a truly managed system. Am I missing something? I know the pricing is the pricing of a fully managed system.

[@dancummings8178]

Excellent vid. Very informative. I'm stuck at about 28:20 in the vid. My session starts with that same screen. "sh-4.2$ at the far left and then, I'm not sure what you did, but you end up with an ssm-user prompt and are now able to move into the kafka folder. How did you do that?

[@dancummings8178]

"su sudo" gets me basically to where I need to be, so that I can do all the necessary setup, but I refused to setup GIT 😄, just so that I can get kafcacat to work.

[@kalyanimuppidi498]

I have got a query as to how you accessed the kafka prompt in session manager during 28:27 time stamp in the video. I need the commands to do it. It will be of great help if you can provide with it.

[@arpankhetani7818]

I too got stuck at this point , how did from session manager , he went directly to kafka folder or prompt

[@frankmunz4262]

@@arpankhetani7818 hey guys, sorry for the late reply. I used this prompt from an EC2 instance. I had Apache Kafka installed there to be able to use the Apache kafka standard tools. if you can access MSK you could even run create topic etc from your local machine for demo purpose. hope that helps!

[@mongaunique86]

@Munz AndMore :- Could you please let me know how to connect to kafka cluster from other instance.

[@letslearnjava1753]

You have to make sure the security group allow the traffic, that's all. You can refer the below videos for details (the videos explanation the Network setup in-depth)--- Getting Started with AWS Managed Streaming for Kafka with in-depth service setup kzbin.info/www/bejne/eHeunoR3lst4lcU Creating a Serverless Apache Kafka(MSK) publisher using AWS Lambda kzbin.info/www/bejne/q53IhYmPbLGeg68 Using Amazon MSK as an event source for AWS Lambda kzbin.info/www/bejne/npDCapqfeb2Jm6c End to End Streaming Data Pipeline Using AWS MSK & AWS Serverless Services kzbin.info/www/bejne/ombcoYp4fbKmhLs

[@bitaihang3836]

what instance need to use? not found kafka folder in instance

[@frankmunz4262]

Hi Bi. It is a managed service, you don't need a Kafka folder. Please watch this video about how to get started with the MSK service. It is easy!

[@kennethlopez3857]

@@frankmunz4262 what video need to watch?

[@borokopshi]

I dont have instances (from 28 minute told about them in video). Can somebody explain plz how to add them?

[@FrankMunzAndMore]

Did you create an MSK cluster as explained 5 minutes before around min 23 in the video?

[@borokopshi]

@@FrankMunzAndMore yes, sure

[@FrankMunzAndMore]

@@borokopshi are you in the same VPC subnet or in a peered one? ports open in sec group for Kafka and ZK?

[@FrankMunzAndMore]

Find the slides here: speakerdeck.com/fmunz/apache-kafka-on-aws-amazon-msk

[@UdayKumar-xq1ve]

Dr.Frank thanks for the tutorial. I would like to know is there a tutorial, on how to start a session with SystemManger ? I am running through hell!

[@frankmunz4262]

Hell?

[@random-characters4162]

23:15

[@MrMadmaggot]

Give me a job :'/

[@mikecmw8492]

Too much architecture just get to the point

[@FrankMunzAndMore]

Mike. It's a video (and it is free!), so just fast forward if you like. What could possibly go wrong if people just click and do not consider "architecture"? Thanks for your feedback.