Getting Started with AWS Managed Streaming for Kafka with in-depth service setup

Amazon Managed Streaming for Apache Kafka (Amazon MSK) allows you to operate Apache Kafka-based applications on AWS.
We’ll look at the Amazon MSK service in this vidoe, which can capture, store, and analyze real-time streaming data.
Prerequisite:
---------------------------
Learn AWS VPC , Subnet , Route Table, Internet Gateway with Hands-on Demo
• Learn AWS VPC , Subnet...
VPC with Public and Private Subnets in-depth intuition with Hands On
• VPC with Public and Pr...
AWS VPC NAT Gateway In-depth intuition
• AWS VPC NAT Gateway In...
AWS VPC ENDPOINT in depth intuition & hands on
• AWS VPC ENDPOINT in de...
Using Amazon VPC Endpoints to Access DynamoDB in-depth
• Using Amazon VPC Endpo...
Steps for the Lab:
-------------------------------
github.com/Sat...
Govern how your clients interact with Apache Kafka using API Gateway
aws.amazon.com...
Check this playlist for more Data Engineering related videos:
• Demystifying Data Engi...
Apache Kafka form scratch
• Apache Kafka for Pytho...
Snowflake Complete Course from scratch with End-to-End Project with in-depth explanation--
doc.clickup.co...
🙏🙏🙏🙏🙏🙏🙏🙏
YOU JUST NEED TO DO
3 THINGS to support my channel
LIKE
SHARE
&
SUBSCRIBE
TO MY KZbin CHANNEL

[@adityanjsg99]

Was struggling with confluent. This video solved most of my issues.

[@KnowledgeAmplifier1]

Glad to hear this @adityanjsg99! Happy Learning

[@sreesaradhi5709]

Great video on MSK service setup, appreciate you sharing the knowledge with us.

[@KnowledgeAmplifier1]

Thank you sree saradhi! Happy Learning

[@shrikantpandey6401]

Your videos is great. Please do end to end kafka spark project on aws

[@KnowledgeAmplifier1]

sure SKP! Stay Tuned

[@alfbaez]

Brilliant! 3 Things done! Brilliant knowledge sharing!

[@KnowledgeAmplifier1]

Thank you Sir Alf Baez! Happy Learning

[@jagdishbelapure7521]

awesome content, thank you so much for such a great explanation!

[@KnowledgeAmplifier1]

Glad to hear this @jagdishbelapure7521! Happy Learning

[@amitk6701]

Great video, thank you for posting!

[@KnowledgeAmplifier1]

You are welcome Amit K! If you want to explore more on AWS MSK , you can refer this playlist -- kzbin.info/aero/PLjfRmoYoxpNq-pjHW0n1AfkKNi_OOMUbi Happy Learning

[@felipeozoski]

Great video, thank you so much for thorough explanation :)

[@KnowledgeAmplifier1]

Thank you so much for your kind words @felipeozoski! Happy Learning

[@vipinsharma-jt3qt]

Please Create Video on MSK with SASL/Scram authentication from public internet.Without using EC2 instance in AWS.

[@JesusJimenez-xs5fe]

Excellent video, but now I have one question left, if I wanted to activate public access to the cluster it would have to be from IAM?, the idea is to be able to connect through the brokers

[@KnowledgeAmplifier1]

Hello Jesus Jimenez, if you want to enable public access for Kafka Broker , you can refer this -- docs.aws.amazon.com/msk/latest/developerguide/public-access.html

[@showmethemoney824]

this is cool, let's say cluster broke and nothing is working after consumer consumed some amount of message. Now my question is how do the consumer will consume the message from the left message and not from the beginning after restart.

[@KnowledgeAmplifier1]

Hello Manas Samantaray, if the cluster breaks and the consumer needs to resume processing from where it left off, 𝐨𝐟𝐟𝐬𝐞𝐭 𝐜𝐨𝐦𝐦𝐢𝐭𝐬 play a crucial role. By committing offsets after successfully processing a message, the consumer ensures that it won't start from the beginning upon restart. These offsets are also typically stored in a dedicated Kafka topic (__𝐜𝐨𝐧𝐬𝐮𝐦𝐞𝐫_𝐨𝐟𝐟𝐬𝐞𝐭𝐬) , ensuring that the consumer can retrieve its last position and continue seamlessly. For in-depth understanding on this topic , you can refer below video -- kzbin.info/www/bejne/mpKYkJZ-YrmrasUsi=fOg7NVS7jSYWu8Ma Hope this will be helpful! Happy Learning

[@akshaychampavat6795]

Its really great explanation . Can you please explain one thing . I understand MSK is serverless and we dont need to worry about managing clusters. Also client machine EC2 we can automate through cloud formation or terraform . I want to understand how we can automate the configuration you are doing inside EC2 instance like editing Kafka config . adding bootstrap servers to kafka config and all .is there any way to automate that . or how we will do that in production ? will it be manual effort or it can be done via automation . pls help in this .

[@jeffdeng1652]

Great Video - Can you comment on or make a video the differences of when to use MSK vs Eventbridge/SNS/SQS?

[@KnowledgeAmplifier1]

Hello Jeff Deng, I have noted in backlog ,here sharing the answer -- MSK (Managed Streaming for Apache Kafka) and Amazon EventBridge/SNS (Simple Notification Service)/SQS (Simple Queue Service) are both messaging services offered by Amazon Web Services (AWS) for building event-driven architectures. MSK is a fully managed, highly available, and scalable service that provides a managed Apache Kafka cluster, enabling you to build real-time streaming data pipelines for processing and analyzing data in real-time. MSK provides features like automatic scaling, automated software upgrades, and multi-AZ replication for high availability. On the other hand, Amazon EventBridge/SNS/SQS are fully managed services that provide reliable, scalable messaging infrastructure for building event-driven architectures. Amazon SNS is a pub/sub messaging service that enables message delivery to multiple recipients, while Amazon SQS is a fully managed message queuing service that enables decoupling and asynchronous communication between microservices. Amazon EventBridge provides a serverless event bus that makes it easy to connect applications together using events. The key difference between MSK and Amazon EventBridge/SNS/SQS is that MSK is a fully managed Kafka cluster that provides real-time streaming data pipelines for processing and analyzing data in real-time, while Amazon EventBridge/SNS/SQS are messaging services that provide reliable, scalable messaging infrastructure for building event-driven architectures. MSK is suitable for use cases that require real-time data processing and analysis, such as real-time analytics, monitoring, and fraud detection. EventBridge/SNS/SQS is suitable for use cases that require event-driven communication between microservices, such as decoupled architecture, fan-out, and fan-in messaging patterns. Ultimately, the choice between MSK and Amazon EventBridge/SNS/SQS depends on the specific requirements of your use case. If you require real-time data processing and analysis, MSK may be a better choice, while if you need reliable, scalable messaging infrastructure for building event-driven architectures, Amazon EventBridge/SNS/SQS may be a better choice. Hope this will be helpful! Happy Learning

[@HaiDo-b9d]

Thanks for the video. i have a question. to consume the data, i just need to use Get method?

[@PhongNgo-u9d]

thank bro🥰

[@KnowledgeAmplifier1]

You are welcome Phong Ngo! Happy Learning

[@chenguo8767]

Thank you for the great lecture. Would you mind comparing the pros and cons between this approach and the other one in your another video using lambda to publish to MSK? which one is more scalable to handle streaming requests. Thank you!

[@KnowledgeAmplifier1]

Hello Chen Guo, in case of publishing data in MSK Cluster using MSK Rest Proxy , the advantages are-- It's almost no coding approach , kind off plug & play as we have everything readily available from Confluent .. but the disadvantages are -- 1)it need maintenance & monitoring (if something wrong happens in ec2 & the code goes down , then it can't automatically up) 2)Also the approach is not scalable alone , you need to add load balancer to handle the production level traffic .. In case of Lambda based approach (API Gateway --> SQS -->Lambda-->MSK) , the advantages are -- 1)The producer is serverless 2)If something wrong happens in Lambda , the messages will not be lost , as they will be available in SQS 3)Alert using DLQ can be implemented easily .. On the other hand , the disadvantages with Lambda based approach are -- 1)code has to be written instead of using existing framework like Proxy 2)Lambda also has it's own limitation , it will throttle if the traffic is very high.. Hope this will be helpful! Happy Learning

[@PoojaGupta-bi1kw]

I'm new to MSK, I've a requirement that I should use MSK and my springboot producer app should be able to produce messages on MSK, what configurations do i need in my springboot app to connect with MSK?

[@gersonmoralesmarin1666]

Thank you for such amazing video!! I have a question is it possible to propagate authentication from API Gateway to MSK via the REST proxy. Example by using Authorization HTTP header (e.g. --header 'Authorization: Basic ') and avoid keeping credentials in properties files? Is there a way to configure REST Proxy server with Authorization HTTP header maybe using Iam. I appreciate any help you can provide :)

[@mavensept]

I keep getting confused in step 9 - is it MSK EC2 cluster or EC2 created for producer/consumer ? As in the beginning of step 9 - you mentioned its EC2 cluster where you install java and kafka. Then again you mention of creating topic in MSK EC2. Now moving to step 10 - why do start the producer from MSK EC2 ? If some one can clear the mist it will be useful for me to understand. Apologise my ignorance ..

[@SadaMandal-hd3nf]

how can How i can access msk kafka in my nodejs application ?I want to send message to msk from my nodejs application

[@KnowledgeAmplifier1]

I don't have much experience with Node.js , but you can refer this if it helps -- medium.com/@jm18457_46341/using-amazon-msk-with-iam-access-control-for-apache-kafka-and-node-js-with-kafkajs-71638912fe88

[@Data_Avenger]

Thanks a lot again Satadru for this nice explanation, however i could able to implement upto publishing and consuming messages from ssh, but when i am trying to run kafka-rest.properties after making changes, getting kafka-rest.properties: Permission denied No idea what i did wrong, but spent lot of time to fix, no luck yet