1.2 Billion Records Per Hour High Performance Kafka and Spark - End to End Data Engineering Project

[@CodeWithYu]

Please don't forget to like and subscribe! ❣

[@ss426]

Please inform the system requirement to run this project

[@sarahakinbami2959]

Thanks!

[@CodeWithYu]

You’re most welcome! Thanks for the support! ❤️

[@danieladeniyi1558]

You are the real MVP. Will sink my teeth into this

[@CodeWithYu]

Great! Don't forget to leave feedback too. Thanks!

[@thesungodzzz]

This is a greate video!!!! This is what I expected to see. A greate one, Yu!!! Please keep going. You are inspiring people on this field.

[@CodeWithYu]

Thank you! Will do!

[@shusants]

Wonderful content. Thank you so much Yu🙌

[@sathyans3852]

This is gold. Amazing work and explanations.👌👌👌

[@SoarSquads]

The view quality is so much better brother keep it up. God bless you and your family

[@CodeWithYu]

Only the best for you!

[@ravitejapothala977]

Great video! This is what they look in resume and intervie . Not just etl pipeline but Build scalable and fault tolerant one's. Requesting to include kubernetes in the scalable pipelines . Waiting for part 2

[@CodeWithYu]

Great point!

[@mohitupadhayay1439]

Great work brother

[@wiss1998]

Thank you again Youssef !

[@CodeWithYu]

It’s my pleasure Wiss!

[@DExpertz]

🔥I see Yu post a video, I comment first, then I watch the video. Keep up bro

[@CodeWithYu]

Legend! You know the drill! 🙌😀

[@TAUFIQUESEKH]

Very good content. Loved it. Thanks for making this

[@CodeWithYu]

Only the best for you!

[@orafaelgf]

I learn so much with your videos man. Keep going on. If possivel bring to content about Kubernetes ( airflow, spark, strimzi (kafka), data lake, etc.) Tks.

[@CodeWithYu]

Great suggestion!

[@NgynAn-dg3kp]

love the way you go high level architecture explain with blackboard to the, great 🔥🔥🧑‍🚒

[@CodeWithYu]

Glad you like it! More to come! 🔥

[@thesungodzzz]

Great video!!!!

[@mohammadbilalniazi5987]

جزاک الله خیرا

[@CodeWithYu]

Ameen. You too brother

[@parasagrawal1140]

what is the system requirements? can i make this project in my personel laptop?

[@ss426]

What is the system requirement to run this project in personal laptop?

[@ATHARVA89]

thank you

[@dhruvingandhi1114]

Can you please tell from where u get file of Docker-compose yml.

[@shubhambhandari431]

Why do we use checkpoint in spark? Can we use group id in kafka to maintain track of records

[@CodeWithYu]

Good question! It’s for fault tolerance. Think of failures during processing or crashes or similar situations, for Kafka to resume from a close reasonable point before failure, checkpoints help with that. Secondly, during stateful processing, checkpoints make the coordination easier. On the other hand, group id is just for coordination of offsets for message delivery. It has nothing to do with checkpointing or fault tolerance. Hope that helps!

[@PRUTHVIRAJ-wp9vu]

Great 👍

[@CodeWithYu]

Thank you! Cheers!

[@nadiiar75]

🔥

[@thirdojohnson4325]

hello boss, just wanted to know if you are available at your convenience for just a question on any of your socials if you dont mind me asking please. Thank you for all you do for us beginners

[@CodeWithYu]

Sure thing, How can I help?

[@flynnryder7823]

Can you please make your screen smaller, thanks

[@bharathdeva8849]

Hello yu. Great content in recent time❤️✨. You are valuing your audience interest, great ❤️✨. I can't see more people doing that. But I see that source code is only for members which 10 dollars. Coming tier 2 city 10 dollars is very high which is 10% of my salary 😢 which I can't afford. Could you consider my request of changing your membership fee?

[@CodeWithYu]

It’ll be available to all youtube members soon

[@shusants]

I agree with @bharathdeva8849. @CodeWithYu, that would be really helpful. Been your subscriber, really loved your content, keep on

[@salvityagi2288]

@@CodeWithYu when it will be available?

[@jay_wright_thats_right]

I like blackboarding but only if you CAN WRITE legibly. 🤣🤣

[@CodeWithYu]

lol sorry… will improve next time. I sometimes wonder if I write those! 😂😂😂😂

[@chirantharavishka5918]

sir my kafka configuration are correctly work beacuse i send 5 lks msg correctly producer are work but i run the sprk file then get the below error 25/02/02 05:34:30 INFO BlockManagerMasterEndpoint: Registering block manager 13567d606e23:44337 with 434.4 MiB RAM, BlockManagerId(driver, 13567d606e23, 44337, None) 25/02/02 05:34:30 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20250202053430-0001/2 on worker-20250202035206-172.21.0.6-36449 (172.21.0.6:36449) with 1 core(s) 25/02/02 05:34:30 INFO StandaloneSchedulerBackend: Granted executor ID app-20250202053430-0001/2 on hostPort 172.21.0.6:36449 with 1 core(s), 1024.0 MiB RAM 25/02/02 05:34:30 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 13567d606e23, 44337, None) 25/02/02 05:34:30 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 13567d606e23, 44337, None) 25/02/02 05:34:30 INFO StandaloneSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0 25/02/02 05:34:31 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20250202053430-0001/1 is now RUNNING 25/02/02 05:34:31 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20250202053430-0001/2 is now RUNNING 25/02/02 05:34:31 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20250202053430-0001/0 is now RUNNING 25/02/02 05:34:42 WARN ResolveWriteToStream: spark.sql.adaptive.enabled is not supported in streaming DataFrames/Datasets and will be disabled