Realtime Voting System | End-to-End Data Engineering Project

[@CodeWithYu]

Don't just cast your vote in the comments, make your mark on our channel too! Hit that like button as if it's your favorite candidate, share this video like it's a winning manifesto, and subscribe like it's your civic duty. Together, let's form a coalition of knowledge seekers and tech enthusiasts! 🌟 #JoinTheMovement #LikeShareSubscribe

[@dotproduct763]

These are just what I needed for my upcoming data engineering system design interview. You're godsent! Great content, please keep it up and help many juniors like me.

[@CodeWithYu]

I wish you best of luck in your upcoming interview. Break a leg, I AM ROOTING FOR YOU!

[@vishalkamlapure3344]

Really cool project, haven't seen any project on internet with such clear flow and good explanation. This guy is really genuine in terms of making quality content. Keep the good work Man ! Thanks for the learning.

[@chuchip]

Excellent!!. Really great. I'm bored of videos that teach theory but don't show real cases. Thank you very much for your excellent work. I think this is the way to really learn

[@CodeWithYu]

Absolutely! The best way to learn is by doing, theory helps too though!

[@ericlaw7588]

Hi Yu, can you make a separate video on environment setup especially on docker component , how does it work , example to compose your yml file etc. would like to see more on pyspark integrated with aws , dbt , snowflakes , airbyte , mage etc projects :D

[@CodeWithYu]

Sure, will do!

[@sassydesi7913]

Yes this will be helpful

[@benjoanc]

Yes, especially the environment set up video

[@resitk7272]

any updates?

[@ArkoChakraborty4493]

Really need this. I have worked in cloud and there they have these services tightly integrated without much thought. How do I setup a local system with these services working in tandem. You make yaml files so fast. Give a way so that it's easier to grasp that and we can make end to end projects.

[@shakib.polock]

Thank you for this. Please spend some more time explaining the docker-compose file.

[@CodeWithYu]

Thanks for the feedback. Will do!

[@mohammadvohra7315]

Really amazing, love you bro Keep going!!

[@CodeWithYu]

Thanks Mohammad! Don't forget to subscribe, like and share!

[@ajithkumar-iy6mm]

Hi YU, I have query , in the dashboard how it is caculating total voters and the table votes sorted in the state ?

[@workhardforyourfamily4826]

Good content.....we need more projects like this

[@CodeWithYu]

More projects incoming!

[@mauricecolon4359]

DO we have to set up a remote server to run Py spark Scripts if so you completely missed that part.

[@albertopedro8632]

Obrigado por partilhar!

[@wiss1998]

Thank you for the video.

[@CodeWithYu]

You're welcome! :D

[@poooyvvvsagkkohhgvgtt]

AT 56:25 I got error saying This account may not have enough CPU cores to satisfy this request Estimated available: 0, requested: 4 what can i do

[@CodeWithYu]

You should probably reduce your architectural configurations. Your laptop or pc is not powerful enough to host the configs

[@rasmusandreasson1548]

As usual, Great WORK!! Would be really cool to have video about Microsoft Data Fabrics, if that's possible! :D

[@CodeWithYu]

Sure thing. Working on it! 😀

[@jaswanth333]

Hey Yusuf, Really appreciate the amount of time and effort you put into making this video. I have gained knowledge on the spark streaming and key elements to be considered while configuring the streaming process. Quick question: What is your approach while creating docker compose files. I tried to extend airflow image myself by looking in online but I couldn’t find the instructions in official docker websites or any other website for the matter of the fact. Do i need to run docker compose up -d -build for apache-airflow custom image everytime ? If i dont then I see that my init container is moving into exited state with logs stating that”run airflow db init and migrate” Could you please make one dedicated video on docker with some tips for beginners like me so that we can kickstart the projects with ease

[@CodeWithYu]

You can't extend official images as they're IP of their publishers, however, you can create a custom image and publish to your private repo or make it available to the public on docker hub. You only need to tag the docker file see (docs.docker.com/build/building/base-images/#create-a-simple-parent-image-using-scratch). Once that is built, you can publish it to docker by tagging it with a version. For the video, I will definitely add it to my list of upcoming videos! Thanks for the feeback. Protip: You can influence the video being posted by voting in the community polls! 😉

[@lovemary1461]

This was really helpful.. thank you

[@CodeWithYu]

Glad it was helpful!

[@eduardosteps]

Nice video, Yu. Congrats! Are you planning some tutorials with RabbitMQ and Redis?

[@lucaslira5]

Thank you for the video. Can you make a video using databricks + dbt core and azure devops please?

[@CodeWithYu]

Sure thing Lucas, will do!

[@yashamesar9596]

Hi Yu, Can you please make a video on environment setup of all components used in project please

[@OmkarPatil-n9e]

Hi Yusuf , Just wanted to know in the real production , what challenges we can face while using spark streaming? Thanks for the beautiful course , I had learned Kafka and this video helped in revising the kafka i also added few addons here like idempotence and few other things just to make sure the message gets deliver once :P

[@sunitagajurel5799]

is there any specific reason for uisng producer.flush() for voters and producer.poll() for votes ?

[@CodeWithYu]

Nothing specific. However, using flush will wait for all messages to be delivered effectively making the process synchronous but poll will not block, in this case, asynchronous. So depending on your usecase, your choice!

[@johntriple]

How much ram should i give to docker to run this? i have 16gb ram on my laptop. Do you think i can run this project?

[@CodeWithYu]

You can start with 6GB, if you notice any performance issues from your containers, you can increase it to 8.

[@claricestarling9668]

can someone please tell me where is he building the broject? is it streamlit?

[@prathikvijaykumar616]

after docker compose up -d. The localhost:9092 and localhost:5432 dont respond at all. the containers are healthy are running. I am running it in windows machine itself

[@adhiana848]

Hey Yu could u help me, i got an error while running "python spark-streaming.py" java.lang.RuntimeException: java.io.FileNotFoundException: java.io.FileNotFoundException: Hadoop home directory C:\bin\hadoop\bin does not exist

[@aliyazdan6282]

same did you got any solution for that?

[@yellowtofu5751]

Thank you so much for your content Yu! Really learn a lot from you. Will it be possible to see some use cases for mongodb, or some other nosql databases? Thank you!

[@CodeWithYu]

Absolutely! I will share nosql videos soon. Don't forget to turn on notification bell :D

[@AA-kr4hx]

I get again and again same error password authentication failed for user “postgres”

[@mauricecolon4359]

I'm having so many issues with executing scripts in a containerized environment its giving me a massive headache but I'm sure when I'm done with this. I will have a better understanding of it

[@mauricecolon4359]

I hate when I get to the END AND I GET STUCK

[@swarefrector7545]

@@mauricecolon4359 hello maurice, were you able to do this project completely

[@Omer698]

I think you should do a more beginner friendly project going step by step explaining what/why/how you're doing what you're doing that build up to these complex projects. That way students can just refer to the beginner one and understand the "what/why/how" of your projects

[@mauricecolon4359]

most of the projects on the tube lack sequential explanations of the thought process and necessary components for a full understanding of the micro-services but I promise you when I learn all this stuff, I will make Content that doesn't skip pass anything. I am starting to believe most of these projects belong to the institutions and not individuals cause there's so much missing from them!

[@liyamurai5235]

Hi Yu! Im a 2nd year computer science student. I’m looking to do something similar to this for my thesis topic. I’d firstly like to ask for some clarification about this project. Would you categorise this more towards streaming or pipelining? Secondly, if I wanted to utilise tools to ensure the integrity of this voting system, what tools could I use? Thirdly, where is the voting interface, for voters/voting ballot interface? Lastly, if you don’t mind me asking, since this is would likely be my research topic, would you be okay with me reaching out to you off of KZbin (perhaps Instagram or the like)for some more information or help in some sense? Thank you !

[@saber3112]

why you are having kafka again after spark..why not save the output from spark in datawarehouse and then visualize

[@CodeWithYu]

The destination can be anywhere you wanted. I chose Kafka because I want to reduce the number of applications integrated to the solution and the possibility of other systems listening in realtime to the events written on Kafka. You don't have to use the exact same approach, you can slice and dice it up as you wish.

[@saber3112]

@@CodeWithYu thank you..can we have one project for twitter hastag analysis

[@AsjadPuthawala]

from where we get data?

[@aliel-azzaouy7007]

Sir, first, I would like to express my gratitude for all these videos; they have been very helpful. I just have one question, and I hope you can provide an answer. When I run my script on my local machine, the connection fails. However, when I create images for that script, things are successful. Could you please guide me on how to fix this problem? Additionally, when I ping the IP address of my 'postgres' container from my local machine, the connection fails. If possible, could you create a short video tutorial on Docker networking? That would be greatly appreciated. Thank you.

[@aliel-azzaouy7007]

@CodeWithYu

[@CodeWithYu]

Which connection failed? Can you provide more information about this error? I can't be of much help if I don't know the error. As for the postgres ping, which OS are you using?

[@nadiiar75]

🤗

[@amacodes7347]

Please Yu dark screen is best so we can watch for longer hours.

[@CodeWithYu]

Absolutely, my sincere apologies. One of the viewers requested light screen mode and I thought this would be applicable to all. I will definitely switch back to dark mode as I can't code for longer hours with light mode too! Thanks for this! 🌟

[@ismailemir54]

7.40

[@morshedsarwer]

Coding with commentary 🤣 Please continue...

[@CodeWithYu]

😀

[@okwusman]

Also I get this error from kafka-python: from kafka.vendor.six.moves import range ModuleNotFoundError: No module named 'kafka.vendor.six.moves' I am using python 3.12