Spark Structured Streaming Example (Kafka, Spark, Cassandra)

[@PampRedMonster]

I have been looking for a nice end-to-end streaming data pipeline tutorial forever. All tutorials were missing something. This is not. This has it all - it's clear, complete and really well explained. Could not recommend enough.

[@oussamafahd4205]

Thank you Davis for this beautiful mini project it has all what i've wanted to learn as a beginner in the Big Data Technologies

[@hkrishnan91]

World is a better place because of people like you. Very nice presentation of the topic and clear explanation of various steps involved. Thank you very much!

[@wewatchyourwebsite1]

Very well done! I like the fact that you show what you installed and how, and explain every step in perfect detail. Outstanding. I've avoided Scala because it was too complicated, now after seeing your video here, I moved my project from Python to Scala. The API is almost identical. I LOVED the fact that you show your steps for creating the build.sbt, creating the jar file and how to submit it. I also learned how to get rid of the annoying INFO in the log4j file. You also showed how to add the paths so I can run my project from a different folder. Some of this I already knew, but the fact you included it all is outstanding. I'm sure I could have found all this without your video, but the fact that you did it all, is just amazing. Thank you sooooo much for doing this.

[@bigdataprofessional1880]

Super presentation, Practical and thoroughly mentored for a kafka novice

[@jayc4849]

This is great. Exactly what I needed to see, and well explained. I absolutely recommend following the change to the log4j properties file, otherwise your spark output will make it really really hard to find your data as you first test in the console.

[@marcosoliveira8731]

Very good example. Made me think in a lot of ideas. Thanks a lot.

[@taimoorabbasi169]

Just the video i needed. Brilliant work. Really appreciate it 👏👏

[@gasmikaouther6887]

Hi Mohammed, can we reimplement it together

[@taimoorabbasi169]

I haven't worked on scala but I've implemented it on Python. We can do it together if you want. 💯

[@guilhermesilveira2742]

This is exatcly what I was looking for. Thanks!!

[@faaizshah5830]

This is fantastic work!

[@ambar752]

Nicely Explained Davis...Thanks !!

[@My_Tahar]

thank you @Davis for sharing, the tutorial is very clear

[@sankarreddy82]

Superb. Very much helpful. Thanks a lot.

[@savacano2858]

this is exactly what I am looking for!! Thanks!!!!!

[@abdulelahaljeffery6234]

quality content ... thank you Davis! subscribed

[@jagadeeshporalla4041]

Thanks Davis, if there are multiple queries with WriteStream and are done on different datasets derived from the actual Kafka stream using ( readstream) as you did… is this going to grab data for each query independently

[@eatonleo]

Could you please upload a pyspark version instead of scala? Really appreciate.

[@neutrinos401]

thank you so much for your great work

[@ДмитрийСиницкий-р2щ]

My thanks! Great tutorial!

[@devon299]

thank you! this is helpful to get started.

[@sumitkhattar9849]

Great, and Thankyou. keep doing.

[@aomo5293]

Great tutorial Thank you bro

[@amitmiglani5256]

Hi iam getting this error :- Caused by: java.lang.ClassNotFoundException: org.apache.spark.sql.cassandra.package$ can u help

[@nadiafekih2619]

Did you find a solution to this problem ? I'm facing the same issue!!

[@dji5028]

Same here, you tried to run the app on cluster mode ?

[@mouadriali3282]

Same Here >_

[@ryanrickerts5982]

See 34:10

[@srhreza]

Find proper dependency for your spark version.

[@alisahin1466]

you amazing man ... thank you soo much

[@mouhammaddiakhate3546]

How would we do the spark-submit if it was with PySpark ?

[@RajmohanBalachandran]

thanks a lot Davis

[@imosolar]

can I use UI / J notebook instead of command line to run it

[@samuelfernandoperezarpi4251]

Hi, is this lambda architecture?

[@nihedattia9477]

which type of machine VM u work on it ?

[@TechGeniusMinds]

Nice explanation

[@Cal97g]

Bro you heard of docker-compose? Installing kafka?

[@shrikantbhadane478]

good session, Can you teach same thing with PySpark?

[@kunjalsujalshah1992]

Amazing!

[@srhreza]

Awesome.

[@Raaj_ML]

Very helpful tutorial ! What is that Trigger.ProcessingTime("5 seconds") and when did you add it ?

[@hkrishnan91]

He missed to talk about it. They determine the batch interval. From the manual: The trigger settings of a streaming query defines the timing of streaming data processing, whether the query is going to executed as micro-batch query with a fixed batch interval or as a continuous processing query.

[@suryanshsingh6906]

thanks

[@enesuguroglu3542]

It would have been great if all codes implemented with Python.