Data Visualization with Spark Dataframes & Python Matplotlib | From Scratch

Description
---------
Hello Guys,
In this video i have explained, how you can create beautiful data visualizations from spark dataframes, to graphs and pbar charts using python matplotlib library.
Below are the other relevant videos from my channel:
Spark Streaming with kafka and HBase
---------------------------------------------------
• Apache Kafka with Spar...
Spark Streaming with kafka
---------------------------------------------------
• Spark streaming with K...
Installation of kafka on Cloudera quickstart VM
----------------------------------------------------------------------------
• Installing Apache Kafk...
Kafka Offset management
----------------------------------------
• Kafka Offset managemen...

[@joseluisbeltramone599]

Very good explanation, indeed! Thank you very much, Sir!

[@parthkodala]

Thank God... you helped me a lot....Thank you so much...

[@sagarsinghrajpoot6788]

Bro i must say , You explained things in amazing way. you have some quality to explain things in so easy way for less integient people ;) i subsribed your videos. I beleieve if you keep posting nice videos like this , it will be your big source of income as well :) all the very best

[@GKCodelabs]

Thankyou Sagar 😊..

[@sagarsinghrajpoot6788]

@@GKCodelabs Hey Bro can i connect personally for lil bit I need some guidance about real time Hadoop projects which is in your company and i will do practise same kind of set up at Oracle VM at my system....i only need a real story of your project at Banaglore ... ( Email - [e-mail] )

[@ninjawarrior_1602]

Hi, I find this video very helpful please continue the work or if you have any course please let me know. Thanx for your efforts

[@sagarsinghrajpoot6788]

if you have time you can share some knowledge on docker and containers ? These are demanding stuff now a days and you can add some quick python learn in 1 video and quick spark learn in 1 video kind of ? by the way i liked all your videos all are based on real senerio based and you explained to the point and very nice way ....

[@Dzejes]

Thank you for great tutorial!

[@naveenutukuru1463]

i have been following your videos it was so helpful.... but i have queries on big data project ? could you please help me out...

[@kancherlarohit1199]

hi sir, In general the pyspark is used for big data but for ploting data of small dataset we can directly use pandas to read and perform operations , finally you have converted the filter dataset again into pandas dataframe (toPandas) then what is use of using spark and though topandas() does'nt work for large amount of dataset(approx-1million rows) it becomes slow....? can you just tell me visulazing of large datasets with converting it toPandas

[@GKCodelabs]

Reports are never generated on raw data dumps, the Spark is used for all the heavy computations and produce the aggregated results, only necessary for business to get the required information, moreover in most of the cases the entire DWH platforms come into the picture to categorize the facts and dimensions over the aggregated data as well. The video shows the intergration where such scenarios can be handled by pandas dataframes. Hope that answers your question.

[@kancherlarohit1199]

Sir, Just can you help me with visualization of spark data without using topandas() for deployment purpose

[@nikitajha4352]

Thanku so much

[@ranasana9681]

Thank u so much, but i doesn't work for me, please can i contact u ? I have large number of data in my spark.df that 's why i connot convert it to pandas df....

[@indrajit798]

This is NOT working in databricks

[@soulamazing1228]

The point of using Apache Spark is to handle large datasets so why are you converting the values to pandas... Pointless video in real world scenarios.

[@GKCodelabs]

Conversion to Pandas is only done on final, aggregated and normalized data where volume is brought down only for what is required for "Visualization" Dont get confused with Data "processing" and "Visualization" You never run huge Spark jobs when a visual report is requested on the fly, that approach becomes "Pointless" Reports are pulled from aggregated data. And Matplotib is one of the approach, there are many other tools and approaches. Thanks for bringing this up btw, this will help others as well, in case someone has similar doubt.

[@ckhalifa_]

Please make your videos 720p at least. At 360p text is not readable

[@GKCodelabs]

All my videos are 1080p bro.! You can Change it manually, if it doesn't pick 1080 automatically.

[@ckhalifa_]

@@GKCodelabs unfortunately i dont see any other option besides 360p