Implementing Pyspark Real Time Application || End-to-End Project || Part-5 || HiveTable ||MYSQL

In this video we were discussed about storing our Final dataFrames into Hive Tables and Local MySQL
Hive is designed to handle large-scale datasets, including terabytes or even petabytes of data. If your DataFrame is expected to grow significantly in size or if you already have a large dataset, Hive's distribute
If your data ecosystem already involves other Hadoop components, such as HDFS, MapReduce, or Spark, Hive integrates well with these technologies. This allows you to leverage the broader Hadoop ecosystem for data processing, analysis, and integration with other tools and frameworks.
If you work with tools or applications that rely on SQL databases, storing your DataFrame in a local SQL database allows for seamless integration and sharing of data with these systems. Examples include reporting tools, business intelligence platforms, or web applications that interface with the database.
link for notes::
drive.google.c...
part1:
• Implementing Pyspark R...
part2:
• Implementing Pyspark R...
part3:
• Implementing Pyspark R...
part4:
• Implementing Pyspark R...
#azuredatabricks
#dataengineering
#dataanalysis
#pyspark
#pythonprogramming
#dataengineering
#dataanalysis
#pyspark
#python
#sql

[@viratchary3743]

Finally found one best channel for pyspark application project . Thank you sir

[@nguyenngocthien7368]

Huge respect! Following your videos helps me build an A-to-Z personal project, with the idea of building lichess data warehousing. I'm still waiting for your new videos 😄

[@prabhatgupta6415]

Amazing Sir Huge Respect

[@daivat3216]

One of the best channel for big data engineers, waiting for part 6 sir , when ?

[@shaasif]

thank you so much for your real time project explanation on 5 parts it's really awesome..can you please upload remaining multiple files and file name concept video

[@DataSpark45]

Hi actually that concept covered in the Data validation playlist. By creating metadata files. Thanks

[@shaasif]

@@DataSpark45 can you share you email id i want to communicate with you

[@tejathunder]

sir, please upload continuation for this project.

[@mission_possible]

Continue the series

[@nikhilgr7539]

Can you please upload new videos about multi threading and continuation of the project?

[@longhoinh3997]

push more videos sir 😄

[@user-bv5jn9lw2l]

HI Sir, Can you please make one video which cover Spark project build with Docker and deployed into AWS EMR, its really helpful to me

[@jitrana6813]

how can we use spark.sql instead pyspark dataframe select cmds, can you advise how can we do

[@DataSpark45]

Hi when you write df to hive generally we use df.saveasTable() . so that table will created in Hive environment then we can use spark.sql(select * from table). If you don't want to use HIVE then probably we use df.registerTempTable("TableName")

[@adrienseguorla7940]

Hi Sir, i hope are you good. Are you have github account ? Have a good day

[@user-bv5jn9lw2l]

HI Sir, Can you please make one video which cover Spark project build with Docker and deployed into AWS EMR, its really helpful to me