Apache Spark Real Time Scenario | Read BigQuery GCP from Databricks | Using PySpark

#spark #bigdata #bigquery #databricks
Apache Spark | Read BigQuery GCP from Databricks | Using PySpark
In this Session. we will learn how to read GCP BigQuery from Databricks using PySpark.
Apache Spark - Read Files {Scenario Based Real time use case}:
------------------------------------------------------------------------------------
Read Files Recursively into Spark DF - • Apache Spark | Spark I...
Read Data From URL - • Apache Spark | Databri...
Read Complex JSON File - • Apache Spark | Spark S...
Read and Write Excel in Spark - • Spark with Databricks ...
Blog link to learn more on Spark:
www.learntospark.com
Linkedin profile:
/ azarudeen-s-83652474
FB page:
/ learntospark-104523781...

[@shubhamwaingade4144]

Moving file from dbfs to temp location was really helpful! Thank you.

[@varunpatnana7850]

facing issue when creating a data frame stating that failed to create credentials from file. Please Help.

[@AzarudeenShahul]

Can you pls share your code snippet and error message to [e-mail] . Will help you

[@bertobertoberto3]

EXCELLENT

[@AzarudeenShahul]

Thanks much for your support 🙂

[@samk_jg]

Very Nice tutorial!

[@AzarudeenShahul]

Thanks for ur support

[@jeevarathinam2514]

Hi can you help me in writing the results back to big query

[@ImanRahmat]

its possible using pyspark on premise hadoop which have interconnect with GCP ? and how to write table to bigquery from sparkdf.write to table ?

[@kkumar888]

We don't need to import anything for bigQuery?

[@AzarudeenShahul]

Yes, no need to import or install any libraries for bigquery.. I am using Spark 3 in this demo. Try and let me know if you face any issue

[@estebanramos9876]

@@AzarudeenShahul Hey Azarudeen! Great demo, but I ran into a problem. When I display(df) an error occurs "UncheckedIOException: Failed to create Credentials from file" even tho the df is correctly defined as a pyspark.sql.dataframe.DataFrame. Can you help me with this issue? Thank you.

[@kkumar888]

Customer -- Table 1 CUST_ID NAME REGION ADDRESS Product -- Table2 PROD_ID PROD_NAME PROD_DESC PROD_PRICE transactions -- Table 3 CUST_ID TXN_ID TXN_AMT PROD_NAME PROD_ID PURCHASE_DATE question 1 - Product which has been purchased by large number of consumers question2 - Product which has not been sold so far Need to give the answer in Pyspark, Can any one help me with the answer.

[@micahwendorf8808]

Hey Azarudeen. I just wanted to thank you for making these very informative videos, they are a life saver! I just got done watching kzbin.info/www/bejne/d3TTY5ylnrmqh80 as I was having trouble getting spark/scala set up in my local with IntelliJ and you provided amazing guidance on how to do so. Later in that video you said you were going to release another video on how to compile our scala file to a jar file that we can then upload to our spark clusters on our VMs. I was scrolling through your videos and haven't been able to find that one yet. Do you know if you ever ended up creating that one, and if so could you give me the link to it?? Again thank you so much for sharing your knowledge with us! @AzarudeenShahul