Apache Spark Real Time Scenario | Read BigQuery GCP from Databricks | Using PySpark

[@shubhamwaingade4144]

Moving file from dbfs to temp location was really helpful! Thank you.

[@varunpatnana7850]

facing issue when creating a data frame stating that failed to create credentials from file. Please Help.

[@AzarudeenShahul]

Can you pls share your code snippet and error message to [e-mail] . Will help you

[@ImanRahmat]

its possible using pyspark on premise hadoop which have interconnect with GCP ? and how to write table to bigquery from sparkdf.write to table ?

[@jeevarathinam2514]

Hi can you help me in writing the results back to big query

[@samk_jg]

Very Nice tutorial!

[@AzarudeenShahul]

Thanks for ur support

[@bertobertoberto3]

EXCELLENT

[@AzarudeenShahul]

Thanks much for your support 🙂

[@kkumar888]

We don't need to import anything for bigQuery?

[@AzarudeenShahul]

Yes, no need to import or install any libraries for bigquery.. I am using Spark 3 in this demo. Try and let me know if you face any issue

[@estebanramos9876]

@@AzarudeenShahul Hey Azarudeen! Great demo, but I ran into a problem. When I display(df) an error occurs "UncheckedIOException: Failed to create Credentials from file" even tho the df is correctly defined as a pyspark.sql.dataframe.DataFrame. Can you help me with this issue? Thank you.

[@kkumar888]

Customer -- Table 1 CUST_ID NAME REGION ADDRESS Product -- Table2 PROD_ID PROD_NAME PROD_DESC PROD_PRICE transactions -- Table 3 CUST_ID TXN_ID TXN_AMT PROD_NAME PROD_ID PURCHASE_DATE question 1 - Product which has been purchased by large number of consumers question2 - Product which has not been sold so far Need to give the answer in Pyspark, Can any one help me with the answer.

[@micahwendorf8808]

Hey Azarudeen. I just wanted to thank you for making these very informative videos, they are a life saver! I just got done watching kzbin.info/www/bejne/d3TTY5ylnrmqh80 as I was having trouble getting spark/scala set up in my local with IntelliJ and you provided amazing guidance on how to do so. Later in that video you said you were going to release another video on how to compile our scala file to a jar file that we can then upload to our spark clusters on our VMs. I was scrolling through your videos and haven't been able to find that one yet. Do you know if you ever ended up creating that one, and if so could you give me the link to it?? Again thank you so much for sharing your knowledge with us! @AzarudeenShahul