Spark SQL Tutorial | Spark Tutorial for Beginners | Apache Spark Training | Edureka

[@edurekaIN]

Got a question on the topic? Please share it in the comment section below and our experts will answer it for you. For Edureka Apache Spark Certification Training Curriculum, Visit the website: bit.ly/2KHSmII

[@sumitpatange9455]

I really liked your earlier video on introduction on Apache spark, but somehow this video is too fast paced for me, I didn't get much from this one.

[@edurekaIN]

Hey Sumit, thanks for checking out our tutorial. We understand that you are having a little trouble following. Do take a look at these blogs on the same topic. These might help clarify things: www.edureka.co/blog/all?s=spark%20SQL. And, please feel free to get in touch with us if you have any questions. Also, if you would like support and hands-on training guided by an instructor, we suggest that you enroll in our structured Spark training program here: www.edureka.co/apache-spark-scala-training. Our courses are led by industry practitioners who can clarify all your doubts and the course also comes with 24X7 support by our in-house team. Please feel free to get in touch with us if you have any questions. Hope this helps. Cheers!

[@matthewfeeley6226]

Thankyou for sharing your expertise, much appreciated.

[@edurekaIN]

Thanks for watching! Glad you liked it ! We are glad to have learners like you . Do subscribe our channel and hit that bell icon to never miss an video from our channel .

[@zeeshanabbas1578]

Great tutorial to start with Spark SQL. Thanks for the video. Spark Table, Temp Table, View or Temp View stores in memory or disk ? please answer?

[@edurekaIN]

Hey Zeeshan, Table , Temp table creates an in-memory table that is scoped to the cluster in which it was created. The data is stored using Hive's highly-optimized, in-memory columnar format. This is important for dashboards as dashboards running in a different cluster (ie. the single Dashboard Cluster) will not have access to the temp tables registered in another cluster. View , Temp view creates (or replaces if that view name already exists) a lazily evaluated "view" that you can then use like a hive table in Spark SQL. It does not persist to memory unless you cache the dataset that underpins the view Hope this helps :)

[@subhajitroy5850]

Appreciate the effort and thanks to Edureka for sharing the same. Have a question: In order to query, why temporary table/view was created? Is it optional or mandatory? Would like to understand the significance/reason behind bringing the same in this context. Thanks !

[@edurekaIN]

Hey Subhajit, Spark SQL supports operating on a variety of data sources through the DataFrame interface. A DataFrame can be operated on as normal RDDs and can also be registered as a temporary table. Registering a DataFrame as a table allows you to run SQL queries over its data. ables from the remote database can be loaded as a DataFrame or Spark SQL Temporary table using the Data Sources API. You temporary table will not persist in the storage. So, you need to apply all the transformations on temporary table and then convert it to Data Frame. Atlast, save the Data Frame in a file. Hope this answers your question. :)

[@sarathchandra1153403]

Hi Edureka, Great tutorial to start with Spark. Thanks for the video. Is it possible to share the data used in the tutorials in your blog ??

[@edurekaIN]

Hey Vema Rama, thanks for checking out our tutorial. We're glad you liked it. The data used in this tutorial are Edureka course artifacts that can be accessed by enrolling into our course here: www.edureka.co/apache-spark-scala-training. Hope this helps. Cheers!

[@lakshmibhargavi7390]

Hi edureka team, Thank you so much for the practical tutorial. I am a beginner in this concept and I came to learn a lot through this. Can you please let me know if there is any video that is related to "using SparkSQL in batch processing"? . The above video states step by step execution , Can we write the entire code in a batch and run it. if so can you please let me know how to do it.

[@edurekaIN]

Hey Lakshmi, There is a tutorial coming soon for this. Stay tuned!

[@anshumanster]

Great sir

[@edurekaIN]

Hey Anshuman, thanks for checking out our video. Do subscribe and hit the bell icon to never miss an update from us in the future. Cheers!

[@safiahmed5858]

Hi Edureka. You are doing a great job! Please Upload a video on Spark Mlib

[@edurekaIN]

Hey Safi, thanks for the wonderful feedback! We're glad you found our tutorials useful. You can get a primer to MLib in this tutorial: kzbin.info/www/bejne/nnWylGClf9pqeNE. Our course covers it in details. You can check out our course here: www.edureka.co/apache-spark-scala-training. Meanwhile, we have also communicated your request to our team and we might come up with something on the topic in the future. Do follow our channel to stay posted. Hope this helps. Cheers!

[@jiomohbile7610]

33:20

[@girmacentralbiz1527]

can spark sql take data from OLEDB and Excel source ?

[@edurekaIN]

Yes Spark sql can read data from excel . Please go through the below example import org.apache.spark.sql.SQLContext val sqlContext = new SQLContext(sc) val df = sqlContext.read .format("com.crealytics.spark.excel") .option("sheetName", "Daily") // Required .option("useHeader", "true") // Required .option("treatEmptyValuesAsNulls", "false") // Optional, default: true .option("inferSchema", "false") // Optional, default: false .option("addColorColumns", "true") // Optional, default: false .option("startColumn", 0) // Optional, default: 0 .option("endColumn", 99) // Optional, default: Int.MaxValue .option("timestampFormat", "MM-dd-yyyy HH:mm:ss") // Optional, default: yyyy-mm-dd hh:mm:ss[.fffffffff] .schema(myCustomSchema) // Optional, default: Either inferred schema, or all columns are Strings .load("Worktime.xlsx") Spark sql can read data from OLEDB . Please go through the below example val sqlContext = new org.apache.spark.sql.SQLContext(sc) // Construct JDBC URL val jdbcSqlConnStr = "jdbc:sqlserver://xxx.xxx.xxx.xxx;databaseName=WideWorldImporters;user=wwiuser;password=wwiuser1234;" Hope this helps :)

[@abhipsapatra3826]

great video..could you please share your previous videos link?

[@edurekaIN]

+ABHIPSA PATRA, thanks for the wonderful feedback. We're glad you liked our tutorial. You can check out the other tutorials in the series in this playlist: kzbin.info/aero/PL9ooVrP1hQOGyFc60sExNX1qBWJyV5IMb. You can access our complete Spark training by enrolling into our course here: www.edureka.co/apache-spark-scala-training. This course is led by industry experts and comes with 24X7 support and lifetime access to learning material. Hope this helps. Cheers!

[@anilkumarpindiganti4593]

edureka! nice

[@rajeshmrk7494]

I need Ppt for Download

[@edurekaIN]

We are glad to have learners like you. Please do share your mail id so that we can send the notes or source codes. Do subscribe our channel and hit that bell icon to never miss an video from our channel

[@ankitajain3243]

This session is too fast.