Modern Spark DataFrame & Dataset | Apache Spark 2.0 Tutorial

[@chrisf1600]

Excellent - I very much enjoyed this clear and concise explanation. Thank you !

[@gounna1795]

Thank you! Very clear explanation!

[@adbreind]

UPDATE: here's a newer version of the notebook that works with some very recent small changes to the Structured Streaming API: databricks-prod-cloudfront.cloud.databricks.com/public/4027ec902e239c93eaaa8714f173bcfc/7597110674521248/3802148034621840/8012436201908551/latest.html If you're interested in the details, have a look at: github.com/apache/spark/pull/13653 ([SPARK-15933][SQL][STREAMING] Refactored DF reader-writer to use readStream and writeStream for streaming DFs)

[@trinsitwasinudomrod7500]

I love this video. Thank you!

[@SpiritOfIndiaaa]

Thank you so much , really wounderful , sir can you put/share some vid tutorial on how to do partitioning/custom partitioning and its configuration i.e. executors and number of cores on cluster and how to run for better optimization speed.

[@theCanadian808]

Does anyone know how I can get the code for this tutorial the link is broken above ? Thanks

[@TheVikash620]

Very informative. Thanks.

[@davidburt7645]

Looks like the workbook link no longer works. Is it possible to provide an updated link for this please?

[@ebottabi2347]

@Adam is it ideal to use DataFrame in Spark when you don't know the columns upfront, i am building an API service on raw Spark RDD rather using DataFrame. Is

[@anders077]

Hi, tried to run the example, but got an error: :32: error: not found: value spark spark.read.json("file:/databricks/driver/zips.json").createOrReplaceTempView("zip") ^ looks like something changed, tried both the old and the new notebook, thanks for the great video

[@adbreind]

If there's no "spark" value then most likely you're on an older (pre-2.0) version of Spark. Give it another shot making sure you're on 2.0 or newer!

[@shayshaswishes373]

Thanks for your vedio. Awesome Explanation. Can you please explain about "Symbol SQLContext is deprecated. Use SparkSession.builder instead" will be very useful.

[@arunbm123]

classOf([DataFrame] == classOf([Dataset[_] ) returns false on my laptop

[@adbreind]

Thanks for watching! This only works (and is only true) on Spark 2.x, not the experimental Dataset API introduced in 1.6, which might be why it doesn't work on your existing install. There are a few different ways you can try out 2.0: * Download/unzip a pre-built 2.0.0-preview tarball from the Apache Spark Project at spark.apache.org/downloads.html * Build from source by git cloning the open source repo from github.com/apache/spark.git and then building with the command build/mvn -DskipTests clean package * Use the free Databricks CE environment I used in the video by signing up at databricks.com/try then go Clusters > Create Cluster and choose Spark 2.0 from the version list.