54. Databricks | Delta Lake| Pyspark: Create Delta Table Using Various Methods

Azure Databricks Learning: Delta Lake
=======================================================
How to create delta table in databricks development?
Delta table can be created using various methods in databricks. In this tutorial, the most commonly used 3 approaches are covered
1. Using Pyspark without databricks
2. Using Spark SQL
3. Using dataframe with data
#Deltalake, #DeltaTable, #DatabricksDelta, #DeltaTableCreate, #SparkSQL, #PysparkDeltaLake, #PysparkDeltaTable, #SQLDeltaTable, #DataframeDeltaTable,#DeltaFormat ,#DatabricksRealtime, #SparkRealTime, #DatabricksInterviewQuestion, #DatabricksInterview, #SparkInterviewQuestion, #SparkInterview, #PysparkInterviewQuestion, #PysparkInterview, #BigdataInterviewQuestion, #BigdataInterviewQuestion, #BigDataInterview, #PysparkPerformanceTuning, #PysparkPerformanceOptimization, #PysparkPerformance, #PysparkOptimization, #PysparkTuning, #DatabricksTutorial, #AzureDatabricks, #Databricks, #Pyspark, #Spark, #AzureDatabricks, #AzureADF, #Databricks, #LearnPyspark, #LearnDataBRicks, #DataBricksTutorial, #azuredatabricks, #notebook, #Databricksforbeginners

[@manasr3969]

really good series with indepth knowledge

[@rajasdataengineering7585]

Thanks, glad it is helpful!

[@ranjansrivastava9256]

Very well explained Raja !!! Appreciate for your hard work bhai . !!!!!!

[@rajasdataengineering7585]

Thanks Ranjan!

[@mtomazza]

Thanks reaaally helped me

[@rajasdataengineering7585]

Glad it helped

[@manwarhossain3296]

Very nice. I like the sequence of videos you have created. It would be great if you can create some videos of advanced part of databricks.

[@rajasdataengineering7585]

Sure Manwar, will create advanced topics as well soon

[@sravankumar1767]

Nice explanation Raja 👌 👍 👏

[@omprakashreddy4230]

Crystal clear !!

[@rajasdataengineering7585]

Thank you

[@3a8saisamireddi61]

thank you!👍

[@rajasdataengineering7585]

You are welcome!

[@souravdey1227]

Great

[@tanushreenagar3116]

Nice sir

[@rajasdataengineering7585]

Thanks,Keep watching!

[@SureshBabu-kf5jx]

HI Raja, thank you so much for the wonderful videos. I have a question here. As there are 3 ways to define delta table. One is using pyspark and other is SQL and Dataframe. As Dataframe also comes under pyspark programming, Then what is the difference between these 2 ways?

[@kaladharnaidusompalyam851]

Thank you

[@rajasdataengineering7585]

You're welcome

[@rohansrivastwa827]

Nicely explained! Can you make video on how to create delta table using adls location- container -> folder -> folder_delta type location

[@rajasdataengineering7585]

Thanks 👍🏻 Yes we can create delta table using adls location also, which is called unmanaged table. In order to integrate adls with databricks, mount point to be created first. I have already poster video on how to create a mount point. Based on that mount point, the syntax is Create table emp(col1 datatype) Using delta Location Mount point itself contains storage account details and container name

[@karthikeyana6490]

Hi raja, very nice video. When u say that if we dont mention a location explicitly it will store it in hive meta store. So databrics comes with a hive metastore by default? I have seen all your videos in this playlist before this video but still couldnt figure that

[@rajasdataengineering7585]

Hi Karthik, yes databricks comes with hive metastore by default

[@JASWANTHSABBITHI]

can we add primary key and partition by

[@surenderraja1304]

What is difference between MANAGED delta table and EXTERNAL Delta table in azure databricks? Can we do insert , delete , update in both the types.

[@rajasdataengineering7585]

Managed delta table means storing actual data and table metadata both within databricks system (dbfs+ hive metastore). External delta table means storing the actual data outside databricks such as ADLS, hdfs, S3 etc while maintaining only metadata within databricks. Yes we can perform insert, delete and update on both types of delta tables

[@surenderraja1304]

On production which one is preferred. I feel delta tables on top of clean container is fit

[@rajasdataengineering7585]

External is better as we have more control on external storage

[@quiet8691]

Sir where to get the PPT and code of this lectures. It will be much benificial. Thanks

[@abhinavclasses8963]

@rajasdataengineering7585 When we create deltatable using dataframe approach then at what path will it be created?

[@rajasdataengineering7585]

We can specify a path while creation the table. If we don't specify the path, it will be created in dbfs

[@the_class_apart]

If we dont give the location then the tables are created in Hive meta store? is Hive part of DB architecture? I have a project in Azure and using ADLS Gen 2 for storage. where will the table be stored by default If I dont give the location while creating the tables?

[@rajasdataengineering7585]

If you don't give location, it would be created under dbfs, not under adls

[@asfiasultana3085]

Hi, I have a requirement to create a table in lake and there is another databricks script which drives through this table and based on the values of the table, it executes. And one point is the table should be truncate and load (every time the values will change based on need). Could you please help me in my approach?

[@rajasdataengineering7585]

Hi, sure I can Pls drop more info on requirement to email address [e-mail]

[@snehasiktachandra4357]

very great and helpful video. In the 3rd approach , i.e. creating delta table on dataframe, can we save the data as delta file instead of delta table ?

[@jitendrapradhan3016]

could you please help to provide the dataset'

[@GentleManAvenue]

How to set auto increment while create table and start 1

[@rajasdataengineering7585]

We need to use to identify column to generate surrogate key

[@limkangwei6339]

Hi, I am just getting started with this playlist of Delta Lake. Is there any resources or videos that you can refer me for setting up the tools/environment needed ? Thanks.

[@Umerkhange]

Suppose we have created a delta lake table and its schema gets changed over time as a result of merge schema. Do we need to update its definition code while running the cluster? every time or is there a way to create the table using the metadata available on the storage account.

[@rajasdataengineering7585]

No need for to update the definition. While writing data into the data, we can use merge schema option which will update the metadata

[@Umerkhange]

@@rajasdataengineering7585 Yes but when I stop-start the cluster, I need to refer to these delta tables again. so the code that I have written earlier becomes outdated because it does not contain these new column definitions.

[@prabhatgupta6415]

did u get the solution?@@Umerkhange

[@UmerPKgrw]

@@prabhatgupta6415no I have not find a dynamic way of doing it. You need introduce new columns in the code/table.

[@vishalaaa1]

Any delta lake project videos on databricks ?

[@rajasdataengineering7585]

Not yet posted any videos on this topic. Will try to create one soon

[@user-bc5nz2de2c]

If possible, could you share the link to the code used

[@kcsvenkat]

Hi, In DF to create a delta table, I need to give the location. where can I add the location?

[@rajasdataengineering7585]

Df. write.format("delta").save("location")

[@kcsvenkat]

@@rajasdataengineering7585 can we use the save option with saveAsTable ?

[@rajasdataengineering7585]

No, we can't

[@prabhatgupta6415]

df.write.format("delta").mode("overwrite").option("path",output).saveAsTable(DatabaseName.TableName) @@rajasdataengineering7585 Is it not correct sir? I am able to create the tble as well as files r getting stored in ADLS

[@prabhatgupta6415]

let me know

[@Umerkhange]

How to drop delta table using pyspark api's

[@rajasdataengineering7585]

For managed table, we can use drop statement of SQL. For unmanaged table, we need to delete the data folder along with drop SQL statement