Microsoft Fabric Spark Notebook - Learn PySpark and SparkSQL in 2hr(Beginners Course)

Microsoft Fabric End to End Solution - PySpark and SparkSQL Full Course For Beginners
Note:- Download the whole folder
Dataset and Code Link- 1drv.ms/f/s!Ak57yG1_Qg0iey_2D...
---- Video Outline ----
0:01 Introduction
0:15 Loading File From You Local System to Lakehouse
1:40 Reading CSV File Using PySpark in Microsoft Fabric Notebook
4:02 Select and SelectExpr Function in PySpark
7:30 WithCoumn, WithColumns and WithColumnRenamed Function in PySpark
13:02 Where and Filter Function in PySpark
15:38 GroupBy and Aggregation Function in PySpark
19:45 Joining Two Dataframes Using PySpark
25:34 Union and UnionByName in PySpark
32:06 Drop, DropDuplicates and Distinct Function in PySpark
40:17 Write/Load data to a delta Table and Parquet File Using PySpark
45:16 Overwrite, Append , Ignore and Error Write Mode in PySpark
53:40 Create Delta Table and Load Data through Dataframe in PySpark
01:00:34 Create Delta Table with Partition and Load Data into it Using Dataframe in PySpark
01:10:57 Difference Between Managed and External Table
01:19:34 Create delta table using SparkSQL
01:23:15 Create delta table with Partition using SparkSQL and Insert Data from a View
01:28:48 DML Operations over a delta table using SparkSQL
01:34:17 Temporary and Global Reporary View in PySpark
01:38:32 Data Transformation Over a delta table using SparkSQL
01:50:01 Reading Delta table using PySpark and SparkSQL
Microsoft fabric data pipeline script activity: • 03 Microsoft Fabric Da...
Microsoft Fabric data Lakehouse end to end project - • 003 Microsoft Fabric L...
If you want to know more about Microsoft Fabric data Warehouse, Please watch below videos-
1. Create Microsoft Fabric Data warehouse: • 001 Microsoft Fabric D...
2. load data into Microsoft Fabric data warehouse using Data Pipeline: • 02 Microsoft Fabric Da...
3. Create table in Microsoft Fabric data warehouse: • 003 Microsoft Fabric D...
4. load data into Microsoft Fabric data warehouse using Copy into Command: • 003 Microsoft Fabric D...
5. Transform and load data into Microsoft Fabric data warehouse using T-SQL Procedure - • 004 Microsoft Fabric D...

[@KaraokeVN2005]

Thanks for your video, it's very helpful.Love from Vietnam!!!!!!!

[@DataVerse_Academy]

Thank you!

[@amiyarout217]

Great learning 😀😃

[@RiseHigh1988]

Thank you.

[@azobensadio260]

Thanks so much! this is great

[@DataVerse_Academy]

Thank you 😊

[@narendrareddy3576]

Great Video, many thanks for make it for us 😊

[@DataVerse_Academy]

Thank you! 😊

[@DataVerse_Academy]

Dataset and Code Link- 1drv.ms/f/s!Ak57yG1_Qg0iey_2DKTSnObu6Ts?e=QyMSNr

[@hithin4u]

Thank you so much for the video this looks more informative, question: In real time do we have to create delta table or is it DBA responsibility and as my understanding we will not perform any DML operations on Data all we do is get data from multiple sources do some transformation and load into data warehouse house. Please correct me.

[@DataVerse_Academy]

It’s developer responsibility to create and load delta table. Admin will only create a workspace and lakehouse for you. Thank you!

[@adilmajeed8439]

thanks for sharing. Are you planning to do any intermediate and advance courses on PySpark?

[@DataVerse_Academy]

Thank you! Yes I will cover intermediate as well as advance topics in pyspark in upcoming months.

[@adilmajeed8439]

@@DataVerse_Academy Appreciate for the feedback. Looking forward it anxiously ...

[@parajf]

Suppose in delta table if I change type of column it will not update in delta table, so what should we do ?

[@DataVerse_Academy]

It’s the know issue. Microsoft is working on it. community.fabric.microsoft.com/t5/General-Discussion/Lakehouse-Delta-table-columns-not-updating/m-p/3612640 I can think of solution for this is, add a new column to the delta table then update that column with the existing one. Once updated then drop the older one.

[@Prashnanth]

Do videos on how to load the file from databricks to the storage account

[@DataVerse_Academy]

Definitely!

[@Prashnanth]

Need source code .

[@DataVerse_Academy]

I have added in the video description. Dataset and Code Link- 1drv.ms/f/s!Ak57yG1_Qg0iey_2DKTSnObu6Ts?e=QyMSNr