Microsoft Fabric End to End Ecommerce Project - Building Medallion Architecture in Lakehouse

Microsoft Fabric End to End Solution - Building Medallion Architecture in Lakehouse #microsoft #microsoftfabric #azure #lakehouse #dataengineering #dataengineering #modelling #facts #dimesions #technology #ai #chatgpt #india
Note: Download the whole folder
Dataset and Code Link- 1drv.ms/f/s!Ak...
---------- Project chapters ---------
0:01 Project Overview - Microsoft fabric lakehouse end to end solution
01:37 Creating Workspace and Lakehouse
02:38 Ecommerce Dataset Overview
03:31 Lakehouse - Upload Files from Local System
04:30 Loading Bronze Table - Bronze Sales
14:39 Dimensional Model Overview
15:38 Loading Order Return Dimension - Gold Layer
24:44 Loading Order Priority Dimension - Gold Layer
38:54 Loading Ship Mode Dimension - Gold Layer
48:32 Loading Customer Dimension - Gold Layer
54:34 Loading Product Dimension - Gold Layer
01:01:52 Loading Date Dimension - Gold Layer
01:11:23 Loading Fact Sale Dimension - Gold Layer
01:24:17 Mounting to Lakehouse and Archiving Files
01:30:01 Running Notebooks inside a Notebooks
01:34:01 Creating Data Pipeline to run Final Notebook
01:36:16 Creating Semantic Model and Creating Auto Generated Report
01:38:55 Creating Auto Generated Report from Semantic Model
01:39:50 Connecting to Semantic Model Using Power BI Desktop in Direct Lake and Direct Query Mode
Microsoft fabric data pipeline script activity: • 03 Microsoft Fabric Da...
Microsoft Fabric data Lakehouse end to end project - • 003 Microsoft Fabric L...
If you want to know more about Microsoft Fabric data Warehouse, Please watch below videos-
1. Create Microsoft Fabric Data warehouse: • 001 Microsoft Fabric D...
2. load data into Microsoft Fabric data warehouse using Data Pipeline: • 02 Microsoft Fabric Da...
3. Create table in Microsoft Fabric data warehouse: • 003 Microsoft Fabric D...
4. load data into Microsoft Fabric data warehouse using Copy into Command: • 003 Microsoft Fabric D...
5. Transform and load data into Microsoft Fabric data warehouse using T-SQL Procedure - • 004 Microsoft Fabric D...

[@user-su4gv5kq3t]

Very good explanation.

[@DataVerse_Academy]

Thank you 🙏

[@sonyvijai]

Great Video. Crisp and clear

[@priyanthakarunathilake8030]

Really helpful. Thanks.

[@narendrareddy3576]

great one bro, keep uploading real time scenarios.

[@DataVerse_Academy]

Sure, Thank you!

[@narendrareddy3576]

Thanks for the reply@rse_Academy , how do we get the input file name in fabric. I am trying with input fine name function but it's writing blank values.

[@yveshermann]

this guy is a champion!! Thanks so much :):)

[@RiseHigh1988]

Good explanation 🙏🙏.

[@user-dy8xu7uj8k]

this is great video. thanks

[@IEYdel]

Super helpful! Do you have a video that shows the silver layer with an example of joining related data from heterogenous data sources with data cleansing and deduplication? :D Still you are my hero Vishnu! Thank you for this video!

[@KaraokeVN2005]

Great video, Thank you so much!!!!!!!!!!

[@tv.TheDogFather]

Thanks for the video... Gold_Product is still not included in the Code zip file. Can you please include it? Not as Important, but at the same time, can you include the Run_Load notebook?

[@pragatisharma6036]

product script is missing in data code file please upload it

[@priyankaparida8423]

Hello Sir, After line no. 23 it is directly showing line no.77 .the middle part is skipped so not getting the code in between that. can you help with it.

[@user-lj9fk8dg9h]

Hello sir, Thank you so much providing these productive videos. Today, I faced a challenge, and the solution I couldn't find elsewhere. That is How to Extract data from SAP Hana Cloud to Microsoft Fabric (cloud to cloud connectivity). Could you please help me here?

[@rahuldhankhar5346]

Hey Vishnu, This was a great explanation for real time scenario. I have a scenario where i wanna recursively read multiple files and create a combined data frame before the bronze load. I tried different spark examples but not successful so far. Please let me know if you come across a solution on this. Thanks.

[@DataVerse_Academy]

Sure, I will have a look.

[@rahuldhankhar5346]

@@DataVerse_Academy Btw this problem applies to excel as input with multiple sheets. For other file types such as json, parquet, csv etc. we can utilize the solution below: df = spark.read.load('abfss path', format='csv', header=True, recursiveFileLookup=True ) display(df.limit(10))

[@user-dy8xu7uj8k]

Hi , I have some complex "Scalar user defined functions" defined in MYSQL and I have to migrate them to fabric, but as of now fabric doesn't support creation of "Scalar user defined functions" in warehouse. In this scenario please let me know alternative options I can use. Thanks

[@DataVerse_Academy]

you can build that logic inside the procedure. I know you will not able to return a value using a function, but you can build whatever the logic which you are trying to build. If you can give me context, then i will provide you the code as well

[@sanishthomas2858]

Nice. quick question, the Presentation slide shown for Architecture is Power Point or any other software?

[@DataVerse_Academy]

It’s power point.

[@AnisurRahman-wm2ys]

Excellent !!!!! Do you have this type of video for SCD2 ?

[@tv.TheDogFather]

I think for the DIM Merges just wrap the merge inside an Insert Into and change the Update of the Merge Accordingly.

[@user-lj9fk8dg9h]

Sir, How can we build the JDBC/Pyodbc connection between Fabric Data warehouse and Fabric Notebook. I have been finding it since a long time, but un-successful

[@DataVerse_Academy]

But why do you need it, what is the use case which you are trying to implement?

[@user-lj9fk8dg9h]

1.Initially, we are getting data from multiple sources and sinking them in one warehouse (Raw Data). 2. Now we want to extract data from this warehouse (Raw Data) to another warehouse (Transformed Data) through a Notebook wherein we will be performing our transformation logic. Hence, I want to build the connection between warehouse and Notebook only using JDBC or Pyodbc

[@omkarkulkarni3346]

Nice video ,my question is all these things you showcased here can be done Azure Synapse then why Fabric considered here,is there anything synapse cant do here ? Whats striking difference that business should consider to use Fabric as the front runner ahead synapse in future?

[@DataVerse_Academy]

In Microsoft fabric, everything is available at one place. You don’t need to create separate things. And one of most important thing is one lake, where everything is integrated. If you have multiple departments, for moving data from one department to another department you don’t need to create pipelines, by just providing access you can get the data.

[@omkarkulkarni3346]

Can you elaborate a little more in comparison with Azure synapse

[@vinaypratapsingh5815]

When i am clicking on New Semantic Model , I am not able to see all those tables to select a table or all tables . Because of that i am not able to create Semantic Model. Could you please help me here ? Thanks

[@DataVerse_Academy]

Whats the error you are getting ?

[@vinaypratapsingh5815]

@@DataVerse_Academy Thanks for your response. I am not getting any error, but I am not able to select any table to create my semantic model . Under Select all , it's not giving me tables name to select the table name

[@DataVerse_Academy]

Please try below once Settings- admin portal- > tenant settings - > information protection -> allow users to apply sensitivity labels for content - enable this, Then you will be able to create semantic model through lakehouse

[@John.Wick.221]

Where can I get more such data source

[@azobensadio260]

Thanks for the real time scenarios , but at no point did you present the gold_product and the file isn't in the shared folder. thanks for your feedback.

[@DataVerse_Academy]

For Gold_Prodcut- check the video “at 54:34 Loading Product Dimension - Gold Layer”.

[@DataVerse_Academy]

For files, please check the link in the description. Download the whole folder.

[@user-dy8xu7uj8k]

I have a SQL server stored procedure which updates, deletes and merges data into a table , how do I convert the stored procedure to pyspark job, is it possible to update a table in fabric using pyspark?, please make a video on this topic

[@DataVerse_Academy]

It’s very easy to do the same thing in pyspark, we can do all the stuff which you mentioned. I am a on break for couple of months. I am going to start creating video very soon.

[@user-dy8xu7uj8k]

@@DataVerse_Academy please do create a video when you are back from break. Thanks

[@meradbhima3951]

Hey , This is great Video but I am not able to open the code files you have given. I am missing the code you have used

[@CapitanFlintt]

Hello, thanks a lot for the tutorial ! But you just forgot to upload Gold_Product code in the zip file, can you upload it? Thanks

[@DataVerse_Academy]

Sure

[@longphamminh5804]

why did you create two folder "current" and "archive" in Files

[@DataVerse_Academy]

To archive the processed file from current to archive folder.

[@longphamminh5804]

Thank you for answer

[@longphamminh5804]

@@DataVerse_Academy I have one more question: When to use spark.read.table() and spark.sql

[@sularaperera2719]

Hi Vishnu, when creating a Semantic Model - Fabric gives an error saying "Unexpected error dispatching create semantic model to portal" do you have any ideas why? Thanks

[@DataVerse_Academy]

Do you have all the required access to create a semantic model in workspace?

[@sularaperera2719]

@@DataVerse_Academy Yes, I have Trial free account. Had two workspaces for pyspark training sessions and one your project. I had created a semantic model before when I did tutorials. but no luck on creating a semantic model with your project. 😒

[@DataVerse_Academy]

Solution for this is, open Sql analytics endpoint instead of lakehouse, then you will be able to create the model. Microsoft have just recently changed some settings.

[@DataVerse_Academy]

Another solution, Settings- admin portal- > tenant settings - > information protection -> allow users to apply sensitivity labels for content - enable this, Then you will be able to create semantic model through lakehouse

[@cargouvu]

Where can I get the files from to follow along.

[@DataVerse_Academy]

It’s there in the description.

[@DataVerse_Academy]

You can find the code in the same folder.

[@lakshminathreddyyaddula]

Is this completely free course

[@DataVerse_Academy]

Yes