Microsoft Fabric: How to load data in Warehouse using Data Pipeline | End to End Scenario

Microsoft Fabric, How to load and transform data into a warehouse. Use data pipeline to load the data and SQL to transform data. Create a star schema and analyze the data using Power BI. How to new Microsoft Fabric warehouse connector. How to SQL Server management studio (SSMS)
Microsoft has Microsoft Fabric, the next-generation Analytic Platform. Check out how to enable it on Power BI Service. How to start a Fabric (Preview) trial
The Microsoft Fabric Platform provides customers with a SaaS-ified, open, lake-centric(One Lake), full-featured data, analytics, and AI platform that meets all their data estate needs. Power BI, Synapse Data Warehouse, Data Factory, Spark, and Notebook all under one platform
00:00 Create Warehouse
02:30 GIT file to be loaded in Warehouse
04:30 Load Data in Lakehouse using Data Pipeline
12:30 Check Data using SSMS
15:00 Correct Data Pipeline, Re-Run
21:30 Create Star Schema
34:00 Create Model
35:30 Analyze data in Power BI Service
38:00 Analyze data in Power BI Desktop
GIT File link - github.com/amitchandakpbi/pow...
All files: github.com/amitchandakpbi/
Power BI - Azure Free Tier, Create Azure Data Lake Gen 2 and connect that in Power BI: • Power BI: Connect with...
Get Azure SQL DB and Configure it on Power BI: • Abstract Thesis 93: Po...
Official Document: learn.microsoft.com/en-us/fab...
Video from GuyinaCube: • Microsoft Fabric Launc...
Buy a Microsoft Fabric subscription: learn.microsoft.com/en-us/fab...
SKU: learn.microsoft.com/en-us/fab...
Licenses: learn.microsoft.com/en-us/fab...
Enable Fabric: learn.microsoft.com/en-us/fab...
🔵 What is Microsoft Fabric
Microsoft Fabric is an all-in-one analytics solution for enterprises that covers everything from data movement to data science, Real-Time Analytics, and business intelligence. It offers a comprehensive suite of services, including data lake, data engineering, and data integration, all in one place.
With Fabric, you don't need to piece together different services from multiple vendors. Instead, you can enjoy a highly integrated, end-to-end, and easy-to-use product that is designed to simplify your analytics needs.
How to enable Fabric, find details - learn.microsoft.com/en-us/fab...
Microsoft Fabric licenses details -learn.microsoft.com/en-us/fab...
Don't forget to like, comment, and subscribe for more great content!
▶️Please Subscribe: / @amitchandak
▶️Data:
• Covid 19: covid19.who.int/WHO-COVID-19-...
• Get The data and Pbix files on Github: github.com/amitchandakpbi/pow...
• DropBox files: www.dropbox.com/sh/90ycq7hpmn...
---------
▶️Follow me on:
• Medium: / amitchandak
• Twitter: / amitchandak78
• LinkedIn: / amitchandak78
---------
▶️My Other Videos:
• Mastering Power BI: • Power BI for Beginners...
• Expertise Power BI: • Expertise Power BI- Ho...
~-~~-~~~-~~-~
Please watch: "Microsoft Power BI Tutorial For Beginners✨ | Power BI Full Course 2023 | Learn Power BI"
• Microsoft Power BI Tut...
~-~~-~~~-~~-~

[@tauseefroyal6336]

Awesome work bro, please keep posting all the topic from Fabric... appreciate your efforts

[@AmitChandak]

Thanks 🙏. Hope you will like the series Mastering Microsoft Fabric 30 Videos: kzbin.info/www/bejne/pl7ZYXxriJKsmNU

[@sonalimishra3356]

Really appreciate for your efforts, need more videos on ms fabric

[@AmitChandak]

@sonalimishra3356 Thanks 🙏. Hope you will like the full series, added 37 Videos so far. Mastering Microsoft Fabric 35+ Videos: kzbin.info/www/bejne/pl7ZYXxriJKsmNU

[@123jamalq]

Sir, It is really a challenge task and u did effectively.

[@AmitChandak]

Thanks.

[@prashantsuthar7]

Work for me thanks 👍

[@AmitChandak]

You're welcome! 🙏 Hope You will like the series - kzbin.info/www/bejne/pl7ZYXxriJKsmNU Please find 370+ videos, blogs, and files in 70+ hours of content in the form of an organized course. Get Fabricated with Microsoft Fabric, Power BI, SQL, Power Query, DAX Please Like and Share biworld.graphy.com/courses/Get-Fabricated-Learn-Microsoft-Fabric-Power-BI-SQL-Power-Query-DAX-Dataflow-Gen2-Data-Pipeline-from-Amit-Chandak-649506b9e4b06f333017b4f5

[@nitesh791102]

Hi Amit, I tried connecting Dataverse table to Lakehouse using data pipeline. But the pipeline does not respond. Can you please suggest/guide on how to do this.

[@AmitChandak]

I will check and get back

[@GauravShukla-ut2xs]

Hi, I am not able to understand the step where you talked about connecting sql server to Microsoft fabric. Could you please elaborate that step to me. When i launched ssms, which option should I choose to login and where will I get the option to enter my email and password. Thanks a lot.

[@AmitChandak]

check if this screenshot can help www.dropbox.com/s/gc10aca0ohecwvh/Login%20to%20SQL%20Server.PNG?dl=0

[@AmitChandak]

SQL Server Authentication will give both user and password

[@arfanislamabir]

I am getting data type issue while mapping. Can you please check the recent update ?

[@AmitChandak]

@@arfanislamabir , please check all columns have fixed data types, no mixed one. Also you can correct it mapping, location has moved to right

[@saumyakapoor6772]

Thank you for your videos. They are very informative. I am not able create a new data pipeline in the warehouse. When I click on it, I do not get any error but new pipeline window does not open. Kindly suggest

[@AmitChandak]

Clear browser cache, refresh and try again. I am able to open a new pipeline. If it does not open, let me know.

[@saumyakapoor6772]

@@AmitChandak no luck

[@saumyakapoor6772]

I can directly create a pipeline from data factory, but not from the get data option in warehouse

[@uplifthabesha754]

What is the difference between loading data using pipeline and gen 2?

[@AmitChandak]

Dataflow Gen 2, a component of Microsoft Fabric(Came from Power BI world -> Powe Query), offers intuitive data transformation capabilities. Data Engineers can use Dataflow to visually design and execute data transformations without the need for complex coding, reducing development time and increasing productivity. And can finally load data to Lakehouse and Warehouse Data Pipeline can be used to load data. It can also be used to orchestrate and monitor end-to-end. I can call dataflow, spark job in the data pipeline refer learn.microsoft.com/en-us/fabric/data-factory/data-factory-overview#data-pipelines

[@sanumpzha]

Hi Amit, i followed the steps and i couldnt load the table as the _id column BIGINT data type error. I tried in SSMS as well but still..

[@AmitChandak]

In the pipeline on the mapping step(which is now part of the popup) , make sure you have changed the data type. Also, make sure ID is a whole number. Please share the error you are getting. Thanks 🙏

[@sanumpzha]

@@AmitChandak Here is the error. ErrorCode=UserErrorSqlDWCopyCommandError,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=SQL DW Copy Command operation failed with error 'Column 'customer_id' of type 'BIGINT' is not compatible with external data type 'Parquet physical type: BYTE_ARRAY, logical type: UTF8', please try with 'VARCHAR(8000)'

[@bhumikaattarde1827]

i tried loading data from lakehouse to warehouse via pipeline, but at the connect to data destination step, the data type column is not having drop down option to change data type. so not able to change the data type of column at this step. can you help me?

[@AmitChandak]

I such case create a table before hand with required data type and try to load into that. Thanks. Hope you will like the full series kzbin.info/www/bejne/pl7ZYXxriJKsmNU

[@prinkilduhoon]

What is think about PL 300 Microsoft power BI certificate It is worth it

[@AmitChandak]

Yes, Power bi will still hold good. Hope to see some new certificates for fabric in near future.

[@balajidevaraj7167]

Hi I tried to load the data from Azure SQL to this Data warehouse, but in the choose destination, its not showing my data warehouse name. could you please help me on this

[@AmitChandak]

You already created warehouse? It should warehouse in destination option. I will also check Azure SQL and come back.

[@balajidevaraj7167]

@@AmitChandak yes. I created warehouse and from there only using pipeline I'm trying to load the data into warehouse

[@AmitChandak]

I was able to do so. I will create a video and share.

[@balajidevaraj7167]

@@AmitChandak I tried to load the data from LH but there also DW destination is not visible. Not sure if i am missing anything when loading data into DW

[@AmitChandak]

Please find the Video- kzbin.info/www/bejne/aqjNZZVpett5pqs

[@thghtfl]

I still can not believe this warehouse doesn't support Postgres in the cloud as a source

[@AmitChandak]

I have checked again that the pipeline is compatible with both PostgreSQL and Azure Database for PostgreSQL,. PostgreSQL should support for cloud environments. Similarly, Dataflow Gen2 is also supporting these databases.