Microsoft Fabric: Load data in Warehouse- Dataflow Gen2| Note- Warehouse Destination available now

[@AmitChandak]

Watch the video for the new Warehouse Destination Microsoft Fabric Part 18: Dataflow Gen2: Native warehouse Destination- Power Query Transformation - Pivot, Unpivot, Transpose Data kzbin.info/www/bejne/i2fJpHuJnrOjsJY

[@mohammadaamirkhan7830]

Thank you very much Sir Amazing video with a detailed explanation. I have covered each piece of information from this video. Looking forward to learning more from you.

[@AmitChandak]

You are most welcome

[@ruru1419]

Wonderful overview! Thanks so much!

[@AmitChandak]

I'm glad you found the overview helpful!

[@dilipinamdar5523]

Thanks 🙏

[@AmitChandak]

Thanks 🙏. Hope you will like other videos of series too.

[@sanishthomas2858]

Nice. quick question, why we are using Warehouse and also in another video we are using Lakehouse. in which use cases we have to decide on this? Also is this warehouse dedicated SQL pool like we jhave on synapse or is it just a lakehouse platform where warehouse is created?

[@AmitChandak]

Microsoft's Fabric architecture offers several data management options that cater to different needs and scenarios. Here's a breakdown to help you decide between using a Data Warehouse and a Lakehouse: Lakehouse Architecture: Scenario: When you need a unified platform for both big data analytics and machine learning. Data Stages: Typically divided into Bronze (raw data), Silver (cleaned data), and Gold (business-ready data). Tools: You can use Spark SQL for SQL queries or Spark Python for ETL processes. Data Warehouse: Scenario: Best for scenarios where you require traditional SQL databases for transactional processing or when specific SQL performance optimizations are needed. Tools: SQL (like in DB SQL) is used. Data Flow: For instance, you might move from Lakehouse (Bronze) to a hybrid Lakehouse/Warehouse (Silver) and purely Warehouse (Gold) for optimized querying and performance. Choosing Between Lakehouse and Warehouse: Full Lakehouse: Ideal for complete big data and machine learning pipelines, utilizing tools like PySpark for transformation. Warehouse: Preferable if your focus is on leveraging SQL capabilities for complex queries and transactional data processing. Tools and Integration: Dataflow Gen2: This can be used for data transformation in both Lakehouse and Warehouse setups. Notebooks/Spark: Currently, these are more aligned with Lakehouse setups and do not directly interact with traditional Warehouses. In summary, the choice between Lakehouse and Warehouse depends on your specific data processing needs, the scale of data, and the analytical tools you prefer. Lakehouses are versatile for handling large-scale data and machine learning workflows, whereas Warehouses are more suited for intensive SQL querying and transaction-based operations.

[@user-zo8mn4ik4e]

Hi, Thank you for your efforts. Can you please create some more videos on loading in to Lake house and warehouse other than Dataflow Gen2.

[@AmitChandak]

Please refer to full series. Most of the options are explored Mastering Microsoft Fabric 31 videos: kzbin.info/www/bejne/pl7ZYXxriJKsmNU

[@sanilmaru5491]

Hello Amit... When you created a table using SELECT INTO or created a view in Warehouse by copying query from Lakehouse.. I believe it is not actually pointing to Lakehouse. You have the tables with same name present in Warehouse and it actually sourced data from there... Not sure but you can try appending Lakehouse before schema name and see if it works... Though it doesn't work on Azure Synapse

[@AmitChandak]

Yes, seem like tables are there at both places. But with the same workspace, you can query tables across LakeHouse and the warehouse, for example kzbin.info/www/bejne/gXrFpWx9r6aXjc0

[@nissimelaluf1272]

So what is the difference between a data warehouse and a data lake? You basically loaded the same data for the 2 places

[@AmitChandak]

Warehouses are like databases, where data is stored in a structured format and can be accessed and modified using SQL queries. Lakehouses are like data lakes, where data can be stored in a variety of formats, including structured, semi-structured, and unstructured. Structured data in a Lakehouse can be accessed using SQL queries(Read Only) . I am loading the same data to show case various features. Refer first and third video of series kzbin.info/www/bejne/pl7ZYXxriJKsmNU

[@claudiabolog5441]

It looks like the Azure Synapse Analytics (SQL DW) option as a data destination for a Dataflow Gen2 has been replaced with Warehouse, making the process more straight forward as we only need to choose the Warehouse name now.

[@AmitChandak]

Yes, Agree. New one is much better. Discussed in the recent video - kzbin.info/www/bejne/i2fJpHuJnrOjsJY Hope you will like the full series Microsoft Fabric: A Brief Introduction- kzbin.info/www/bejne/pl7ZYXxriJKsmNU How to enable Microsoft Fabric; on Power BI Service| Start the Fabric (Preview) trial kzbin.info/www/bejne/bpysfnujoLZso5Y Microsoft Fabric-What it is, Why Use, Who Should, When and How to use kzbin.info/www/bejne/qZO6f32Dltijm6M Microsoft Fabric: Various Components| Microsoft Fabric Create Workspace, Lakehouse, Warehouse kzbin.info/www/bejne/pofZp5Vjgt-rgdk Microsoft Fabric part 5: How to load data in Lakehouse using Dataflow Gen2 kzbin.info/www/bejne/l4GsZ4GKaNGZqac Microsoft Fabric Part 6: How to load data in Warehouse using Dataflow Gen2 kzbin.info/www/bejne/g5-UeHVup7iYpbM Microsoft Fabric Part 7: How to load data in Warehouse using Data Pipeline | End-to-End Scenario kzbin.info/www/bejne/n5Wlo5ioabOlpLs Let's have a spark in fabric- Microsoft Fabric Part 8: How to load data in Lakehouse using Spark; Python/Notebook kzbin.info/www/bejne/oIind5amgMSeb6s Microsoft Fabric Part 9: Integrated Machine Learning- k-Means clustering: kzbin.info/www/bejne/oqTRp4RqitSHrdE Microsoft Fabric Part 10: Create a Date table in Dataflow Gen2, and use in Lakehouse, and Warehouse| Time Intelligence with Power BI kzbin.info/www/bejne/nWSldomOhsmeY8k Microsoft Fabric Part 11: How to load data in Lakehouse using Data Pipeline | End to End Scenario: kzbin.info/www/bejne/oJ_CdWVjl5qMprs Microsoft Fabric Part 12: What are the options to load local files in Lakehouse: kzbin.info/www/bejne/hpqYeqmhfd19eNU There is no shortcut to success. However, in MS Fabric, you can shortcut to Azure DataLake Microsoft Fabric Part 13: Shortcut- within Fabric/ Azure Data Lake: kzbin.info/www/bejne/gXrFpWx9r6aXjc0 Microsoft Fabric Part 14: Import Azure SQL Data to Warehouse | Multiple tables using Pipeline- kzbin.info/www/bejne/aqjNZZVpett5pqs Microsoft Fabric Part 15: Incremental Data Load(ETL) for Warehouse using Dataflow Gen 2, SQL Procedure, Data Pipeline: kzbin.info/www/bejne/o6Gpg6GQd72mrLM How about incremental without Dataflow, Only using pipeline and SQL procedure for Microsoft Fabric Warehouse Microsoft Fabric Part 16: kzbin.info/www/bejne/p6SyepmKr892ibM Microsoft Fabric Part 17: Load local SQL server data in Lakehouse using on-premise Gateway and Dataflow Gen2: kzbin.info/www/bejne/pXapXp1-i9KpmtE Microsoft Fabric Part 18: Dataflow Gen2: Native warehouse Destination- Power Query Transformation - Pivot, Unpivot, Transpose Data kzbin.info/www/bejne/i2fJpHuJnrOjsJY Microsoft Fabric Video 19: Load Azure SQL Data to Lakehouse & Warehouse simultaneously with Dataflow Gen2: kzbin.info/www/bejne/amGufGCaptqpgqc Blogs Microsoft Fabric: New Age Analytics medium.com/microsoft-power-bi/microsoft-fabric-new-age-analytics-ba90b894633f Microsoft Fabric- What, Why, Who, When, How medium.com/microsoft-power-bi/microsoft-fabric-what-why-who-when-how-638b391df4f1

[@venkat9743]

Could you please upload higher quality video, not able to see details clearly as it is a 360p video

[@AmitChandak]

Hi Venkat, HD Processing was ongoing and finished just a few mins back. Please check now. Appreciate your support for the channel. 🙏

[@venkat9743]

@@AmitChandak thank you

[@ruru1419]

Hey Amit, I am trying to use Dataflow Gen2 to update a Warehouse (not LakeHouse). I'm following exactly what you did, so using the SQL Connection string with target Azure Synapse (SQL DW). However I get this weird error: "An exception occurred: Microsoft SQL: This function doesn't support the query option 'EnableCrossDatabaseFolding' with value 'true'. " I cannot find anyone who had such an error...I tried different options, no luck. Did you get that by any chance?

[@AmitChandak]

I tried again and able to save the Dataflow Gen2. Hope you are getting error before that. Make sure you have taken correct URL and opted for Organization Authentication. Also make sure a warehouse is created; add a table using create statement and check it is working . If everything is fine, connect to me on LinkedIn and share screenshot. I would check the issue www.linkedin.com/in/amitchandak78/

[@user-gw5bl5hw4y]

Is Power BI Copilot will be killed a lot of jobs?

[@AmitChandak]

Power BI developers/designers will have to master Copilot to increase their productivity. That will save Job.