Microsoft Fabric: How to load data in Lakehouse using Spark; Python using the notebook

[@sanumpzha]

Tried and it went well.! thank you Amit

[@AmitChandak]

Glad to hear that. Let me know, if you have any questions Thanks 🙏

[@LearnAtHomewithGulshan]

Kudos for you Amit!!

[@AmitChandak]

Thanks 🙏

[@christophehervouet3280]

Fantastic study Amit

[@AmitChandak]

Thanks 🙏 Hope you will like the series Mastering Microsoft Fabric 35+ Videos: kzbin.info/www/bejne/pl7ZYXxriJKsmNU

[@nabjad94]

very useful thank you

[@AmitChandak]

You are welcome. Thanks 🙏 Hope you will like the full series Mastering Microsoft Fabric 35+ Videos: kzbin.info/www/bejne/pl7ZYXxriJKsmNU

[@Kulfi2241]

Great session. Q:-To use python in this notebooks, what libraries are required Amit.

[@AmitChandak]

Depend on what you need. Basic Libraries are already available. You just need add import. You can check this out for more information learn.microsoft.com/en-us/fabric/data-science/python-guide/python-library-management

[@joelluis4938]

Hi Great video and I have 3 questions because I've started to learn about fabric this week and I would love to hear your comments to keep learning more about this -Can I explore my table using pandas and all his methods? I thought that I had to use Spark lenguage to do that - Why should I transform my table to a Delta Table if it's currently working fine from csv or it could be an SQL view from SQL serve ? I still don't understand the advantage of doing that.. -I saw many posts showing the transformation of flat files to delta files. But what if I have an sql view from SQL server .. We could also change that view to a delta table to improve or no make sense?

[@AmitChandak]

You should be able to use most of the Pandas functions and analyze the data. However, please keep in mind that the Spark engine is running when you execute the code. In Fabric, a Delta Parquet format file is treated as a table. That's why we prefer to save it back as Delta Parquet, enabling us to analyze it using SQL endpoints or Power BI. We need to bring in data from SQL Server to this environment. If you use Dataflow Gen2 or a data pipeline, it will be saved in Parquet format. When you save using Spark, remember to choose the appropriate format while saving

[@samirsahin5653]

I love your fabric videos. In this video we are pulling data from csv, transform it on notebook via pandas and creating table for sql so we can run sql queries on our data. Then visualize it on power bi. This is almost what I was looking for. Next step how can we make sure refresh data in background? If I create a scheduled refresh on PBI, I think it will only get data what is in table. It won’t check csv updateszz How can we create a refesh to make sure get most updated data from csv and python code refresh that gross column? Second question related do datalakehouse. I’m on fabric trial. Can I access from this trial fabric workspace to my current premium workspace and manipulate data on my current pbi reports?

[@AmitChandak]

You should be able to create a pipeline and call the notebook. Check How have done Incremental ( I am not using pipeline) Microsoft Fabric Part 15: Incremental Data Load(ETL) for Warehouse using Dataflow Gen 2, SQL Procedure, Data Pipeline: kzbin.info/www/bejne/o6Gpg6GQd72mrLM How about incremental without Dataflow, Only using pipeline and SQL procedure for Microsoft Fabric Warehouse Microsoft Fabric Part 16: kzbin.info/www/bejne/p6SyepmKr892ibM Try to see if you have Lakehouse connector. Or create a report on Power Bi desktop using new lakehouse connector and try to publish on Premium workspace

[@wb6ez4jd7s]

What if you want to execute that notebook from external editor like VS Code on your desktop? This method wouldn't work.

[@AmitChandak]

Please check, if these can help #microsoftfabric: Use Token Authentication to load local SQL server to #lakehouse using Python code- kzbin.info/www/bejne/l2Sbm5uqj6xprpo Microsoft Fabric: Use Token Authentication to load local SQL server to Warehouse using Python code- kzbin.info/www/bejne/hXjWfX2KrdOVoKc Microsoft Fabric: Load incremental data from local SQL server to Warehouse using on-premise Python- kzbin.info/www/bejne/nXOqmop9f7hoedU Microsoft Fabric: Load local SQL server data in Warehouse using Python, Pandas, and sqlalchemy- kzbin.info/www/bejne/hmHSXpRjbdd5fpI

[@hirenrami7348]

I think your videos are amazing! I have followed your instructions and managed to import data using an API and create a dataframe. However, I have encountered an issue where the column headers are not named as expected and have been automatically assigned. I am wondering if you have any suggestions on how to address this problem?

[@AmitChandak]

Hi @hirenrami7348, Can you check if the column name are in first row. It may happen sometime, In that case we have to use first row as header.

[@faheemiftikhar706]

Hi, really good video. I just had one question. I wonder what other ways are there to load data with similar approach, I mean in your case reading from your github repository. What other sources can I use to load the data. I mean like can i use jdbc or odbc

[@AmitChandak]

In the case of sources for which we have a PySpark connector, we should be able to load them.

[@hirenrami7348]

One more question is I want to make my PowerBi dashboard interactive like I want to take input from user like API and use that API key in my pyspark and load the data accordingly can you suggest how this can be done ? Thanks :)I have a question regarding my Power BI dashboard. I want to make it interactive by allowing users to input an API key, which will then be used in Pyspark to load the relevant data. Could you please advise on how I can achieve this? Thank you very much!

[@AmitChandak]

I have python code in a few video, about how to use key, check if that can help Loading Microsoft Fabric Lakehouse made easy- Python code using OAuth Token: kzbin.info/www/bejne/nJXHoWWDnat2eLs REST API- Use Token based Authentication to load local/on-premise SQL server data to Warehouse using Python code: kzbin.info/www/bejne/hXjWfX2KrdOVoKc if these can not help. Refer support.gpsgate.com/hc/en-us/articles/360021707379-How-to-get-an-API-token-and-authorization-in-REST-with-Python medium.com/python-rest-api-toolkit/python-rest-api-authentication-with-json-web-tokens-1e06e449f33 medium.com/geekculture/how-to-execute-a-rest-api-call-on-apache-spark-the-right-way-in-python-4367f2740e78

[@parajf]

Now same table if I want to use in spark sql is it possible, how

[@AmitChandak]

We can Manage the Lakehouse table to managed using Spark Python and Spark SQL Check Managing Microsoft Fabric Lakehouse using Spark SQL: Data Manipulation Insert, Update, Delete, Alter-kzbin.info/www/bejne/hqbRfJWwac52e8k

[@prachijain5955]

Hello Sir, I want to access SFTP Server data using Azure Notebook but for that I need to set Inbound Rule . As Notebook will have dynamic IP then how can we allow Notebook IP to access Port 22? Thank you in advance . Please advise

[@AmitChandak]

Please check and allow ips for app.powerbi.com learn.microsoft.com/en-us/power-bi/admin/power-bi-allow-list-urls

[@prachijain5955]

@@AmitChandak Thank you so much Sir

[@GenZSchool]

Amit Can weuse python in Microsoft fabric to read data from SFTP server?

[@AmitChandak]

If we want read from SFTP we can do that using python and load that to Fabric. In case you need encryption while transfer you have option for that

[@bulletkip]

is it possible to use python locally to send data frame or csv to lake house

[@AmitChandak]

Refer if these can help #microsoftfabric: Use Token Authentication to load local SQL server to #lakehouse using Python code: kzbin.info/www/bejne/l2Sbm5uqj6xprpo Microsoft Fabric: Use Token Authentication to load local SQL server to Warehouse using Python code: kzbin.info/www/bejne/hXjWfX2KrdOVoKc Microsoft Fabric: Load incremental data from local SQL server to Warehouse using on-premise Python: kzbin.info/www/bejne/nXOqmop9f7hoedU Microsoft Fabric: Load local SQL server data in Warehouse using Python, Pandas, and sqlalchemy: kzbin.info/www/bejne/hmHSXpRjbdd5fpI Refer full series Mastering Microsoft Fabric 45+ Videos in English: kzbin.info/www/bejne/pl7ZYXxriJKsmNU

[@trone_tip]

Can you please create video on how to data pull from Excel file to fabric

[@AmitChandak]

Sure.

[@faheemiftikhar706]

Hi is there any way to integrate notebook with Github ?

[@AmitChandak]

The current list does not include a notebook, but this may be covered soon. I will check and confirm learn.microsoft.com/en-us/fabric/cicd/git-integration/intro-to-git-integration

[@faheemiftikhar706]

@@AmitChandak Okay thanks