Microsoft Fabric: Creating a OneLake Shortcut to ADLS Gen2

[@endjin]

Thank you for watching, if you enjoyed this episode, please hit like 👍subscribe, and turn notifications on 🔔it helps us more than you know. 🙏

[@MatthewRoche]

Another great installment - thank you!

[@akthar3]

wow! pleased to see a compliment from @MatthewRoche !

[@akthar3]

worked beautifully - thank you !

[@moeeljawad5361]

Hello Nice video. could you please explain from where exactly did you get the dfs url that you had pasted in the ADLS shortcut connector? in my url i am having a blob part in it and that is preventing me from doing the connection. Thanks

[@endjin]

Hi there, You should be able to just change "blob" to "dfs" if your storage account is ADLS Gen2 enabled. If it's not ADLS Gen2 enabled, then sadly you can't create a shortcut to a Blob Storage account at the moment. Ed

[@moeeljawad5361]

@@endjin Thanks for your reply. I found out that my storage account is not ADLS Gen2 enabled.

[@malleshmjolad]

Hi Ed, We have loaded few tables using synapse link into adls gen2 and created shortcut to access the adlsgen2 files in fabric,but while loading the files into tables,we are not getting the column names for the tables and it is showing as c0,c1....etc which is causing an issue,can you please give some insights on how to overcome this and load the tables with metadata also

[@endjin]

Hi - thanks for the comment! Which Synapse Link are you using? Dataverse? If so, this uses the CDM model.json format which doesn't include header rows in the underlying CSV files. You would have to read the shortcut data, apply the schema manually, and then write the data out to another table (inside a Fabric notebook or something) if you wanted to use that existing data. However, if you're using Synapse Link for Dataverse, you should instead consider using the new "Link to Microsoft Fabric" feature available in Dataverse: learn.microsoft.com/en-us/power-apps/maker/data-platform/azure-synapse-link-view-in-fabric. This will include the correct schema.

[@levidevos15]

Hi Ed, as a proposal for part 6, could you show us how you would go about copying files from bronze to silver and explain us the reasoning behind switching between files and (managed) tables?

[@endjin]

Good suggestion - stay tuned!

[@endjin]

Part 6 kzbin.info/www/bejne/pnS3nWCkpqmHlZo and Part 7 are now available kzbin.info/www/bejne/qZDQeZSCd9F0b5Y

[@MucahitKatirci]

Thanks

[@endjin]

Glad you're enjoying the videos

[@endjin]

Part 8 - Good Notebook Development Practices - is now available: kzbin.info/www/bejne/i6q2Z42LnM1gjcU

[@ChrisDowns88]

Once you have the shortcut connected, would you be converting it into a table format? I've seen that you can do this by right clicking and selecting 'Load to Tables'. However, how does this work when the .csv you're referencing has new potential columns/rows added to it? Would it pick this up on your lakehouse table? Thanks :)

[@endjin]

Generally everything in Fabric ends up in the table format, so, yes - it's likely any shortcut-ted files would ultimately end up in a Delta Table. But oftentimes you'd want to do a bit of processing of the core files before inserting it into a table. So I think the most likely scenario would be to have a notebook that reads the files, does the necessary processing and then inserts into a table. Regarding whether the files and table stays in sync when you use the "Load to tables" button is a good question, and one I don't have an immediate answer to! My hunch is that this is a one-off operation which would take a snapshot of the data at the time you created the table. Any changes to that file wouldn't show unless you re-loaded to that table. I suggest you give it a go (if you haven't already!)

[@prince26902690]

So, we defently have a ADF pipeline which runs on schedule bases, process the files with shortcuts and lods into lakehouse? then only we can query it, correct?

[@endjin]

@@prince26902690 I'm not sure I understand what you mean by "only we can query it" - could you elaborate, please?

[@lifewithoutborders1348]

from where you got other data

[@endjin]

Hi there. The Postcode data comes from: geoportal.statistics.gov.uk/datasets/a8a2d8d31db84ceea45b261bb7756771/about Ed

[@johnsonbolaji3722]

Hi Ed, how did you ingest the ONS postcode directory data into the ADLS in the first place. Did you upload it?

[@endjin]

Have you seen the previous episode? kzbin.info/www/bejne/o5qnkmd8j75qfbc

[@endjin]

Part 8 - Good Notebook Development Practices - is now available: kzbin.info/www/bejne/i6q2Z42LnM1gjcU

[@endjin]

I *think* it was via a copy activity within a Synapse Pipeline. But my colleague ingested that data :)

[@ullajutila1659]

The video was ok, but it did not explain how the ADLS storage account networking needs to be configured. More specifically, how to configure it in a secure manner, without allowing access from all networks.

[@endjin]

Hi there, Thanks for the feedback! This video was recorded at a time when it wasn't possible to connect to ADLS Gen2 if it wasn't publicly accessible. However, now that's changed: if you'd like to understand how to access a partially restricted ADLS account, please see the documents here: learn.microsoft.com/en-us/fabric/security/security-trusted-workspace-access. If you've fully disabled public access, there's no easy way to create a Shortcut that I'm aware of, sadly. Hopefully that functionality will come. Ed