Dynamics 365 FO: entity store export to Azure data lake

[@faroukghandour6638]

Amazing walk through! I was struggling with how unclear the documentation was out there and found this video a saving grace! Thank you sir! :)

[@basvangameren3956]

@Markus Nobauer: Great video, well explained! Why is it so imortant to have the hierarchical namespace setting of the storage account to "disabled"? Is this a current limitation of the export process?

[@ERPCoder]

It didn't work otherwise. I made an other video on powerbi dataflows which also uses a stroage account and unfortunately they are not compatbile. You need two different types of storage accounts.

[@basvangameren3956]

@@ERPCoder , Thank you for your response, my experience is the same (an error is displayed in the .csv file available in the storage account) when enabling the hierarchical namespace. However, in the document here (docs.microsoft.com/en-us/dynamics365/fin-ops-core/dev-itpro/data-entities/entity-store-data-lake#entity-store-data-in-azure-data-lake-full-push-and-trickle-feed), it is explicitly mentioned to be enabled: "In the Advanced options dialog box, you will see the Data Lake storage Gen2 option. Select Enable under the Hierarchical namespaces feature. If you disable this option, you can't consume data written by Finance and Operations apps with services such as Power BI data flows." I've tested on F&O versions 10.0.8 and 10.0.10 both show similar behavior - error in csv file when enabling hierarchical namespace in storage account.

[@TheJavieregh]

Markus which version D365 are you use? Because I have only Enable Data Lake Integration and Trickle update data lake . I don't have Enable Common Data Store integration

[@ERPCoder]

The last version 10 VHD from LCS. Did you enable all the required features via flighting?

[@viniciuschiovitti6269]

thanks markus!

[@AlejandroGonzalez-vk1zz]

You help me with this video, thank. I have a question, Can I to do a conexión with Power BI with the Storage Account?

[@ERPCoder]

Yes you can. D365 now supports exporting tables directly to a Gen2 Data Lake Storage account. PowerBI supports access to ADLS G2 in plain CSV mode or in Common Data Model (CDM) mode.

[@djedjigabouzidi2769]

Thank you so much for this video, I want to copy data from dynamics365 to Data Lake Gen2 using ADF Pipeline, but there is the REST API limitation, I have a very large files with 60 million rows and the limitation of rest API is 20.000. Do you have any idea. Thank you.

[@DanielWillen]

How would you export regular entities? Such as the one you can select to export from Data Management in D365FO?

[@ERPCoder]

maybe BYOD would be an option for these entities. We have done this for different integration scenarios via flow. Data lake is IMHO ideal for big data

[@baski2020]

Can we use different cloud tenant? Eg. D365 in one tenant and Data lake in another tenant

[@ERPCoder]

You should be fine if you create the storage account in another tenant but store the connection string to this storage in a key vault that is in the same tenant with your D365. So you can give D365 access to the key vault in azure portal where the connection string is stored.

[@baski2020]

@@ERPCoder Sorry. I didn't get your point. If possible can you please share the video or document?

[@LegendSandeepYT]

can we add simple entity to Data Lake? rather than Aggregate Entities?

[@ERPCoder]

IMHO the integrated data lake export uses the aggregated tables. But you can export the entities e.g. to a SQL Azure and create a flow that puts them in the data lake

[@damn68]

How do you get this vhd sample for testing

[@ERPCoder]

LCS asset library

[@damn68]

@@ERPCoder And how do you get access to LSC as a freelancer. Is there a license I can purchase from.

[@ERPCoder]

@@damn68 you need an azure AD account to signup

[@damn68]

​@ERPCoder hmm, I have AD account, what am I missing. Do you want me send you a message on your blog.

[@ERPCoder]

@@damn68 go to lcs.dynamics.com/Logon/Index and sign up. You may have not an actual project but you should have access to the asset library where the VHDs are

[@samprakash8771]

Exported CSV does not have header what is the cause?

[@ERPCoder]

because the definition is in the model.json file

[@samprakash8771]

@@ERPCoder thanks, if I connect data lake to power BI, I need the data with header. Then what should I do?

[@samprakash8771]

@@ERPCoder definition which means data with header?

[@ERPCoder]

@@samprakash8771 the full integration with CDS is comming in wave1 2020 docs.microsoft.com/en-us/dynamics365-release-plan/2020wave1/finance-operations-crossapp-capabilities/data-common-data-service-phase-1-2