Dave, thank you for your time in this Lab, it is really helpful and you explain very well!
@Taletherapper Жыл бұрын
Thank you for this!
@imageofparadox3 жыл бұрын
Hi Dave, thanks for the video, very helpful, I'm wondering if it's possible to pull the header from a table from power apps to Data Lake Storage?
@DaveDoesDemos3 жыл бұрын
Hi, yes this would be possible in various ways. You should be able to write directly from Power Apps, although it's been a while since I looked at Power Apps. If you're storing in a CDM model then in theory you can just access that as part of the lake too. If your app writes into a database then use ADF to query that database. I'd be interested what you're trying to achieve here as that will drive the design. You could also do a push from the Power App using a Logic App with REST API front end and have the Logic App write to the storage as data comes in. A Function App would also achieve this.
@imageofparadox3 жыл бұрын
@@DaveDoesDemos many thanks for replying. Yes I watched one video where Logic app is used to get the data with header but it's a pull, What're are trying to achieve is, we have our data on CRM (cloud) and using power apps via export to data lake to export tables data to storage. we can see a model file in Json format is also created when we export the data, it has a property called header which is set to false. we're able to get the data without any problem on storage but we would like to have a header as well. Not sure how we can get the header this technique is a push one so we need to schedule any thing data gets syncronised automatically. So in short while exporting to data lake via power apps can we get the header too or we need to use log app to get the header? Thanks
@DaveDoesDemos3 жыл бұрын
@@imageofparadox are you sure that JSON isn't just the CDM format? If it's Dynamics that's what I'd expect to get out of the system these dats. If that's the case it will include the metadata, and tools like Data Factory can just read that data when processing. If you want to move it to a denormalised CSV that would be the correct way to achieve it, but CDM is still a normalised format including linking between tables. In this instance your CDM data set would be considered your raw dataset, and then you'd have a pipeline in ADF to transform it to flat CSVs if that's what you wanted to work with. You could also transform it to Parquet or Delta, or ingest into Synapse depending on what you want to do with the data. CRM data will usually be flattened out and ingested to a data warehouse for consumption in Power BI alongside other data.
@imageofparadox3 жыл бұрын
Thanks Dave, we have only one problem we need column header in csv files too. Our ultimate goal is to pull csv data (csv files data is prepared via pushed via export data lake from power apps) from storage to an oracle database. I think I was not clear what we're after. It's just the missing column names I.e.header in the csv files. We don't want to use json model to get just column names...
@DaveDoesDemos3 жыл бұрын
@@imageofparadox I think I'd need to take a look at the Power App to properly answer that one. It definitely sounds like you're exporting from CRM in CDM (Common Data Model) format which is not actually a CSV export even if it looks like that. If that's the case then you'd either need to change the export to straight CSV with a header, if it's supported, or transform the data after export using something like Data Factory. Data factory does support Oracle databases so you could write directly to the database. As an alternative, you could open the JSON and process it for the columns, but I wouldn't recommend this as it would be very complex for what you're trying to achieve.