Azure Data Factory | Loop through multiple files in ADLS Container | Lookup & ForEach Activities

Azure Data Factory | Loop through multiple files in ADLS Container & load into one target azure sql table Lookup & ForEach Activities
Loop through Multiple inputs and load into Multiple Destination files.
studio.youtube...

[@sravankumar1767]

nice explanation bro. do more videos on ADF with latest scenario's.

[@sureshpallapolua8]

Really appreciate bro

[@OnlineForward]

Can you add another hierarchy on that, add multiple folders.

[@innowarue]

You should use get metadata which actually gets childItems with a "name" attribute which points to the filename. Using ...output.value is a bad way to loop as its reading rows of the actual data in the files and results in many of the infite loops some viewers are experiencing

[@KeshavLearnTSelf]

True I did another video with Get metadata as well.. this is to show all the options..thanks for the comment I appreciate the suggestion..

[@vru5696]

@@KeshavLearnTSelf how can extract table names from file and then iterate those table names using for each? Please share link if you have already.

[@priyankapatel9461]

Thanks for posting great content, i learnt from it. If I have multiple excel file in blob and copy to SQL tables how can I achieve?

[@SiriVlogs-w8d]

Hi, Just a quick question when I load the data from multiple files which got 2 sheets each, it is asking me to select a sheet name. Could you please make a video on how to load the data from multiple files with 2 sheets each. I really appreciate.

[@ajaxaj4885]

Difference between look up and meta data activity?

[@alangutierrez1345]

Nice video, I tried step by step but I don't know why the loop keep process through the files until the eternity jaja any idea?

[@dharmveerchaudhary7060]

Pls give any idea ,how to kill infinite loop created inside the until activity in azure data factory

[@moesmael]

Thank you for the good video you have done, I tried step by step as you did many times but it gives me an error. file Output generated from lookup as Jason file is not showing filename in the folder, it shows data inside the file like this : I think there is a missing step you did not mention, can you please let me know how to overcome and loop over files in the folder, what shall I do to get filenames instead? { "count": 102, "value": [ { "Prop_0": "SR_RETURNED_DATE_SK", "Prop_1": "SR_RETURN_TIME_SK", "Prop_2": "SR_ITEM_SK", "Prop_3": "SR_CUSTOMER_SK", "Prop_4": "SR_CDEMO_SK", "Prop_5": "SR_HDEMO_SK", "Prop_6": "SR_ADDR_SK", "Prop_7": "SR_STORE_SK", "Prop_8": "SR_REASON_SK", "Prop_9": "SR_TICKET_NUMBER", "Prop_10": "SR_RETURN_QUANTITY", "Prop_11": "SR_RETURN_AMT", "Prop_12": "SR_RETURN_TAX", "Prop_13": "SR_RETURN_AMT_INC_TAX", "Prop_14": "SR_FEE", "Prop_15": "SR_RETURN_SHIP_COST", "Prop_16": "SR_REFUNDED_CASH", "Prop_17": "SR_REVERSED_CHARGE", "Prop_18": "SR_STORE_CREDIT", "Prop_19": "SR_NET_LOSS" }, { "Prop_0": "2452497", "Prop_1": "52390", "Prop_2": "2466", "Prop_3": "22231", "Prop_4": "887843", "Prop_5": "4018", "Prop_6": "1346", "Prop_7": "4", "Prop_8": "11", "Prop_9": "236085", "Prop_10": "51", "Prop_11": "78.03", "Prop_12": "7.02", "Prop_13": "85.05", "Prop_14": "49.83", "Prop_15": "939.93", "Prop_16": "31.99", "Prop_17": "39.59", "Prop_18": "6.45", "Prop_19": "996.78" }, { ... etc and I got this error ine one of Foreach Activity where it executed manytime : ErrorCode = SqlColumnNameMismatchByCaseSensitive, 'Type = Microsoft.DataTransfer.Common.Shared.HybridDeliveryException, Message = The column "SR_RETURNED_DATE_SK" in the data set "[TPC_DS]. [Store_returns]" cannot be found in the physical SQL database Case sensitive. Column "sr_returned_date_sk" seems similar. Check dataset configuration to continue., Source = Microsoft.DataTransfer.ClientLibrary, '' Type = System.InvalidOperationException, Message = The specified ColumnMapping does not match any Column match in source or target., Source = System.Data, ' I make sure that the source and target are matched

[@KeshavLearnTSelf]

Hi Mohammed, Seems there is issue at mapping side for you.. if you look at error it says invalid column.. re-check the mapping once.

[@moesmael]

@@KeshavLearnTSelf Thank you. I have checked. The problem is that Lookup is reading file contains, and when files will be very big, this lookup is not a good idea. What do you think. Can i use another Activity. I just read that the lookup maximum is 2000 records.

[@addyassefa5656]

Thats it