13. How to copy last modified files from multiple folders into another folder using ADF pipeline

In this video, we learnt how to get the last modified filenames from multiple ADLS folders and copy those files to another ADLS location using ADF pipeline
#adf
#azuredatafactory

[@luvvkatara1466]

Thank you Annu for this awesome content and Explanation

[@sravankumar1767]

Nice explanation Annu 👌 👍 👏

[@azurecontentannu6399]

Thankyou Sravan😊

[@KaraokeVN2005]

Good content!!!!!!!

[@vishnugupta9955]

Thank you 🙏

[@akeemadedigba8279]

Nice video. But what if destination is multiple folders and not dumping to one folder., what do you need to add or change?

[@azurecontentannu6399]

Thanks. In that case sink dataset needs to be parameterized and we need to use @item().foldername to specify the folder which would be dynamically taken by the source foldername

[@sep5959]

good explanation annu garu

[@azurecontentannu6399]

Thankyou 😊

[@RajeshAkarapu]

Awesome..!! Little bit fast 😮

[@azurecontentannu6399]

Thankyou..will consider your feedback

[@idreesiqbal561]

Thanks a billion for this video

[@azurecontentannu6399]

Welcome 😊

[@KaraokeVN2005]

I have question: Can we use incremental Load from Blob to Azure SQL Server with this pipeline or we have to make a store proc to get the lastest record/file.

[@KaraokeVN2005]

I have this use case : Customers requires me to import files from Blob Container and Just load the newest record into SQL Table , how can i do that. Thank for your help !!!! 💓💓💓💓

[@sravankumar1767]

If we don't know which folder has latest modified files how can we write expression for this scenario

[@akeemadedigba8279]

Can you this exact video in opposite direction ?

[@azurecontentannu6399]

Hello sorry I didn't get what do you mean by opposite direction

[@OnlineForward]

I appreciate all your videos. Honestly speaking, in most of your examples you refer to "CONTAINERS" as "Folders". I hope this is not offensive.

[@azurecontentannu6399]

Thankyou for the feedback . Yes I agree , I did that by mistake. Thanks for pointing out

[@userazure-th9bs]

If i do have hundreds of files in the folder and also every day a file gets placed under that folder then won't it be an ideal way right? How can we do that then?

[@repalasanthosh7452]

thank you so much. Is there a video on how to load /copy a specific file from specific folder to a specific table using parameterization?

[@azurecontentannu6399]

kzbin.info/www/bejne/fZWynoKMZ6hknNU

[@azurecontentannu6399]

kzbin.info/www/bejne/d17XooJsfLGomrc

[@repalasanthosh7452]

@@azurecontentannu6399 wow amazing tutorials, tq so much. i have a task just need an idea from you to implement. we have different files in different folders which manually should be uploaded in a storage account from where we need to load data into respective synapse tables. we need to automate this without manually providing folder names or file names in runtime also. And one more thing is we need store these files in a backup folder in the same storage account/container. Can you please help me with your inputs? TIA.

[@Tillutelugu]

Do scenarios as early as possible 😊...

[@azurecontentannu6399]

Sure 😊

[@purvajpradhan8841]

How do we do it for Sub Folders in the same container?

[@hritiksharma7154]

Suppose I have written some code in databricks and it is throwing some error . Can we retrieve this error message from databricks notebook and send it on a mail which contains pipeline name and error message both.

[@JesusSevillanoZamarreno]

i have folders for different entities that have underscore like 'ENT_B2KBA'. Im getting error in the child get metadata activity saying that ' the specified resource has invalid characters' What can i do???

[@jramachandram884]

Hi, one doubt if we don't know specific folder like demo, each folder has different latest modified files how can we write the logic

[@azurecontentannu6399]

Hello.. Thank you for watching the video.. Please check part 14 of the playlist to get the latest filename from multiple folders.

[@krishnachaitanyareddy2781]

Hi can you do video on how to transfer files from Adls gen 2 to sftp server. Thanks

[@akeemadedigba8279]

How to copy last modified different files from one folder to each respective folders using adf?