ADF metadata driven copy activity , how to process multiple tables dynamically in ADF copy activity

[@TheyCalledMeT]

since most ingestion tasks typically go to a data lake, wouldn't it be intuitive to also use the data lake for the config/metadata table(s)?

[@saqibkhan-ih1uu]

thanks for the detailed explanation. how and where to add audit columns like load timestamp to target tables?

[@ketanmehta3058]

@TechLake How we can add filter clause to the query?

[@TRRaveendra]

Use lookup query and add sql query there

[@ketanmehta3058]

@@TRRaveendra I mean this process will create three pipelines (Bottom_Level, Middle_Level, Top_Level) but I do not find any option to apply 'where' clause. I can see only the column details for the selected table. How do I mention my filter criteria to a specific table?

[@TheyCalledMeT]

19:24 really select max watermark from target table? not select last watermark from watermark-table filtered for this table name?

[@RodrigoBocanegraCruz]

Hello, is there any way to define wildcard paths in the data lake destination? for example to have a folder structure in year/month/date/*. Thank you!

[@guptaashok121]

What all sources are supported for this?

[@snad256]

Hi, From where did you get the Sales table?

[@TRRaveendra]

U will get default sample database in azure sql and dedicated sql pool

[@agbtttt5717]

Where can we give PK name for upsert logic for each table?