Azure data factory || Incremental Load or Delta load from SQL to File Storage

[@tahamansoor7396]

Best channel for learning ADF

[@mallikarjunap7302]

what a explanation madam.Its simply superb and amazing. Once again proved women has a lot of patience the way you have explained in this video. I am expecting more videos like that. Thanks once again

[@learnnrelearn7553]

Thanks a lot

[@gulchehraamirjonova2339]

Thank you so much You can’t imagine how it was helpful

[@alladamk]

Very clear explanation, thanks for the video

[@learnnrelearn7553]

thank you

[@sagar_patro]

Thanks for keeping the explanation so simple, it has given me a clear concept. Waiting for your new videos.

[@learnnrelearn7553]

This weekend another video will be uploaded. Please keep watching this space

[@desparadoking8209]

Very good explanation! 👍🙂

[@vemarajulasya9585]

very informative. please upload more videos on loading the data. Im working on azure it is really helpful for me hope you will do more on azure data factory.

[@learnnrelearn7553]

this weekend i will upload more videos

[@kids58tv13]

Informatic video

[@learnnrelearn7553]

Thanks

[@arupnaskar3818]

Great maam excellent.. 🧚💐💐

[@kajalcraftandart924]

wow so good infomation ❤️❤️❤️❤️keep it now ❤️❤️❤️❤️

[@learnnrelearn7553]

thanks

[@MonirHossain-qm8hh]

excellent, need more incremental load video, from On-premises SQL server more than 1 table to load Azure SQL Database. please upload

[@learnnrelearn7553]

Thanks for this positive feedback... Surely I will make video on incremental load from multiple table to one table

[@priyadarshinichandrasekara4656]

Hi... In this case, suppose we don't have any firstrow.newwatermarkvalue... what parameter should we pass to copy activity

[@kollikr]

Do you have any recommended approach to deal with the deleted rows? In your example inserts and updates are handled well but I am also looking for a solution to find any hard deletes in the source table

[@nandinisugandhi748]

Let me know if you have find any solution for this, I am too looking for deleted rows.

[@learnnrelearn7553]

To find the deleted delta records, you need to do a lookup to the source to find if any record is missing..... else from source we can have a delete flag to identify which record is deleted.(soft delete)

[@rahulpathak868]

thats a good one but, I'm assuming how we will set the values for first run? lets say for first run?

[@ZaikaUnlimited]

nice one thanks for sharing

[@learnnrelearn7553]

Thanks

[@Mukesh_Sablani]

@@learnnrelearn7553 I have provided same task i followed your video but i have stuck in copy activity even I want join your personalize training please conatct me my email [e-mail]. I really need your guidance madam

[@tapas6574]

Hi, good content for learning ADF pipeline. Can you please also share how did you enable on premise sql server set up in windows and mac system in order to show end to end . Thanks in advance.

[@raocoachingclasses598]

Excellent, good job

[@MoAnwar]

Can the watermark table and stored proc be created in either the destination DB or a separate DB than the source DB? We have a scenario where the source DB is locked down for making changes like these.

[@learnnrelearn7553]

Watermark will be in the destination...............

[@williamtenhoven8405]

Hi, it looks like this doesn't work with parquet. if I do the same in the pipeline expression builder in the copy data Sink and I press Validate ALL it says Syntax error : Opening brace has no closing brace. Thing is..... there is a closing brace...

[@anandk4759]

Can you share the ddl of the watermark table

[@MaheshBabu-lu9uq]

Excellent

[@learnnrelearn7553]

Thanks

[@lucamel8766]

thanks for explaining a complex topic, but pls check the audio :(

[@learnnrelearn7553]

Sure I will take care the audio.....Thanks for the feedback....

[@sabastineade2115]

Thanks for the great information but i have one question, can we not just implement the incremental load where the new data is being appended in the same file rather than creating a new file for the incremental records?

[@manojreddy8360]

I guess in sink expression we have to use append instead of concatenate with run id and creating new file.

[@learnnrelearn7553]

Yes we can append the file content to a single file using Dataflow....

[@investmentlifetime]

Great

[@learnnrelearn7553]

Thanks

[@cantonschool]

How do we write copy activity from Kusto (Azure Data Explorer) ? order_payload | where Last_modified_date > "@{activity('Lookup_max_watermark').output.FirstRow.maxdate}"

[@lucaslira5]

Thanks for the video, is it possible to just 1 file and adding the new records to it? So you don't have to create multiple files in the blob

[@Michael_Sam_05]

Nice

[@learnnrelearn7553]

thanks

[@priyab9225]

Hi, Thanks for this video. Could you please tell me how to sync/incremental load two azure databases dynamically that too using primary/incremental key as adding lastmodifieddate to existing tables is not feasible for us. Please correct me but this is for one table , how can we go for multiple tables. TIA

[@learnnrelearn7553]

Yes, we can do it using parameterized database/table....

[@pallarajesh]

Hii can you take mock interviews for azure databricks and data factory aspires

[@rohitsethi5696]

what is water map table . i have of staging table and worked in staging table

[@musketeers3344]

Can we use same Watermark table for different table , so that multiple tables water mark value get update in same table ?

[@dileeprajnarayanthumula4652]

Hi Maam, the explanation is very good I would like to know if there is any possibe chance providing personal training on ADF?

[@chinmaykshah]

Hi Can you provide script you have used in this video?

[@kishorkumar007]

Thankyou for such a explained video on Incremental load. One question : Actually I have exhausted my 30 days azure trial and I don't have any subscription left to create any resources in Azure. Could you please advice if there any any other methods to get some credits for free in azure ??

[@learnnrelearn7553]

Yes, You can always go for the Pay-as-You go option anytime you need with the same account. If you are still practicing with the features you can create one more free account with different mailid.

[@souranwaris142]

Hello, I have a problem with the incremental load I want to create an incremental pipeline from the Oracle on-premise server to Azure data lake(blob storage) I don't have Azure SQL. I just want to push in blob storage as a CSV file. in my case, I have confusion about where I should create the watermark table. someone told me in your case you have to use parquet data. please help me with this I am stuck for many days.

[@zahidalam7831]

+1

[@UmerAzeem]

is this method valid from Aws redshift to blob?

[@learnnrelearn7553]

not sure about the AWS redshift .... but this concept can be used for incremental data load in ETL Process...

[@anonymous-254]

you just proceeding and proceeding ahead... plz tell what is Incremental Load ?in which scenarios we can use in real life,,, you have explained any backgroung info

[@arihantsurana3671]

I followed all the steps.. but while debugging.. I am facing the problem to run pipeline.. my lookups are running but copydata one is failing... plz help..

[@sharavananp5570]

Hi, Awesome explanation. I did the same for pipline and found a error due to T and Z coming in output of lookup values, where as in sql the datetime format has no T and Z. I tried using formatDateTime ('2000-07-07T23:34:32Z',yyyy-MM-ddTHH:mm:ssZ') and still facing issues. COuld you kindly suggest whats the best approach to solve this

[@ushneshadaripa1163]

I think you need to use "timestamp" data type instead of "datetime" data type.

[@sharavananp5570]

@@ushneshadaripa1163 hi, thanks for the reply. But now I have solved it by using datetime itself.

[@sidsan000]

@@sharavananp5570 Hi Can you please tell me how u resolved this issue without timstamp and using datetime only

[@surendrag7537]

Nice Training ,please take one solution and use in this that will be blow.

[@learnnrelearn7553]

Thank You

[@learnnrelearn7553]

Yes, I am now implementing a mini project using azure data and AI components... i will upload the videos soon.

[@MaheshBabu-lu9uq]

Ok

[@ayaansk99]

Why we have used store procedure activity here anyone?

[@aadityasharma63353]

mam can you please provide the source code which is in the starting of the video.

[@SatishBurnwal]

The video is really informative. Just that your voice is very feeble to hear.

[@PersonOfBook]

That is too much work for what should be a straightforward task.

[@rahulchavan7822]

U r voice very very very low