#94. Azure Data Factory: Implement SCD Type II

This video takes you through the steps required to implement slowly changing dimension type II.

[@kromerm]

Bravo! Nice example of building an SCD Type 2 in Azure Data Factory

[@AllAboutBI]

Im honored and elated to see your feedback on the channel and in twitter Mark. I have learnt a lot from your videos and blogs. Thanks for all the knowledge sharing 🙏🙏🙏

[@kromerm]

@@AllAboutBI Keep up the great work :)

[@mehmetadalar7606]

Filtering the is_active at the end of your video saved my day! Thank you!

[@sarthaks]

Great Content!! . Clear and precise. Cant wait more for more scenario based ADF Data flow implementation..

[@MrZoomok]

Thanks for this tutorial, it's very helpful. I just modified this to use Surrogate key and take changed data only. it works perfectly. Thanks

[@iu665]

Small suggestions from our side - For which situation for what scenario we can use this.. ? what type of problems we can solve with this. And Don’t make video complicated to understand (for all videos).. your all videos are very useful thanq mam

[@pankajbhatt7373]

Very clear explanation

[@venkateshreddy2137]

Thanks for making vedio.its very helpful foor PPL from other technology..

[@oriono9077]

No words to thank enough. Excellent

[@AllAboutBI]

Thanks so much

[@krrishb]

i was working ETL tool .i loss job past few months ago .one of My friend suggested me to learn Azure Technology .I started looking for good videos In you Tube today only i saw few videos Very very very useful for me madam & easy explanation.thank you very much for such a clear explanation madam .once i complete all videos i hope i will get job on azure in two months . if you have any pdf document prepared by you please share me madam.thank you so much madam.

[@pushkarwani6099]

Hello Mam, Can we update the start date dynamically for the updated record, as previous_column(End_date +1). I am trying to do that but unable to figure out.

[@rajuchoudhary8695]

Thank you very much for such a helpful video..

[@AllAboutBI]

Glad it helps.

[@alphacharith]

Thank You for the great video.

[@AllAboutBI]

Thank you 🙏

[@Obizzy8]

Good content as always!

[@AllAboutBI]

Thanks 👍

[@rudresheswarappa4624]

Hello Mam, Will the flow work in the same way if there are multiple updates occurs in the source , like Nattu's department changes 2 times in the source, 2 records to get updated

[@ADFTrainer]

Here what alter roe does and sink does in updation

[@viralkotak6581]

Hello Mam. Thanks for the clearly explaining implementation steps for scd2. Just one doubt what is Hasref in the upper part of the data flow?

[@abhifimois345]

thanks for this. Will this work if we have multiple changes on the same day

[@sitaramakella8922]

Nice video..thank you .

[@sivchavan]

Great video..can we create video on tumbling windo scheduling part using dependency.

[@shriyankarath5388]

From where can i get dataset

[@easyyuz2707]

Thank you TJ! Great video, very clear and informative as always. How can we add a summary email at the end of this pipeline that shows the audit information in terms of how many rows were inserted, updated and send a notification email to team for success and failure.

[@easyyuz2707]

is it possible to send the summary email in table structure also for inserted and updates records

[@easyyuz2707]

Thank you very much Suba. I still do have this question. Why don't we create the destination Dimtable surrogate key column with Identity, so that we don't need to worry about generating a SK, maxSK or cross join and all that. What do I miss here?

[@AllAboutBI]

Hi Emma.. we can create identity column in sink and ignore mapping that field in mapping tab.

[@easyyuz2707]

Thanks a lot Suba. Great! That'd make life easier :)

[@kunalrajput7077]

Thank you for this tutorial. I have a basic question and I would appreciate your response. In this data flow there are two source streams, the first source stream has a sink which uses SQL table. This same SQL table is being used as a source in second source stream. My question is are these source streams execute in a sequence or in parallel? Please correct me if I have misunderstood anything. Thanks for your time.

[@manojkbabu1094]

kzbin.info/www/bejne/p4O5oJ1oj7d9l68

[@shivanidubey1308]

Thanks for video

[@AllAboutBI]

Thank you

[@rakeshmishra4650]

Hi This is indeed a nice tutorial. Thanks for the clear explanation. I've implemented it but in my case, for changed records, the IsActive flag remains still 0 whereas a new rows correctly inserted with the changed values but the exiting rows flag is not updated though and the updated date (for existing record) remains the same as '9999-12-31'. please help me

[@AllAboutBI]

Pls check out the mappings. It should be the culprit

[@rajeevsharma2664]

In the first flow, before existing checkin components - there is two derive transformation. One is coming from the csv (obvious). How the below one i.e. HasRef connected with the SQL (i.e. sink). Have you passed any variable?

[@AllAboutBI]

I'm sorry , I don't get ur question. Can you pls rephrase??

[@rajeevsharma2664]

@@AllAboutBI There are 2 flows inside a pipeline and each flow has got 2 derived control. I want to know for the upper one, how the HasRef been populated?

[@maheindra5572]

@@rajeevsharma2664 You are right! @Mam, Can you please answer this question

[@codeworld8981]

I tried this but getting some issue in active flag value updated as 1 for old records... Can any one clarify

[@ratandara1099]

Hi, thanks for videos I am learning a lot. I need some advice in our scenario we are using parquet file as sink not sql server, how do I update rows in parquet file.

[@AllAboutBI]

I don't think ADF currently allows appending or updating files.

[@santhoshgondela2951]

Hi Madam, Nice explanation.Could you please let me know the Git repository url..

[@விரேவதி]

Super mam👌

[@AllAboutBI]

Thanks mam. 🙏

[@sathishkumark8386]

Great

[@AllAboutBI]

Thank you

[@amiyamaharatha2154]

itslike story ...difficult to understood

[@harishkonakandla]

informative

[@AllAboutBI]

Thank you

[@rajeshprajapati5625]

ma'am your method is wrong

[@AllAboutBI]

Please tell me where, so that i can take a look.

[@rajeshprajapati5625]

@@AllAboutBI in Update part there is update all the existing data not perticular one like if I update only one recored then it would be updated all the existing record and making flag 0 and only one recored which is updated there is flag 1

[@AllAboutBI]

Thanks , would be helpful to the viewers. I will try giving it a shot too

[@BalavardhanM]

@@AllAboutBI Could you please explain how to handle if source is having duplicate records , for example : file 1: id,name,dept,startdate 1,sachin,insurance,01/21/2023 2,dravid,finance,02/22/2023 1,sachin,trading,03/21/2023