Slowly Changing Dim Type 2 with ADF Mapping Data Flows

[@MichaelN19]

thank you for the video and documents, managed to implement with bit of adjustments

[@athrun9800]

you can use Filter transformation instead of Exist transformation to prevent using the DimEmployees again as your right stream. You already have the needed columns in comparing column values since you are using Lookup transformation with DimEmployee as your lookup stream

[@learnnrelearn7553]

Yes .. thats correct

[@thebiccountant]

Very nice! Cannot wait to see how you handle the integration/replacement of the surrogate key in the fact tables ;)

[@tobman781]

Would this ADF Example work also using Snowflake on Azure as a destination database?

[@sid0000009]

Surrogate keys and Start/End Dates for SCD Type 2 cannot be same as shown in the video. For the Update set, the Insert branch should generate a new Surrogate key and its start date should actually be one more than the end date of the corresponding update branch.

[@MSDataFactory]

Yes, I should add that to the flow

[@sid0000009]

@@MSDataFactory great..if you have something as a difference use case between variable and parameter in Data Factory , please pass on the link. Thank you.. I found things can be achieved with either parameter or variable..not sure on the difference ...

[@MSDataFactory]

@@sid0000009 Parameters are what you use to make your pipelines and flows configurable. Param values can be passed between pipelines and data flows, for instance. Use variables to store temporary values in your pipeline. There are no variables yet in data flows.

[@learnnrelearn7553]

Very nice ......

[@paulmaksimovic2641]

Apologies but this video is very poor. The easiest way for people to learn is to build from the ground up during the video. I can understand showing a finished flow ( to highlight where you want to get to ) but the next step would be to build it again during the video. To just show a finished flow and then rush through it moving rapidly from one screen to the next makes if very difficult to follow and overwhelming.

[@MSDataFactory]

These videos are to be under 15 mins, so this is just an overview of the SCD Type 2 pattern in ADF using Data Flows. A more complete tutorial is not yet complete at this time. However, you can use this pattern from this complete example today using ADF. In the ADF UI, just go to New > Pipeline from Template > Data Flow > Slowly Changing Dimension Type 2 and you can add this data flow to your pipeline. Hope it helps!

[@kromerm]

Did you follow the link in the description, which I mentioned in the video? Here is the blog post describing the pattern: mssqldude.wordpress.com/2019/04/15/adf-slowly-changing-dimension-type-2-with-mapping-data-flows-complete/. But you don't need to recreate it. Just go to the ADF UI, go to New > Pipeline from Template > Data Flows > Slowly Changing Dimension Type 2. It's right there! :) Also, I highly recommend watching Bob Rubocki from Pragmatic Works' video where he walks through building it from scratch, which sounds more like what you're looking for: kzbin.info/www/bejne/pqSUY6JvaLt0pZI

[@PATRICKCHUAD]

paul maksimovic i agree. And also cant hardly follow since it is too fast. Slow down since the one who will watch this video are new to adf. Talking too fast is not helpful as the viewer will not get what you want to convey. Please improved.