Data Lake in a Day - Lab 5 creating the Power BI report

[@samuelrocha9079]

Very good explanation, Dave! It is helping me a lot. Thank you!

[@DaveDoesDemos]

Be the first to comment and win some kudos! What would you like to see next on this channel?

[@pranavkumar9030]

Hi Dave nice explanation with real-time tips to enhance the performance, would like to know in detail about CDC using Azure Data Factory

[@DaveDoesDemos]

@@pranavkumar9030 what do you mean by CDC here?

[@pranavkumar9030]

@@DaveDoesDemos Change Data Capture I tried in ADF, But there is no option in ADF except we can do with MSSql server with some tweaks, but for other sources like oracle, MySQL, or Postgres we can't, have to use third-party tools like Streamsets or Qlik Replicate. I am wondering if that you can point out any sources so that we can accomplish CDC without any third party tolls in Azure Data Factory. Thanks in advance.

[@DaveDoesDemos]

@@pranavkumar9030 unfortunately this is different for every platform. Even SQL has some constraints around it. Building a full solution will depend on what you're building it around. If there is a service bus architecture you might be better capturing there rather than the database. In an ideal world the database should be designed with time stamps on all rows such that you can just query for all rows after a given time (or between them, using tumbling windows). Alternatively, some platforms allow you to read from the log files to get this information. Occasionally you might be completely stuck for a way to do this and need to ingest a whole table every day and replace the data set in the lake entirely - obviously this only works on smaller tables but the majority of tables which don't include a date will tend to be smaller. The larger tables are usually some kind of ledger and will often include a time anyway. If you're really stuck on this look for a local MS partner who should be able to help out, or reach out to your local MS sub if you're at a bigger customer as you may have CSAs who can help.

[@pranavkumar9030]

@@DaveDoesDemos Thank you Dave for your complete reply 👍

[@samuelrocha9079]

One quick question, you are using Direct Query instead of Import for Synapse, why? It is because Synapse is very powerful processing data? Thank you again

[@DaveDoesDemos]

This was purely because it was a demo/training day, no real reason other than showing how :)