Data Engineers need this lineage trick!

In this video I take you through creating data lineage within your data set. This simple approach allows you to keep full and detailed information on what has changed, where data came from, how it was processed and more right there with the data itself. No need for centralised solutions, no need for extra tooling, and no need for connectors and plugins. This is an operationally very efficient technique and quite possibly allows the storing of the most detailed information since the schema you use for your lineage is up to you. You can also use this technique to save quality information alongside the data.
Find the full detailed instructions on creating this demo at github.com/davedoesdemos/Data... if you'd like to give it a go.
0:00 - Introduction
3:25 - Demo setup
4:18 - set up demo data
5:03 - Data Factory Pipeline (ADF and CSV)
9:16 - Data Flows Pipeline (Spark and Parquet)
13:35 - View in Power BI
14:49 - Wrap-up
For all of my other demos, go to davedoesdemos.com or go straight to the GitHub page at github.com/davedoesdemos/Demo.... Also please subscribe to the channel to make sure the latest demos show up in your playlist!

[@samuelrocha9079]

I missed your videos, Dave

[@DaveDoesDemos]

Thanks for the comment, I've missed making them!