Data Pipelines: How to make them better

[@pipicovers]

Steps to make pipeline better 1. Good auditing and logging: error handling 2. Repeatable and identical 3. Self healing: finding a way to find the delta , log files and compare, add a data lake before data warehouse , add hash or water marks before compare 4. Decouple EL and T: Landon Rae formate, transform to Dwh, make reporting table clean, 5. Always available: trancate and load refresh faster than update. Or build semantic layer 6. CICD: coded, git connected, versioned , rollbacks

[@MrHaste12]

Thanks for the video. Do you have an example of a pipeline built from scratch following the best practices mentioned in the video? Text/book or course-based doesn't matter

[@user-ho4bl8wo2p]

great video thanks for your effort but could you make more videos about building pipelines with open source tools that would greatly benefits people who just started in that field before jumping directly in the world of cloud