What are some common data pipeline design patterns? What is a DAG ? | ETL vs ELT vs CDC (2022)

[@MasQred]

While extraction of data from operational database. Won't that affect the operetional databases performance. How to extract data without affecting it.

[@ITkFunde]

Good Qs thats why in old times ETL pipelines used to run overnight when the operational systems were not under heavy use but today there are seamless replication tools like attunity that can replicate the data from source by reading the logs.

[@patmclaughlin107]

Love this, man! Our engineers made it so hard to understand what DAG was. I thought I was not smart enough, but now I know they were either deliberately making it hard, or maybe they didn’t understand it themselves.

[@nikhilgurram6569]

Thanks!

[@ITkFunde]

thanks

[@bajicdusko]

It always amazes me how we can have knowledge like this one click away! Fantastic content, keep up with good work.

[@sumit12072007]

I was thinking the same while going through this video.

[@jeganj2009]

Excellent Explanation.. on all design patterns in simple way ....

[@daves4026]

Perfect. Full respect to your kindness and sharing of your knowledge

[@andygarnet7191]

Thanks man! Your explanations so clear and straight fwd. For years I spoke to do many engineers who would over -complicate pattern concepts or straight low ball the documentation to cover themselves when the pipelines blow up and impact the business. Keep up with these great videos!

[@AamirAzizYouTube]

Thanks so much, sir! This topic was a nightmare for me, you made it so simple to grasp. Keep up the good work!

[@rahuldey1182]

In my project, we are using CDC + EtLT design pattern for our data pipeline. All the design patterns of data pipelines are covered here. Very well presented, good job, keep going.

[@ITkFunde]

Thanks Rahul ♥️

[@ashishmaharana5865]

This is god send. I was just trying to implement CDC in my organizaton. This was most helpful.

[@deeptihazari3233]

How this Amazing channel was hidden till now ..this is called Quality content delivery 👍

[@sultanqureshi2766]

Though its not exactly related to my current profile but its make me happy to learn more about the whole software industry from core and you are best at making this understand by making it simple. Understood the 4(ETL, ELT, ETLT, CDC) data pipeline at once. Video was not long at all Thanks

[@ITkFunde]

Thanks Diwakar for your support as always 🙏☺️

[@ASHighlights668]

Very helpful sir your videos converts my nervousness into confidence !!

[@metaocloudstudio2221]

Good point , also the pros of using ELT over cons of ETL is creating normalizing tables and real-time materialized views

[@SheetalKumari-lk1vv]

Thank you.. it was amazing..

[@benoyeremita1359]

Really good explanation man. Hats off to you

[@mzeeshan]

Loved the details mate!.

[@ITkFunde]

Thanks Zeeshan☺️

[@francis191]

Fantastic tutorial

[@swaragupta7932]

Easy Explanation, Detailed video

[@ITkFunde]

Thanks Swara

[@harishb8790]

Amazing explanation. 👏

[@ITkFunde]

Glad you liked it

[@sunnyj1967]

Its a terrific presentation.

[@ITkFunde]

thanks sunny

[@vigneshbaskaran7931]

Love this content, Thank you so much for all the efforts.

[@ITkFunde]

Thanks

[@AnandhabalanRadhakrishnan]

Well explained, keep sharing valuable information like this.

[@mangesh4231]

Very detailed explanation, helpful. thanks a lot for all work and efforts.

[@HassanYoussoufFossi]

I studied Spark and read DAG many times but just understand it now that i'm watching ur tutorial. thks

[@raghurajsawant24]

You are doing a fantastic job. Love your videos.

[@Meowlah32]

Not exactly a backend developer or data engineer, but this video is very informational on the various data pipeline designs!

[@ITkFunde]

thanks

[@mayurarun]

This is such a gem video. This would help me so much. Great work.

[@rohithsai5265]

Great content 💯

[@vaidyanathashankar7441]

Fantastic explanation, thanks for the wonderful session.

[@Poornima_life]

Absolutely…I liked the video ,content and your valuable efforts….thanks

[@JJ-ki2mw]

Thank you so much the way you described it is so easy to understand

[@mohit.srivastava]

both this and the previous connected video explained the concept really well. thanks!!

[@Liubov_110]

Thank you so much for this detailed video 👍

[@greenshadowooo]

Thanks for your sharing ! 😀😀😀

[@almamun8291]

Thank you very much, got clear concept about data pipelines

[@jagss3472]

Lovely explanation and very insight details.

[@ITkFunde]

Glad it was helpful Jaga!

[@connect_vikas]

Love you brother for beautifully explained this.

[@ITkFunde]

Thanks Vikas

[@the.abhisheksinha]

nicely explained !

[@guidodichiara2243]

Great job. Keep going on!

[@PiyushSharma-jq8rr]

This was really good :-)

[@hsiaoshuang]

Very informative!

[@federicogonzalez7673]

Im glad that I found your video in my feed, nice one

[@javierruizdiaz8656]

Thank you, excellent Video.

[@itneka]

Thanks for the information

[@TheAfroKingPlay]

Very nice video man. Thanks I need this class. Take my like.

[@emmanuelaolaiya]

Great job and thanks

[@augugninfin1034]

Thank You!

[@ITkFunde]

♥️♥️♥️

[@francksgenlecroyant]

perfect video about Data Pipelines 👌, thanks!

[@AmanGupta-yf1hj]

Wonderful content

[@kristhomas8295]

Thank you so much for this!

[@jayanth1376]

👌👌👌

[@mohammadateef3339]

ur entry is osm sir

[@ashokrajur09]

nice one, very informative

[@SeanRomberg]

Thanks for the share - you have helped me better understand the pipeline automation software that delivers orchestration, ingestion, transformation, and activation all in one. This makes sense now.

[@wendypark3848]

I learned a lot network and data pipeline knowledge from you. It''s really hard to learn these from a book. Thanks a lot!

[@chinuamareashwar8146]

nice explanation brother

[@aditiaditi3302]

Thanks for sharing this video :)

[@altamashjawad6691]

Thank you so much, very nice and comprehensive video!

[@DepressedMonkeyGaming]

Great Video, Simple and detailed explanation

[@rdprasad2225]

thank you very much

[@jay2151000]

Thank you...

[@UTUBDZ]

Great content, thank you very much sir !

[@victoraf4274]

such an amazing video! not bored at all (im not joking) hehe

[@arond.g1120]

Feel like I am learning in my own language. ❤❤❤

[@ITkFunde]

Thanks Aron ♥️♥️🙏

[@ashisharora9649]

AMAZING

[@Vikas.007]

Awesome content 👍👍 Datamart video link in description plz share 🙏

[@ajaykiranchundi9979]

Thank you so much! BTW it was certainly not at all a long video.

[@ITkFunde]

Thanks Ajay ☺️❤️

[@davidcamiloespitiamanrique9]

Good one! probably, you can talk about AWS DMS and AWS GLUE

[@GabrielJambert]

Thank you

[@prashantprashant1291]

Your videos are full of knowledge.. are u Data Solution Architect

[@masterh6868]

hey your video as usual full for information and with crystal clear concepts of understanding.. Thanks posting such useful video as industries trends.. can you make video data pipeline , which does not fall DAG pattern ...... like Ml pipeline maybe..(not sure)

[@ITkFunde]

Thanks buddy for your feedback and suggestion ☺️

[@Lebrao09]

great video!

[@StaceyJ1908]

First, your videos are amazing....I have learned so much! I am looking at our current GCP implementation and trying to identify key risks across each step in the pipeline to determine if we have the correct controls in place or gaps...what are key risks to address at each stage of the data pipeline?

[@lwhieldon1]

DAG concept is talked about a lot in data science. Can you talk about how this concept in data science correlates with the DAG design?

[@VlasTrunov]

It's good you focus on DAG's. But for those new to the subject it might be too abstract, I guess. What I would do is I would show how things flow in Airflow, for example, for those who perceive information visually. This way you would spread the (butter on the bread) information in your video uniformly, makking people get the grasp of the information in one pass, if you know what I mean. It's just a suggestion. But to me personally, the detailization you give is perfect.

[@ITkFunde]

Thanks Vlas such useful feedbacks helps me better my content. I will defintely take your thaughts and do something better nxt time.😊

[@ganeshsrinivasannv4296]

Thanks and it's a great work. Can you share a content on the data captured received as XML messaging pattern and advise on how to store that

[@sreechalasani9268]

Thanks for the video ! In the cdc model, rather than capture all the prior versions, when a new version of the data comes in - why can’t the previous versions be invalidated? Wouldn’t that be more efficient?

[@upendrakumar-ok3tr]

Can you please make a video on Baremetal and Hypervisor

[@TheyCalledMeT]

would you put data- cleansing / preparation as part of the t of EtLT pattern? or in the T?

[@brookster7772]

Great Video! Can you tell where a Vector Database fits into this model? Isnt it at some point all data must be converted to Embeddings / vectors to be stored into a massive Vecotr Store to be used for AI similarity searches?

[@arijitsinha2955]

Can you make video on data bricks along with an example please ?

[@deepsy4786]

I would like to discuss on considering CDC as a data pipeline design pattern. My understanding would be that CDC is more related to data modelling concept. You would have to build an ELT or ETL pipeline anyways. CDC more relates to Load or Transformation technic instead of being an individual pipeline. However, all the insights shared were helpful and did helped me relate my work with some of these concepts.

[@ambarishhazarnis9531]

Here CDC referred to storing the delta on a separate table. This way we don’t need to do a read on source table again to extract the change.

[@arundhutinayak8221]

Now I can put technical terms to my current task. Can you do something on API

[@lastboomer6164]

Hello I very much appreciate the training. Would you consider a white board exercise whereas the ETL Jobs and Transformations are using a Metadata Data Driven ETL. - I learned that this is a good practice....but one downside is that this data design can not feed a data catalog "lineage"

[@anilmantri2139]

Hi, how to pull the source data into EL DAG in the CDC pattern. I mean what tech stack to be used?

[@imnischaygowda]

What is purpose of Sink ? Can t we store data directly to DataWarehouse ?

[@debrajpradhan5500]

Sir, I am interested on AWS analytics.so can u plz tell what AWS data services read 1st,2nd?

[@lcsxwtian]

At CDC, you had said that max() would get the latest snapshot of the data. I am assuming max() would get the maximum count of the data - correct? If that were the case, what if the last change was to DELETE some data, then I don't think max() would be right?

[@AnishBhola]

yes your right! timestamp based cdc is generally not a good option to process deletes. There are other types of cdc such as log based (most optimal) which you can use for such situations. This video primarily talks about implementing difference based cdc (where 2 snapshots of target systems are compared).

[@chinnap8987]

coding is required to learn this?

[@SamS-oi5pz]

Hi how do we identify changed data from source?

[@adamjapal7370]

do you have a reference or pdf book file of the data pipeline concept? if you do, could you take me to the link? thank you.

[@veeek8]

'Hope you're not bored', never 😁

[@pm4306]

please give some concrete business example instead of 'n' and 'n+1' as as example will help to clarify and walk thru in a better way - i think you should give concrete real-life business examples for all cases that u discuss......you are missing actual business examples in your videos.

[@antonfernando8409]

Never heard of most of the terms like (ETL, ELT, CDC) mentioned, I guess these are specific to cloud computing, still in terms of data pipeline, its useful to learn I think. Thanks

[@egor.okhterov]

No, it’s not about cloud computing. It’s about data analytics in general. When you want to build web dashboards that draw graphs of some business processes or want to analyse customer behavior, you build this data pipeline. TLDR: you cannot run SQL on your logs. You need to push your logs into MySQL in order to be able to query your logs.

[@ITkFunde]

Hi Anton these terms are quite old but have become more prominent with new age data management. May be you are not from Data, Business Inteligence background, but its good to learn these

[@ernesto8738]

and here I am with a cyclic graph problem {{{(>_

[@dylanmccullough2679]

Question regarding the ELT pattern. You said that you should use SQL at the (T)transformation part. Could you use spark instead of SQL at this point? For example - Data Factory Data Flows, instead of putting compute pressure on the EDW with SQL queries?

[@chandrakanthotkar7262]

Whenever we say ELT basically we do transformation after data has been landed in DWH or Database. Like Bigquery (GCP). As Spark engine is basically used for transformation during flow.

[@GernPudman]

Thanks!