Chapter #9 - How to design data pipeline on gcp (Google Cloud Platform) ?

[@metaocloudstudio2221]

As a Data Engineer who worked on Cloud solutions for 7 years, I have never seen such a correct and clean solution. I am interested to see the same solution on AWS as well.

[@antonioflores4240]

I'm trying to build a data lake for an undergrad cloud class and honestly I was feeling pretty hopeless but after watching your video I feel way more prepared to tackle this!!!! Thank you!!!

[@ITkFunde]

Thanks Antonio I am so happy it helped in some little way 😊

[@sanjayroberts3453]

Incredibly helpful -- interviewing for senior level data engineering positions and many have data pipeline design sections. This is perfect for that

[@shubhamsadhuka]

This is very informative. Clean and clear of complex Jargons. Coming from a non-IT background I felt this video was very informative and descriptive. Thanks!

[@ITkFunde]

Thanks Shubham

[@kaushabvithani8504]

Siddhi baat no bakwass perfect way / An Eye opener to a beginner / wana Pursue GCP

[@ankursahu4731]

This is one of the great video to make me understand how Project architecture is there in GCP, I would request to please create a series to build end to end demo project in GCP utilizing all these services.

[@prithvinoojibail3734]

I am glad you have tutorials in English (since I don't speak or understand Hindi). Thanks for this video!

[@ITkFunde]

Thanks Prithvi

[@anandakumarsanthinathan4740]

Absolutely, absolutely wonderful and insightful presentation. Thank you very much. Just an observation. With BigQuery as your data warehouse, companies need not go for datamarts any more. BigQuery being a scalable solution, personnel from the various departments such as Sales, HR, Marketing, Finance can run their queries against a single database and still get results back without compromising on performance. Security can be implemented at the user/department level.

[@ITkFunde]

Thanks Ananda I cant agree more, BQ could be a one stop solution for vatious diff use cases

[@AmeyChittar]

Hi. So glad for posting this. I was going through the coursera course for this and I never understood the big picture. It's so clear now. Thank you so much.

[@ITkFunde]

Thanks Amey so happy it helped 🙏

[@vardannegi]

For data transformation and enrichment in GCP, Dataprep is the best tool

[@ITkFunde]

Yes Vardan Dataprep is good tool

[@hemanth1262]

Very informative to the beginners of cloud but helped with on-prem solutions, thanks tons!!!

[@Poornima_life]

It’s perfect explanation….you are genius man

[@joannwatu7603]

Thank you for sharing. Your explanation of the data lake structure for both on-prem and cloud solutions was very clear and easy to follow.

[@ITkFunde]

Thanks Joan ☺️🙏

[@chehz]

Thank you very much for all your hard work and the nice lecture.

[@sriharshanamuduri8694]

Preparing for the GCP Data Engineer certification. These videos are practical and helpful for my prep. Thanks!

[@hailstorm7868]

3rd party -> gcs stage can also be done via arbitrary code inside a container on "cloud run jobs". I did that, and downstream what is shown in the video almost exactly on one of my freelance projects.

[@naginafathima4562]

Great explanation. Can you also do a video on how the traditional DW system is used in AWS and AZURE?

[@jackleung2519]

really enjoy watching your video, very clear and well-explained.

[@pankajmanik7789]

This video is too good !! So many topics covered crisply in 1 video ... Thank you very much Sir :)

[@ITkFunde]

Thanks Pankaj ☺️🙏

[@gauravjha7249]

Very nice ......thanks for enlightning with so many information

[@hellmutmatheus2626]

I really learned a lot through your videos. Thank you very much, cheers from Brazil!

[@RizalIsmail]

Really great video. So humble you are and you give good first impressions at 1:25

[@ITkFunde]

Thanks Rizal ☺️🙏

[@manuelbolivar9791]

Thanks for posting and thanks for your hard working!! I'd like to see, whenever possible, similar videos for AWS and Azure cloud offerings. Using the same left side starting point. Cheers!

[@LeonidAndrianov]

Thank you, mate! I like your optimism as well as the explanation.

[@ITkFunde]

Thanks Leonid🙏

[@aamirsuleman9815]

Dataform is another amazing tool for ELT jobs

[@maayanyacobi8665]

great lecture, well explained - Thank you

[@gauthamvijayan]

Phenomenal Content.

[@hassansyed7125]

Awesome, easy to understand.

[@hareeshgunda8371]

Great work. We expect some realtime project design on GCP.

[@mainajnabee]

Profound knowledge you are sharing with us. Thank you so much! One request: whenever you refer your other videos, can you please display them in those bubbles in a corner so that we can click on them and go to those videos? You know those small bubbles with link that pop up..

[@SeafireCH]

Very nice video, great overview that is very helpful in navigating the jungle of google tools. Thanks a lot!

[@ITkFunde]

Thanks Pesche

[@justices_xs]

You might make my next job a bigger paycheck,nice video

[@ITkFunde]

Thanks buddy

[@sanjayg2686]

Sooooper Simple, you made sir, Thanks a lot

[@ITkFunde]

You are most welcome

[@syedaliuddin7046]

Thank you for sharing the information really helpful

[@ITkFunde]

Thanks Syed

[@prasann26]

Very nicely explained. All the best. thank you!

[@siddeshsharma968]

Really Great video, been looking for such explaination

[@SamvithDevi]

This has been very helpful. Thanks a lot for the video, and your detailing.

[@ITkFunde]

Thanks a lot🙏☺️

[@moushmidas2584]

Thanku sir As i requested you really created a video❤️❤️🙏 so nice video...

[@ITkFunde]

Thanks Moushmi hope you learnt something new from it pls keep sharing ur suggestions

[@victorganguly4230]

Nice explaination. 👍 Want to have some more videos like this for real time data processing with Google cloud (for example IOT data,sensor data)

[@swarupdeshpande4759]

Thanks. it was good presentation. To the point.

[@rushabhparmar7519]

Very informative and detailed! Thank you!

[@bhalachandrapatil]

excellent

[@ramkumarsundaravel8301]

Really good video and explained very well.

[@felipeozoski]

Great stuff thank you

[@vanib6123]

Very Informative !! Thank you for the great efforts !!

[@yshah9042]

Excellent.

[@MoisesTrelles]

Great explanation, very well presented

[@ITkFunde]

Thanks Moises ☺️🙏

[@siamakfarjami2116]

Thank you very much. Great video.

[@aneyewitness]

Awesome video - thank you! 👍 Would love to see an equivalent video for AWS and Azure also.

[@learner9670]

Too good learning material!

[@59600muslim]

Très intéressant merci

[@rohanasnani4768]

Loved it, Thank you soo much for this!

[@sanaayakurup5453]

brilliant brilliant explaination!keep it up!

[@sreetenap]

Thx a bunch! Excellent explanation.

[@ITkFunde]

Thanks Sreedhar☺️

[@pankajchaudhari95]

Thanks, Amazing video and very helpful

[@ChoandCho]

Thank you so much for the hard work you have taken for making these concepts very easy to understand.. Great work.. Keep posting such informative videos

[@venkatanithish7757]

Good one :) :D

[@swatitraveltales]

Reallly Helpful videos! An aadvanced video on Bigquery & one on Cloud data fusion too:). Thanks for sharing these videos

[@ITkFunde]

Thanks Swati sure will try ☺️

[@shameemferdous]

Amazing !

[@thammayagupta]

Awesome Explanation.....

[@meeral8703]

You are amazing!! Thanks for the clear video

[@ITkFunde]

Thanks Meera ☺️

[@informationsatellite5155]

great content..thank you so much...

[@RakeshGupta23]

Great explanation..one request .. please use mic 🎙️ as voice is Little low as compared to other educational videos.... great job..

[@ITkFunde]

Sure Rakesh thx for feedback and support 🙏

[@yuvabhagvatkathakarshriyag9305]

this is indeed a helpful video

[@ITkFunde]

Thanks a lot

[@samirchandra2637]

Very informative boss, keep going

[@ankitajain6817]

Very very informative and useful video.Thanks for sharing !

[@ITkFunde]

Thanks Ankita 😊

[@VinodKumar-wc6bu]

It's really excellent with kind of explanation . Sir do you have gcp pde paid course as well,plz comment

[@priteshraka6408]

Thanks for this, I was waiting for one such video from long time.

[@ITkFunde]

Thanks Pritesh🙏

[@khanbabababa6396]

Okay

[@ambar752]

Very well explained, loved it !!

[@ITkFunde]

Thanks Ambar🙏🙏☺️☺️

[@dr.kumaraswamybattula8428]

Thank you ❤

[@JOHNSMITH-ve3rq]

absolutely amazing video!!!!!!!!!!!!!!!!!!

[@ITkFunde]

Thanks John

[@lifeofpayal972]

Thanks for the video..!

[@sathishgowda4272]

Very good explanation, please make some videos related to complete data engineer project flow with pyspark or SQL

[@ITkFunde]

Thanks Sathish ☺

[@pawnyogi]

@@ITkFunde hi , is this is roadmap for data engineer using google cloud ?

[@pradeepdewani6267]

This is really helpful ..

[@ITkFunde]

Thanks Pradeep

[@praveengarg6090]

Great

[@emmanuelihetu9848]

This is soo good

[@CPat-yt6dm]

Thanks

[@amitjaiswal781]

Please make a video real time projects in gcp

[@hemantfegde460]

Good explanation Sir, could you please make video on Streamsets pipeline?

[@sunilsahoo5199]

can you do an end to end project in google cloud using this services.Then it will good for us.A lot of thanks to you for giving this type of content.

[@ITkFunde]

thanks for suggestion

[@sanjayg2686]

Thanks a lot

[@ITkFunde]

Most welcome

[@anandnerurkar8482]

good one. what is datamart, for datamart you have been using data studio,automl,looker? can u pls explain why??

[@capooti]

Nice intro! well done

[@ITkFunde]

Thanks a lot Roland

[@rdhundare]

Gud explanation

[@ITkFunde]

Thanks Rahul

[@shubhamsinnarkar2923]

AMAZING VIDEO!

[@manpreetsohal11]

Informative video. Why haven't you used cloud dataflow for ETL? Can cloud data fusion perform better ETL tasks than cloud dataflow?

[@pankajsangle2752]

om shree ganpatye namo:

[@oguzhangunes]

thank you !

[@zudt]

just G.R.E.A.T.

[@Pkmafffy]

Great video! I really appreciate the effort you put into this video and it really helped me out. Thanks a lot1

[@TradeWithCodeOfficial]

very helpful video.. one question.. do we need to create a VM instance if we go by this approach for GCP. Where do we trigger the gsutil command ? How do we go to the backend of fusion instance?

[@meeral8703]

Question sir: on the Google solution, why do you have two separate Cloud Storage components?

[@ArjunSingh-fm9sm]

Hey Thanks for this video, I have 1 question. If we want to do performance test for this type on msging queue application, what should be our approach.?? and how we can do this.. plz

[@nilavasen8631]

Dear Anshul, how are you ? I have been working as Data Engineer in on-prem solution of our company for more than 10 years and now wish to move to Cloud Based solution, like GCP Data Engineer. Can you please do help me with the below queries I have :- 1. I am having total 15+ years of IT Experience. Will it be a good decision to switch to Cloud Data Engineer Roles now ? It seems to me bit late to start journey in this domain now. Please do correct me , if I am wrong. 2. In order to start / work with the GCP Data Engineering Role, can you kindly suggest me what are the topics / tools / GCP Services I need to learn ? I have basic ETL Concept and also working with PLSQL , Python stuff. Thanks in advance !!

[@ITkFunde]

Hi Nilava, 15+ years of exp is quite a lot, in todays age tech roles are defying age and exp boundaries but in indian IT your total exp and respective role does put an image on your career profile thus i would suggest you move towards cloud data architect sort of roles or solution architects. Also if you starting from a clean slate then you should also think of becoming a Data Lead or Manager and manage a team of Data engineers under you. Think big !

[@nilavasen8631]

@@ITkFunde yes very true Anshul.. thats what I was thinking also. 🙂

[@kushagrak4903]

Sir really amazing video ! I was wondering where can I learn basics of the software which you mentioned ( SAP , IBM ) ?

[@ITkFunde]

Thanks youtube has everything for free or else try udemy for reasonable courses

[@user-kk4yp8vd4g]

In your architecture where is the pub sub messages are saving is in cloud storage and then etl pupeline pick and load in to big query?

[@vamsikri1234]

Really good video and good explanation learned a lot. I just wanted to ask one thing ,the cloud model you showed is it conventional ETL flow or its more of ELT with subset of ETL , because here we are using both the flow or its some kind of new approach which we can use, if would be really good if you could share your thoughts on this

[@PradeepSinghRajputOfficial]

The cloud storage in the middle is wrong. You can't store structured data in Cloud Storage.

[@anandakumarsanthinathan4740]

@Pradeep Singh, can't we store the raw data as text files? Acts as a datalake and also as a backup of our original, untouched, unedited data.

[@rushipuneet1270]

Is there any practical video creating a data pipeline?

[@krish_telugu]

We are facing problems with almost all services that we have been using with GCP, very limited plugins and features serve to basic use cases. even connectivity between services is also troublesome. long way to go