Cloud Composer - Orchestrating an ETL Pipeline Using Cloud Dataflow

GitHub url: github.com/vig...
Follow us in LinkedIn: lnkd.in/gDT3ESdm
Follow us in Instagram: lnkd.in/gZ278ShA
Follow us in Facebook: lnkd.in/gQGF_3Eb
Follow us in Twitter: lnkd.in/gh7dZACW
Join in Telegram channel: lnkd.in/guFt2sAg
Join in WhatsApp group: lnkd.in/gAqkuDPA
Connect with me here:
Instagram: / vignesh909_ss
LinkedIn: / vignesh-sekar-sujatha-...
🙏🙏🙏🙏🙏🙏🙏🙏
YOU NEED TO DO BELOW THINGS to support my channel
1. LIKE
2. SHARE
&
3. SUBSCRIBE
TO MY KZbin CHANNEL
#gcpcloud #datafusion #bigdata #dataengineer #cloudplatform #cloudcomposer #composer #dataflow #apachebeam

[@pradipfunde2099]

Useful video

[@cloudaianalytics6242]

Thanks a lot

[@vigsulagam2294]

Thanks!

[@cloudaianalytics6242]

😊

[@user-so1ev9xd2b]

I will appreciate your effort. If possible could you please specify the way to replicate the same by end user step by step if you create storage bucket and create the script file and to save the same in which location so that we will follow you to understand in detail the way to achieve the same in different scenario. Hope you understand.

[@learnwithvivs7063]

This is very useful, thanks. Can we trigger trigger Apache Beam pipeline on Dataflow without using Template? I think Template is mandatory with composer 2

[@cloudaianalytics6242]

True

[@ashwinjoshi3331]

Thanks a lot . This is really helpful. One question - here source is CSV file . What changes are required to be done if the source is "Oracle on premise data" . I actually tried with Datastream but due to limitation at client end, we are asked to use Dataflow. I tried to check but could not get any specific detail . Could you please suggest any reference here from oracle connection point of view ?

[@cloudaianalytics6242]

Sure. Please find the link below precocityllc.com/blog/configuring-cloud-composer-for-oracle-databases/

[@ashwinjoshi3331]

Thanks for the link. It's really helpful.

[@Ar001-hb6qn]

I am unable to create a GCP Composer environment. After around 45 minutes it shows the error "Some of the GKE pods failed to become healthy". I have configured the setting and given the necessary access. I am using composer-2.7.0-airflow-2.7.3. But it failed to create the environment. Can you please help with this? Thanks.

[@student_voice]

It is better to provide a step by step note through Google drive.

[@cloudaianalytics6242]

Sure 😊