Export to Cloudsql (PostgreSQL) from Cloud DataProc | PySpark Job - Google Cloud Platform

Export to Cloudsql(Postgresql) from Cloud DataProc | PySpark Job - Google Cloud Platform
github url: github.com/vig...
#Install libraries
pip3 install pyspark
pip3 install sqlalchemy
Start a cluster using the gcloud CLI
gcloud dataproc clusters start ${CLUSTER_NAME}
--region=${REGION}
Command to create single node cluster for sql instance disk size configuration
PROJECT_ID=learngcp-ace-guide-342819 REGION=us-east1 INSTANCE_NAME=mypostgresql CLUSTER_NAME=dataproc-demo1 gcloud dataproc clusters create ${CLUSTER_NAME}
--region ${REGION}
--scopes sql-admin
--single-node
--master-machine-type n1-standard-2
--master-boot-disk-size 500
--image-version 2.0-debian10
--initialization-actions gs://goog-dataproc-initialization-actions-${REGION}/cloud-sql-proxy/cloud-sql-proxy.sh
--metadata "enable-cloud-sql-hive-metastore=false"
--metadata "additional-cloud-sql-instances=${PROJECT_ID}:${REGION}:${INSTANCE_NAME}"
Command to submit pyspark job
gcloud dataproc jobs submit pyspark datatoPostgresql.py --cluster=dataproc-demo1 --region=us-east1 --jars=gs://demo_bucket_97/jars/postgresql-42.2.20.jar
Stop a cluster using the gcloud CLI
gcloud dataproc clusters stop ${CLUSTER_NAME}
--region=${REGION}
Resources:
github.com/Goo...
Apache Beam - Playlist
• Apache Beam - Python SDK
Subscribe to Channel: lnkd.in/ehFZbVH5
Follow us in LinkedIn: lnkd.in/gDT3ESdm
Follow us in Instagram: lnkd.in/gZ278ShA
Follow us in Facebook: lnkd.in/gQGF_3Eb
Follow us in Twitter: lnkd.in/gh7dZACW
Join in Telegram channel: lnkd.in/guFt2sAg
Join in WhatsApp group: lnkd.in/gAqkuDPA
Connect with me here:
Instagram: / thearrow0494
LinkedIn: / vignesh-sekar-sujatha-...
🙏🙏🙏🙏🙏🙏🙏🙏
YOU NEED TO DO BELOW THINGS to support my channel
1. LIKE
2. SHARE
&
3. SUBSCRIBE
TO MY KZbin CHANNEL
#gcpcloud #dataproc #pyspark