Database replication(CDC) || Replicate CloudSQL data to BigQuery in real-time using Datastream

Рет қаралды 3,654

TechTrapture

Күн бұрын

Пікірлер: 15

@malleshdatta8832 3 ай бұрын

hello @techcapture Can you please upload videos ..from Teradata to GCP migration

@lug__aman Ай бұрын

i have my data in postgres when in the last I validate it show replication error I run all commands for replication how to replicate ?

@ashwinjoshi3331 Жыл бұрын

Thanks for the video. It helps a lot. One question, how to import oracle or mysql data using cloud data fusion ? Is there any material available to perform that scenario ?

@techtrapture Жыл бұрын

In Data Fusion you can create connections by adding jdbc drivers for respective Database. You will easily find documentation on data fusion on connecting to MySQL/oracle. Here one sample article I found on medium rspraneethkumar.medium.com/oracle-to-bigquery-create-a-pipeline-to-extract-data-from-oracle-and-load-it-to-bq-using-data-69ac855f473f

@ashwinjoshi3331 Жыл бұрын

Thanks . I actually referred to same document . While creating connections string for data source , there are connection arguments to be mentiond . As per documentation , It mentions as " Connection Arguments: A list of arbitrary string tag/value pairs as connection arguments. These arguments will be passed to the JDBC driver, as connection arguments, for JDBC drivers that may need additional configurations. This is a semicolon-separated list of key-value pairs, where each pair is separated by a equals ‘=’ and specifies the key and value for the argument. For example, ‘key1=value1;key2=value’ specifies that the connection will be given arguments ‘key1’ mapped to ‘value1’ and the argument ‘key2’ mapped to ‘value2’. " What does it mean ? These details I am not able to find.

@techtrapture Жыл бұрын

@@ashwinjoshi3331 thanks for letting me know...Let me try it my environment. I will get back to you. Also let me know your exact requirements so I will try to find suitable solution.

@ashwinjoshi3331 Жыл бұрын

Vishal, I really appreciate for your efforts . I am from MSBI (SSIS,SQL Server) background. So I am novice over here. So my doubts could be simple but hope you will consider it. On my laptop , I need to do a simple task of creating a flow i.e. a) take data from RDBMS source (Oracle (preferred/mysql /postgres/SQL server) b) Push the data in staging/ final table (BigQuery) c) Schedule a job It's a typical kind of data warehouse scenario . This can be done by various ways- Dataflow,Cloud Data Fusion, Datastream (current video),Python (as an intermediate). To my knowledge , Cloud Data Fusion almost does these tasks and it's much more simpler also. But there I faced issue related to connection arguments (explained above) . So I am unable to proceed with other tasks related to POC. It would be really helpful if you can suggest here.

@techtrapture Жыл бұрын

Thanks for details. I got your query now. Yes Data fusion you can setup , that I will try in my local. But if you need real time replication then Datastream is more easy but it support MySQL oracle and postgress .