Load Data from GCS to BigQuery using Dataflow

[@kinjal_suryavanshi]

So helpful ✨

[@techtrapture]

Thanks Kinjal

[@chandrasekharborapati4599]

Hi bro.. good day.. i have one query.. is it possible to delete bigquery records after processed all the records using dataflow job in gcp. Using java api.. please provide a solution if it is possible...

[@Sriharibabup-w6f]

what are the Transimittion we used in Data Flow

[@chetanbulla9185]

Nice video ..I am able to execute the DataFlow.. Thanks

[@archanajain99]

hii, i need your help that i need to create a GCP dataflow pipeline using Java. This pipeline should take file in GCS bucket as input and write the data into Bigtable. how to work on it? please guide.

[@techtrapture]

Here some idea from another video kzbin.info/www/bejne/gaOlZ3emoNt8eacsi=ZWBjt3CrCVJmwkQ5

[@SugunaA-k1d]

Hi , I have tried the same but i am facing an issue. Error message from worker: org.apache.beam.sdk.util.UserCodeException: java.lang.RuntimeException: Failed to serialize json to table row: id,name org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:39) Could you please shed some light on what is going wrong .

[@iloveraw100]

I need to remove the header rows as this is getting populated. How to do that?

[@NangunuriKarthik]

Hi can you please me how to move the tables from Oracle to big query using google dataflow

[@AdityaBajaj-z7d]

How to upsert data in Dataflow?

[@mulshiwaters5312]

Good realtime handson experience. I assume when I create Data pipeline using Dataflow which get executed when I click on RUN JOB. How I can use this pipeline for daily data load from GCS to BQ ? is this possible with Dataflow or do I need tool like Cloud Composer to schedule this job at certain intervals ?

[@techtrapture]

Cloud composer is too costly , you can schedule it using cloud scheduler , check this video for your use case kzbin.info/www/bejne/mGacZJurh8yLn8U

[@Eno-AbasiAkpan]

Where is the code for this video?

[@premsoni0143]

Is there need to configure VPC for streaming between cloud spannerto GCP pubsub? I tried to set up and it failed using: "Failed to start the VM, launcher-202xxxx, used for launching because of status code: INVALID_ARGUMENT, reason: Invalid Error: Message: Invalid value for field 'resource.networkInterfaces[0].network': 'global/networks/default'. The referenced network resource cannot be found. HTTP Code: 400."

[@techtrapture]

It depends on how you are streaming...if you are doing it using dataflow which i seem from error then it's an error for dataflow worker vm. So you are missing details in dataflow configuration.

[@sikondyer2068]

How to load csv file with comma in data? do you know how to escape the comma? thanks

[@techtrapture]

Comma is deliminator or its part of data?

[@sikondyer2068]

@@techtrapture It's part of the data, like for example the column Address has a value of "Bangkok, Thailand"

[@gnm280]

i have exaclty the same issue with data rows with comma@@sikondyer2068

[@chandanpatil2704]

Hi, I have been using same approach like you but with different CSV file(UDF is same) but I am getting following error (Loyalty Number is Integer column): Error message from worker: org.apache.beam.sdk.util.UserCodeException: java.util.concurrent.CompletionException: javax.script.ScriptException: :5:12 Expected ; but found Number obj.Loyalty Number = values[0]; ^ in at line number 5 at column number 12 Can you tell me what the error is actually?

[@techtrapture]

Check if datatype of bigquery column and CSV data is same

[@VishalKumar-z4p9v]

How can we load the same data from csv file to pubsub topic and then through dataflow job in bigquey ?

[@techtrapture]

First thing you need to create dataflow job with template "Text files on Cloud storage to Pub/Sub" and now to load data from pub/sub to bigquery you don't need dataflow , Google added new subscription option for pubsub where we can directly load to BQ.

[@natannascimento7388]

Hello I am getting the error below. org.apache.beam.sdk.util.UserCodeException: java.lang.RuntimeException: Error parsing schema gs://fazendo/mentloja.json Caused by: java.lang.RuntimeException Caused by: org.json.JSONException Can you help me?

[@rahulhundare]

One more Question:- Why do we need to specify temp folders here?.

[@techtrapture]

During job execution it stores some metadata and temporary staging files in temp folder. You can monitor it during job execution

[@SarvaKaahi_108]

Can you share the CSV file?

[@techtrapture]

Help me with your email id , I will share it with you

[@shwetarawat4027]

Can you also attach the .csv file so that we can download and use?

[@techtrapture]

Sure, Can you share me Email id , i will share it with you for now.

[@shwetarawat4027]

@@techtrapture I've used another .csv file for now... thank you

[@shwetarawat4027]

Also, when trying to give the bigquery dataset name while creating the job i.e project ID:datasetname it is giving error : "Error: value must be of the form ".+:.+\..+"".... how to resolve this? Also, when I am giving the table name, it says' Table not found'

[@techtrapture]

Use format Projectname.datasetname.tablename

[@shwetarawat4027]

@@techtrapture I am doing the same, still the same error

[@adijos92]

Can you send me that CSV.format file and all three files to my mail id..??

[@techtrapture]

Share me your email id

[@MiguelPumapillo-jd3ug]

thanks

[@jaykay7057]

How to create USD as I do not have any java knowledge

[@techtrapture]

USD?

[@Kashishsethi_]

I think by USD he mean user define function

[@VarshiniAleti]

can you pls share .csv file

[@rahulhundare]

Hello I am getting below error. org.apache.beam.sdk .util.UserCodeException: java.lang.NoSuchMethodException: No such function transform at org.apache.beam.sdk.util.UserCodeException.wrap(UserCodeException.java:39) why so

[@techtrapture]

This is something related to your code you are using...don't think anything related to GCP environment

[@rahulhundare]

@@techtrapture Yes you are correct that is with invalid function name inside the code. Thanx for prompt reply... :)

[@techtrapture]

@@rahulhundare glad you found it.