Seamless Data Integration: ETL from Google Cloud Storage Bucket to BigQuery with Cloud Functions

  Рет қаралды 22,527

Cloud Quick Labs

Cloud Quick Labs

Күн бұрын

Пікірлер: 38
@erinquinn6852
@erinquinn6852 17 күн бұрын
What changes to the code needs to be made for CSV files? Does the yaml file need to have different structure as well as changes to the main code?
@cloudquicklabs
@cloudquicklabs 15 күн бұрын
Thank you for watching my videos. Code should be modified accordingly csv read functionality of python and rest all should be same. I shall create new video on this topic soon.
@nathaniasantanigels
@nathaniasantanigels 3 ай бұрын
Is it possible for me to create a pipeline that scrapes data from Google Sheets, and whenever there's an update in the Google Sheet, the bot updates the data without overwriting the existing data? How can I ensure that when there's an update, it won't overwrite the previous data?
@cloudquicklabs
@cloudquicklabs 3 ай бұрын
Thank you for watching my videos. It should be possible considering below points. 1. Google document event should be captured in GCP (need to figure out if it is possible) 2. We can stop overwritten with the pipeline set up.
@shreyojitdas9333
@shreyojitdas9333 20 күн бұрын
sir can you please create an entire app(microservices preferably) with flask and gcp(cloud fuctions and most of the services.Please.I love your contents.
@cloudquicklabs
@cloudquicklabs 19 күн бұрын
Thank you for watching my videos. Indeed I have plan to do more videos on GCP soon.
@Prathibha-v8c
@Prathibha-v8c Ай бұрын
I have experience as an Azure data engineer. I was recently assigned a GCP project in that project they are dealing with XML files. from GCS bucket XML file to big query table using cloud function.
@cloudquicklabs
@cloudquicklabs Ай бұрын
Thank you for watching my videos. I have plans to create many videos on GCP ahead. I shall consider requirement as well there.
@osmanbaba1485
@osmanbaba1485 20 күн бұрын
So would the equivalent of Azure Data Factory be this? Or would it be a different google product?
@tejaspise4638
@tejaspise4638 9 ай бұрын
Great video, where can i learn to write the script like the one used in the video(i want to learn how to use the google cloud library)
@cloudquicklabs
@cloudquicklabs 9 ай бұрын
Thank you for watching my videos. Glad that it helped you. To get started you can check GCP documents for developers using APIs, I shall create new videos in future as well.
@theamithsingh
@theamithsingh 11 ай бұрын
do an entire series that, shows how to engineer data on gcp :)
@cloudquicklabs
@cloudquicklabs 11 ай бұрын
Thank you for watching my videos. Appreciate your valuable inputs here. I shall make this in my plan.
@ananbanerjee3111
@ananbanerjee3111 2 ай бұрын
great video, thanks
@cloudquicklabs
@cloudquicklabs 2 ай бұрын
Thank you for watching my videos. Glad that it helped you.
@andrewbateman2282
@andrewbateman2282 Жыл бұрын
Useful informative video. Thanks.
@cloudquicklabs
@cloudquicklabs Жыл бұрын
Thank you for watching my videos. Glad that it helped you.
@varra19
@varra19 Жыл бұрын
@@cloudquicklabs Informative video... its previous video(where code is explained) is not having Audio. kzbin.info/www/bejne/mHmnaZSIlryNhbs. from 14 minutes
@sprinter5901
@sprinter5901 Жыл бұрын
8:27 I don't have the entry point function like you have. Its just an empty function with some comments inside.
@sprinter5901
@sprinter5901 Жыл бұрын
for those who want the code- import functions_framework # Triggered by a change in a storage bucket @functions_framework.cloud_event def hello_gcs(cloud_event): data = cloud_event.data event_id = cloud_event["id"] event_type = cloud_event["type"] bucket = data["bucket"] name = data["name"] metageneration = data["metageneration"] timeCreated = data["timeCreated"] updated = data["updated"] print(f"Event ID: {event_id}") print(f"Event type: {event_type}") print(f"Bucket: {bucket}") print(f"File: {name}") print(f"Metageneration: {metageneration}") print(f"Created: {timeCreated}") print(f"Updated: {updated}")
@cloudquicklabs
@cloudquicklabs Жыл бұрын
Thank you for watching my videos. Cloud is always evolutionary, it might have changed. Please find the necessary files from GitHub link shared in videos description. Note that GCP auto populating the code syntax as soon as you choose the run time at code option while creating Cloud Function.
@prashantshankavaram
@prashantshankavaram Жыл бұрын
Hi Anjan, thank you for the nice video. But the code given does not work. is it because the entry point has to be modified
@cloudquicklabs
@cloudquicklabs Жыл бұрын
Thank you for watching my videos. And thak you for sharing your inputs here.
@erinquinn6852
@erinquinn6852 23 күн бұрын
I get an error having to do with health checks on the container: The error message you're encountering means that the Cloud Run service failed to start properly due to an issue with the container's health check. Specifically, the container did not start and listen on the required port (PORT=8080) within the allocated timeout. Anyone have solves for this?
@cloudquicklabs
@cloudquicklabs 19 күн бұрын
Thank you for watching my videos. It must issue of proper configuration as I don't see reason for container health check here. Try from fresh once again.
@zzzmd11
@zzzmd11 10 ай бұрын
Hi, Thanks for the great informative video. can you explain the flow if the data srouce is from a Rest API. Can we have a dataflow configured to extract from a Rest API to big query with dataflow without having cloud functions or Apache beam scripts involved? Thanks a lot in advance..
@cloudquicklabs
@cloudquicklabs 10 ай бұрын
Thank you for watching my videos. Your requirement is custom ,where data source is API and you should query API to get data. And I believe in this case Cloud Function best suitable as API need invocation here.
@Prathibha-v8c
@Prathibha-v8c Ай бұрын
how to implement this same for xml data. if you have any reference please share them with me
@cloudquicklabs
@cloudquicklabs Ай бұрын
Thank you for watching my videos. Currently I don't have this video but I would create it soon.
@CarlosMarin-lp9xe
@CarlosMarin-lp9xe Жыл бұрын
Hi! I got this error "NameError: name 'data' is not defined". Does anyone know how to fix it? Thanks in advance.
@cloudquicklabs
@cloudquicklabs Жыл бұрын
Thank you for watching my videos. It looks to be code syntax issues. Please check you code again, May be you can re-use the file I shared in description
@hilo-coding-tutorials
@hilo-coding-tutorials Жыл бұрын
i had the exact same issue and copy/pasted your code directly into the cloud function. What line in your code do you define this variable?@@cloudquicklabs
@iFunktion
@iFunktion Жыл бұрын
Not sure how you managed this at all, I just get an error saying Container Failed to Start. Any tips on what might have failed because google cloud does not appear to give any help
@cloudquicklabs
@cloudquicklabs Жыл бұрын
Thank you for watching my videos. While I understand the difficulty to use GCP service (as community is very small), to me it looks that you have issue at setting Cloud Function, May be try creating new Cloud function once again , do follow the video carefully. All the best.
@erinquinn6852
@erinquinn6852 21 күн бұрын
I have also gotten this error, any advice?
@varra19
@varra19 Жыл бұрын
its previous video(where code is explained) is not having Audio. kzbin.info/www/bejne/mHmnaZSIlryNhbs.
@cloudquicklabs
@cloudquicklabs Жыл бұрын
Thank you for watching my videos. Yes there was miss in recording , Apologies. But the required code of the video can be found in video description which is missing piece in this video.
GCP Composer | Airflow GCS to BigQuery and BigQuery Operators
19:24
Anjan GCP Data Engineering
Рет қаралды 17 М.
진짜✅ 아님 가짜❌???
0:21
승비니 Seungbini
Рет қаралды 10 МЛН
Thank you mommy 😊💝 #shorts
0:24
5-Minute Crafts HOUSE
Рет қаралды 33 МЛН
Big Query Live Training - A Deep Dive into Data Pipelining
1:02:05
MeasureSchool
Рет қаралды 63 М.
Data Journey EP-02: Batch Ingestion 📦 - 5 ways to ingest files into Google Cloud
13:28
Load Data from GCS to BigQuery using Dataflow
15:58
TechTrapture
Рет қаралды 31 М.
Real time - Streaming Data from PubSub to BigQuery Using Dataflow in GCP
8:18
How to load data into BigQuery and query the data in table|How to Load CSV data into BigQuery in GCP
13:52