GCP Tutorial - 22 : Load data from GCS to BigQuery | BQ Load | Python API

  Рет қаралды 507

Swatech Talks

Swatech Talks

Күн бұрын

from google.cloud import bigquery
Create BQ Client
client = bigquery.Client()
table_id = "gcp-dev-409815.bq_gcs_loads.peoples"
Creating the schema for BQ tables
schema = [
bigquery.SchemaField("index", "INTEGER"),
bigquery.SchemaField("user_id", "STRING"),
bigquery.SchemaField("first_name", "STRING"),
bigquery.SchemaField("last_name", "STRING"),
bigquery.SchemaField("gender", "STRING"),
bigquery.SchemaField("email", "STRING"),
bigquery.SchemaField("phone", "STRING"),
bigquery.SchemaField("dob", "DATE"),
bigquery.SchemaField("job_title", "STRING")
]
Create Job Configuration
job_config = bigquery.LoadJobConfig(schema=schema, skip_leading_rows=1, source_format=bigquery.SourceFormat.CSV)
uri = "gs://gcp-dev-409815/csv_data/people-100.csv"
load_job = client.load_table_from_uri(uri, table_id, job_config=job_config)
load_job.result()
destination_table = client.get_table(table_id)
print(f"Total number of records in table : {destination_table.num_rows}")

Пікірлер
Big Query Data Transfer Service | GCS to Big Query data transfer
12:42
Anjan GCP Data Engineering
Рет қаралды 12 М.
Andro, ELMAN, TONI, MONA - Зари (Official Audio)
2:53
RAAVA MUSIC
Рет қаралды 8 МЛН
Create A Python API in 12 Minutes
12:05
Tech With Tim
Рет қаралды 749 М.
Load and extract data in BigQuery using Python
16:05
TechTrapture
Рет қаралды 6 М.
Load Data from GCS to BigQuery using Dataflow
15:58
TechTrapture
Рет қаралды 32 М.
ELT batch pipeline with Cloud Storage, BigQuery and Composer/Airflow
30:22
GCP Learning with Mazlum & GroupBees
Рет қаралды 2,5 М.
GCP Big Query Batch data loading | console, bq tool, Python API
13:27
Anjan GCP Data Engineering
Рет қаралды 8 М.
BigQuery Pricing - How It Works and Best Practices
6:46
Ternary Data
Рет қаралды 10 М.
Real time - Streaming Data from PubSub to BigQuery Using Dataflow in GCP
8:18