Refreshing External Snowflake Tables Automatically for Amazon S3 using SNS

  Рет қаралды 1,498

Knowledge Amplifier

Knowledge Amplifier

Күн бұрын

Prerequisite:
----------------------
Configuring a Snowflake Storage Integration to Access Amazon S3
• Configuring a Snowflak...
Refreshing External Snowflake Tables Automatically for Amazon S3
• Refreshing External Sn...
Code:
-----------
Step 1: Create IAM role for Snowflake
Step 2:Execute below queries in Snowflake
drop database if exists s3_to_snowflake;
--Database Creation
create database if not exists s3_to_snowflake;
--Specify the active/current database for the session.
use s3_to_snowflake;
--Specify the role
use role ACCOUNTADMIN;
Step 3:Create s3 bucket--demoytsnsautomation
Step 4:Execute below queries in Snowflake --
--Storage Integration Creation
create or replace storage integration s3_int
TYPE = EXTERNAL_STAGE
STORAGE_PROVIDER = S3
ENABLED = TRUE
STORAGE_AWS_ROLE_ARN = ''
STORAGE_ALLOWED_LOCATIONS = ('s3://{}')
COMMENT = 'Testing Snowflake getting refresh or not';
--Describe the Integration Object
DESC INTEGRATION s3_int;
--External Stage Creation
create stage mystage
url = 's3://demosnowflakesns'
storage_integration = s3_int;
list @mystage;
--File Format Creation
create or replace file format my_csv_format
type = csv field_delimiter = ',' skip_header = 1
field_optionally_enclosed_by = '"'
null_if = ('NULL', 'null')
empty_field_as_null = true;
Step 5:Create SNS Topic --demoytsnsautomation
Access Policy --Make a copy
Step 6:Execute below query in Snowflake & update SNS access policy--
select system$get_aws_sns_iam_policy('{}');
Step 7:Create event notification for s3
Step 8:
--Table Creation
create or replace external table s3_to_snowflake.PUBLIC.Iris_dataset (Id number(10,0) as (Value:c1::int),sepal_length number(10,5) as (Value:c2::number(10,5)),
sepal_width number(10,4) as (Value:c3::number(10,4)),petal_length number(10,3) as (Value:c4::number(10,3)),
petal_width number(10,4) as (Value:c5::number(10,4)), CLASS_NAME varchar(20) as (Value:c6::varchar)) with location = @mystage file_format ='my_csv_format'
aws_sns_topic = '';
select * from s3_to_snowflake.PUBLIC.Iris_dataset;
Check this playlist for more AWS Projects in Big Data domain:
• Demystifying Data Engi...

Пікірлер
@adityakommu344
@adityakommu344 2 жыл бұрын
can same topic be configured to multiple s3 buckets?
Quando A Diferença De Altura É Muito Grande 😲😂
00:12
Mari Maria
Рет қаралды 45 МЛН
黑天使被操控了#short #angel #clown
00:40
Super Beauty team
Рет қаралды 61 МЛН
Solving one of PostgreSQL's biggest weaknesses.
17:12
Dreams of Code
Рет қаралды 228 М.
Snowflake BUILD | How To Use Apache Iceberg With Snowflake And AWS
14:33
Snowflake Developers
Рет қаралды 7 М.
Configure Snowpipe with SNS | Snowflake | Big Data
19:53
Knowledge Amplifier
Рет қаралды 4,1 М.
AWS S3 Event Notifications with SNS and SQS
14:11
AWS Training by Gaurav Agrawal
Рет қаралды 4 М.
Delta Live Tables A to Z: Best Practices for Modern Data Pipelines
1:27:52
Load Data from Amazon AWS S3 Bucket to Snowflake Data Warehouse
11:19
Saad Qureshi Official
Рет қаралды 16 М.
Think Fast, Talk Smart: Communication Techniques
58:20
Stanford Graduate School of Business
Рет қаралды 44 МЛН
Building Pipelines with Dynamic Tables
21:23
Snowflake Developers
Рет қаралды 18 М.
AWS S3 File Upload + Lambda Trigger - Step by Step Tutorial
12:44
Be A Better Dev
Рет қаралды 125 М.
Quando A Diferença De Altura É Muito Grande 😲😂
00:12
Mari Maria
Рет қаралды 45 МЛН