Configure Snowpipe with SNS | Snowflake | Big Data

[@kushalappabe1993]

dude! thank you so much for this, now i can probably stop my brain thinking about the pipeline over and over again. The sns via topic worked like a charm for me!🤓 i was trying with sqs but the pipeline never got established even though all the necessary configs were done as per snowflake documentation, still could'nt find out what went wrong. I would also like to see the same content via sqs if possible. thank you once again! keep up the good work✌

[@KnowledgeAmplifier1]

Hello Kushalappa BE , you can refer this video , if you want to implement Snowpipe with SQS -- kzbin.info/www/bejne/b5nWlaGCp91nn6c Happy Learning

[@kushalappabe1993]

@@KnowledgeAmplifier1 but according to snowflake documentation there is no need of python, anyways i will refer your link , ✌

[@KnowledgeAmplifier1]

@@kushalappabe1993 For snowpipe working , Python is not needed , I added Python to perform some transformation activity in s3 before leading the data in Snowflake using Snowpipe ...

[@varunjain2645]

very lucid and concise ..thanks 👍

[@KnowledgeAmplifier1]

You are welcome Varun Jain! Happy Learning

[@hindukilaru2042]

When I added a new record to the file and upload it again to the bucket do i need to run the pipe again or running the select statement is enough?

[@ravikreddy7470]

Can you make a video on SNS vs SQS, when should we use what service? Edit: I got the answer from your other video about SQS. Thanks !! :)

[@KnowledgeAmplifier1]

Glad to hear that Ravi K R! Happy Learning

[@prashanth.s6540]

Hi ... How to get email notifications or error message for snowpipe data load to snowflake tables

[@KnowledgeAmplifier1]

Hello Prashant , please refer this video -- Snowflake Snowpipe - Email Alert Mechanism kzbin.info/www/bejne/bYmYY32sZ9qZotE Hope this will be helpful! Happy Learning :-)

[@balajits9081]

Hello your contents are helpful, I need to load the JSON file data to snowflake db using SNS. Please help me on this

[@KnowledgeAmplifier1]

Hello balaji ts , first of all, Glad to know the video is helpful to you 😊! Now coming your requirement , hope from this video you understood clearly how to configure Snowpipe using SNS , now all you need to know , how to use Snowpipe to load Json data , for that , you can refer this video -- kzbin.info/www/bejne/gXTSdGygqpJ1nKc , so all you have to do is combine the 2 concepts , that's all. Hope this will be helpful! Happy Learning

[@balajits9081]

@@KnowledgeAmplifier1 Thanks for your response, the twist is need to trigger the SNS using lamda function. Which means i already created one sns to trigger snowpipe once JSON file arrives on S3 I have another bucket once files arrives there need to run lambda function to call the same SNS..how we can achieve it

[@KnowledgeAmplifier1]

@@balajits9081 from the another s3 bucket also , you can send the events to same SNS directly na? do you really need Lambda there?

[@balajits9081]

@@KnowledgeAmplifier1 yes I need lambda there that is the requirements

[@balajits9081]

@@KnowledgeAmplifier1 Hi any luck? how we can call SNS using lambda

[@vikinist]

Nice one but its very fast here

[@KnowledgeAmplifier1]

Sorry for the inconvenience Vikram , actually some part of the topic is similar like snowpipe with SQS setup which I already covered in my previous video , so went bit fast so that it won't appear reiteration of previous video and feel boring 😅 You can decrease the playback speed in setting as per your convenience. To explore more on Snowflake with AWS , you can have a look in this doc --doc.clickup.com/37466271/d/h/13qc4z-104/d4346819bd8d510 Happy Learning :-)