How to import CSV file from Amazon S3 to Redshift using AWS Glue Jobs

[@asishb]

Very very underrated video. I hope it reaches everyone. A question - what is the use of the classifier ?

[@yadi4diamond]

Very much helpful , simple, short and covered all details. Thank you!

[@antonietapintorebelo5020]

Thank you so much! This was a very helpful video. Keep it up!

[@cafeconlechecodes]

Hello there! I'm really glad that I helped you. Thanks for your kind words!

[@denisgomonov2320]

@@cafeconlechecodes Hey ! What is the difference between doing this via Glue or DataPipeline, can both be automated in the same fashion?

[@cafeconlechecodes]

@@denisgomonov2320 Hello! The main difference for me is that with Glue you "only" focus on the ETL job whereas DataPipeline allows you to manage even the smallest instance details (like VPC in EC2 for example). Yes, both can be automated to do the same job. Fun fact: I have even replaced Glue with only Lambda for some specific tasks.

[@rajatpathak4499]

great job

[@cafeconlechecodes]

Thank you 😃

[@rajatpathak4499]

if you can add more videos with some transformation in glue then load data, and some workflow setup that would be great

[@cafeconlechecodes]

Hello! Sure, I will try to add more.

[@alexissebastian7614]

how can you assign a dynamic name in glue??

[@roshinidrakshapally5011]

Hi, I am getting an error "An error occurred while calling o128.pyWriteDynamicFrame. Exception thrown in awaitResult:" when I run the job. Please help me with this. Thanks in advance!

[@SreevishakhBC-yo4kv]

Can you send that CSV file ?

[@AmolModhekar]

Can you tell me if it is we have 1000 table in S3 then how we dealing with redshift special schema issue if possible suggest proper way to resolve this

[@cafeconlechecodes]

Hello. If you have 1000 csv in S3, you can run a first crawl to enable the crawler to record the initial schema. If every time you run a Job, you have 1000 new csv files, then I suggest setting up your crawler for New files only.

[@AmolModhekar]

@@cafeconlechecodes thanks for reply

[@neon_horizon]

I got an error when creating crawler-redshift. I found VPC S3 endpoint validation failed in cloud watch log.

[@asishb]

You need to add a VPC endpoint connection - repost.aws/knowledge-center/glue-s3-endpoint-validation-failed

[@nehalverma-g4t]

Am not able to have successful connection established to redshift. Please help?

[@cafeconlechecodes]

Hello. Which error are you getting? And also tell me in which part of the video you are getting the error.

[@fawadh]

I'm Having an error in a job! Can't fix it! can you help? An error occurred while calling o131.pyWriteDynamicFrame. Exception thrown in awaitResult:

[@cafeconlechecodes]

Please, tell in which part of the video you are getting the error. I guess it's in the Glue Job?